av9ash/GitBugs huggingface dataset:
Bug: Support of Columnar result(ColumnarBatch) in org.apache.spark.sql.Dataset flatMap, transform, etc
Status: Open | Priority: Major | Resolution: 
Created: 04/Sep/22 11:48
Affects Version/s: 3.3.0

Sometimes result of data transformation in JVM program available from native code in Apache Arrow columnar data format. Current Dataset API require unnecessary data transform from columnar format wrapper into row with additional allocation on JVM heap. 

In this proposed feature I ask for propagation of columnar data in DatasetAPI without unnecessary InternalRow->Row->InternalRow conversion.

 

Current solution use [ColumnarBatch wrapper|https://github.com/igor-suhorukov/spark3/blob/master/src/main/java/com/github/igorsuhorukov/arrow/spark/ArrowDataIterator.java] on top of ArrowColumnVector and rowExpressionEncoder.createDeserializer() to transform data [into Row|https://github.com/igor-suhorukov/spark3/blob/c655d4b6058fdd4529aa59093edfe2333d96fb05/src/main/java/com/github/igorsuhorukov/arrow/spark/ArrowDataIterator.java#L53]
Bug: Upgrading to Spark 3 is giving NullPointerException
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 02/Sep/22 16:45
Affects Version/s: 3.2.2

Getting below error while upgrading to Spark3

 

java.lang.RuntimeException: Error while decoding: java.lang.NullPointerException: Null value appeared in non-nullable field:
- array element class: "scala.Long"
- root class: "scala.collection.Seq"
If the schema is inferred from a Scala tuple/case class, or a Java bean, please try to use scala.Option[_] or other nullable types (e.g. java.lang.Integer instead of int/scala.Int).
mapobjects(lambdavariable(MapObject, LongType, true, -1), assertnotnull(lambdavariable(MapObject, LongType, true, -1)), input[0, array<bigint>, true], Some(interface scala.collection.Seq))
    at org.apache.spark.sql.errors.QueryExecutionErrors$.expressionDecodingError(QueryExecutionErrors.scala:1047)
    at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Deserializer.apply(ExpressionEncoder.scala:184)
    at org.apache.spark.sql.catalyst.expressions.ScalaUDF.$anonfun$scalaConverter$2(ScalaUDF.scala:164)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)
    at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:350)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:131)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1491)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Bug: try_sum() should throw the exceptions from its child
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Sep/22 23:21
Affects Version/s: 3.4.0

Similar to [https://github.com/apache/spark/pull/37486] and [https://github.com/apache/spark/pull/37663,]  the errors from try_sum()'s child should be shown instead of ignored.
Bug: Support more than Integer.MAX_VALUE of the same join key
Status: In Progress | Priority: Major | Resolution: 
Created: 01/Sep/22 15:01
Affects Version/s: 3.3.0

For SMJ, the number of the same join key records of the right table is greater than Integer.MAX_VALUE, the result will be incorrect. 
Before SMJ JOIN, we will put the records of the same join key into the ExternalAppendOnlyUnsafeRowArray. ExternalAppendOnlyUnsafeRowArray.numRows overflow may cause OOM. During SMJ JOIN, SpillableArrayIterator.startIndex overflow may cause incorrect result.

For example, one of our production table:

!image-2022-09-01-23-02-15-955.png!
Bug: Re-enable the `read_excel` tests
Status: Open | Priority: Minor | Resolution: 
Created: 05/Sep/22 23:16
Affects Version/s: 3.4.0

So far, we've been skipping the `read_excel` test in pandas API on Spark:
https://github.com/apache/spark/blob/6d2ce128058b439094cd1dd54253372af6977e79/python/pyspark/pandas/tests/test_dataframe_spark_io.py#L251

In https://github.com/apache/spark/pull/37671, we installed `openpyxl==3.0.10` to re-enable the `read_excel` tests, but it's still failing for some reason (Please see https://github.com/apache/spark/pull/37671#issuecomment-1237515485 for more detail).

We should re-enable this test for improving the pandas-on-Spark test coverage.
Bug: Improve pushdown for orc & parquet when cast scenario
Status: In Progress | Priority: Minor | Resolution: 
Created: 06/Sep/22 11:26
Affects Version/s: 3.4.0

Table Schema:
| a| b| part|
| int|string  | string|

 

SQL:
select * from table where b = 1 and part = 1

A.Partition column 'part' has been pushed down, but column 'b' not push down.
!https://user-images.githubusercontent.com/15246973/188630971-4eed59c0-d134-4994-a7aa-0d057a49c5dc.png!

B.After apply the pr, Partition column 'part' and column 'b' has been pushed down.
!https://user-images.githubusercontent.com/15246973/188631231-d30db19d-1d70-4712-bf75-bccaa2c790f4.png!
Bug: Python version for UDF should follow the servers version
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Sep/22 12:59
Affects Version/s: 3.4.0

Currently, we artificially pin the Python version to 3.9 in the UDF generation code, but this should actually be the correct server vs client version.

 

In addition the version should be configured as part of the function definition proto message.
Bug: Ignore stage fetch failure caused by decommissioned executor
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Sep/22 21:54
Affects Version/s: 3.4.0

When executor decommission is enabled, there would be many stage failure caused by FetchFailed from decommissioned executor, further causing whole job's failure. It would be better not to count such failure in `spark.stage.maxConsecutiveAttempts`
Bug: Propagate decommission executor loss reason during onDisconnect in K8s
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Sep/22 16:15
Affects Version/s: 3.4.0

Currently if an executor has been sent a decommission message and then it disconnects from the scheduler we only disable the executor depending on the K8s status events to drive the rest of the state transitions. However, the K8s status events can become overwhelmed on large clusters so we should check if an executor is in a decommissioning state when it is disconnected and use that reason instead of waiting on the K8s status events so we have more accurate logging information.
Bug: Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Sep/22 11:58
Affects Version/s: 3.4.0

*1.5.2-3 VS 1.5.2-4*

!image-2022-09-22-20-03-44-348.png|width=833,height=251!
Bug: Improve pyspark.sql.function example experience
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Sep/22 10:03
Affects Version/s: 3.3.0

There are many exanple in pyspark.sql.function:
{code:java}
    Examples
    --------
    >>> df = spark.range(1)
    >>> df.select(lit(5).alias('height'), df.id).show()
    +------+---+
    |height| id|
    +------+---+
    |     5|  0|
    +------+---+ {code}
We can add import statements so that the user can directly run it.
Bug: Negative size in error message when unsafe array is too big
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Sep/22 00:30
Affects Version/s: 3.4.0

When initializing an overly large unsafe array via {{UnsafeArrayWriter#initialize}}, {{BufferHolder#grow}} may report an error message with a negative size, e.g.:
{noformat}
java.lang.IllegalArgumentException: Cannot grow BufferHolder by size -2115263656 because the size is negative
{noformat}
(Note: This is not related to SPARK-39608, as far as I can tell, despite having the same symptom).

When calculating the initial size in bytes needed for the array, {{UnsafeArrayWriter#initialize}} uses an int expression, which can overflow. The initialize method then passes the negative size to {{BufferHolder#grow}}, which complains about the negative size.

Example (the following will run just fine on a 16GB laptop, despite the large driver size setting):
{noformat}
bin/spark-sql --driver-memory 22g --master "local[1]"

create or replace temp view data1 as
select 0 as key, id as val
from range(0, 268271216);

create or replace temp view data2 as
select key as lkey, collect_list(val) as bigarray
from data1
group by key;

-- the below cache forces Spark to create unsafe rows
cache lazy table data2;

select count(*) from data2;
{noformat}
After a few minutes, {{BufferHolder#grow}} will throw the following exception:
{noformat}
java.lang.IllegalArgumentException: Cannot grow BufferHolder by size -2115263656 because the size is negative
	at org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder.grow(BufferHolder.java:67)
	at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter.initialize(UnsafeArrayWriter.java:61)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.aggregate.Collect.serialize(collect.scala:73)
	at org.apache.spark.sql.catalyst.expressions.aggregate.Collect.serialize(collect.scala:37)
{noformat}
This query was going to fail anyway, but the message makes it looks like a bug in Spark rather than a user problem. {{UnsafeArrayWriter#initialize}} should calculate using a long expression and fail if the size exceeds {{Integer.MAX_VALUE}}, showing the actual initial size in the error message.
Bug: protoc-3.21.1-linux-x86_64.exe requires GLIBC_2.14 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Sep/22 06:32
Affects Version/s: 3.4.0

Compile Connect module on CentOS release 6.3, the default glibc version is 2.12, this will cause compilation to fail as follows:
{code:java}
[ERROR] PROTOC FAILED: /home/disk0/spark-source/connect/target/protoc-plugins/protoc-3.21.1-linux-x86_64.exe: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /home/disk0/spark-source/connect/target/protoc-plugins/protoc-3.21.1-linux-x86_64.exe)
/home/disk0/spark-source/connect/target/protoc-plugins/protoc-3.21.1-linux-x86_64.exe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by /home/disk0/spark-source/connect/target/protoc-plugins/protoc-3.21.1-linux-x86_64.exe)
/home/disk0/spark-source/connect/target/protoc-plugins/protoc-3.21.1-linux-x86_64.exe: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /home/disk0/spark-source/connect/target/protoc-plugins/protoc-3.21.1-linux-x86_64.exe)
/home/disk0/spark-source/connect/target/protoc-plugins/protoc-3.21.1-linux-x86_64.exe: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.5' not found (required by /home/disk0/spark-source/connect/target/protoc-plugins/protoc-3.21.1-linux-x86_64.exe) {code}
Bug: Improve error when cogrouping groups with mismatching key sizes
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Sep/22 13:54
Affects Version/s: 3.4.0

Cogrouping two grouped DataFrames in PySpark that have different group key cardinalities raises an error that is not very descriptive:
{code:python}
left.groupby("id", "k")
    .cogroup(right.groupby("id"))
{code}
{code:java}
Traceback (most recent call last):
py4j.protocol.Py4JJavaError: An error occurred while calling o726.collectToPython.
: java.lang.IndexOutOfBoundsException: 1
	at scala.collection.mutable.ResizableArray.apply(ResizableArray.scala:46)
	at scala.collection.mutable.ResizableArray.apply$(ResizableArray.scala:45)
	at scala.collection.mutable.ArrayBuffer.apply(ArrayBuffer.scala:49)
	at org.apache.spark.sql.catalyst.plans.physical.HashShuffleSpec.$anonfun$createPartitioning$5(partitioning.scala:650)
{code}
*Note:* This is Python-specific as cogrouping with differing group key sizes is not possible in Scala. The respective Scala API is fully typed on the key.

The problem is that {{EnsureRequirements.ensureDistributionAndOrdering}} calls into {{HashShuffleSpec.createPartitioning(clustering)}} where length of {{clustering}} is smaller than largest bits ({{{}v.head{}}}) in {{{}hashKeyPositions{}}} (EnsureRequirements.scala:159):
{code:java}
hashKeyPositions.map(v => clustering(v.head))
{code}

Possible fixes:
 # Assert identical size for group keys, and provide meaningful cogroup-specific error message.
 # {{EnsureRequirements}} identifies this situation and provide a meaningful distribution-requirements-specific error message.
 # Ideally both.
Bug: Spark Sum increases the precision of DecimalType arguments by 10
Status: Open | Priority: Minor | Resolution: 
Created: 05/Sep/22 12:19
Affects Version/s: 3.2.0

Currently in Spark automatically increases Decimal field by 10 (hard coded value) after SUM aggregate operation - [https://github.com/apache/spark/blob/branch-3.2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L1877.]

There are a couple of questions:
 # Why was 10 chosen as default one?
 # Does it make sense to allow the user to override this value via configuration?
Bug: Spark session does not update number of files for partition
Status: Open | Priority: Minor | Resolution: 
Created: 14/Sep/22 20:30
Affects Version/s: 3.1.2

When a spark session has already queried data from a table and partition and new files are inserted into the partition externally, the spark session keeps the outdated number of files and does not return the new records.
If the data is inserted into a new partition, the problem will not occur.

Steps to reproduce the behavior:

Open a Spark session
Query a count in a table
Open another spark session
insert data into an existing partition
Check the count again in the first session


I expect to see the inserted records.
Bug: Documentation error in "Integration with Cloud Infrastructures"
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Sep/22 15:31
Affects Version/s: 3.3.0

The artifactId that implements the integration with several cloud infrastructures is wrong. Instead of "hadoop-cloud-\{SCALA_VERSION}", it should say "spark-hadoop-cloud-\{SCALA_VERSION}".
Bug: Upgrade Scala to 2.13.9
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 22/Sep/22 02:51
Affects Version/s: 3.4.0

release notes:

[https://github.com/scala/scala/releases/tag/v2.13.9]

!image-2022-09-22-10-53-10-579.png!
Bug: Support automatic data format conversion for shuffle state db
Status: In Progress | Priority: Minor | Resolution: 
Created: 16/Sep/22 02:50
Affects Version/s: 3.4.0

After SPARK-38888,  Spark starts to support use RocksDB as the storage type of shuffle state db, but when users switch storage types, the original data store in `LevelDB/RocksDB` will not be automatically converted to another kind of storage now. The original data store will be retained and the new type data store will be  created when switching storage types.

This is not very friendly for migrating stock users to use RocksDB. If necessary, consider adding the function of automatic data format conversion:
 * LevelDB -> RocksDB
 * RocksDB -> LevelDB
Bug: Introduce withColumnsRenamed
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Sep/22 23:40
Affects Version/s: 3.0.3

Add a scala, pyspark, R dataframe API that can rename multiple columns in a single command. Issues are faced when users iteratively perform `withColumnRenamed`.
 * When it works, we see slower performace
 * In some cases, StackOverflowError is raised due to logical plan being too big
 * In a few cases, driver died due to memory consumption

Some reproducible benchmarks:
{code:java}
import datetime
import numpy as np
import pandas as pd

num_rows = 2
num_columns = 100
data = np.zeros((num_rows, num_columns))
columns = map(str, range(num_columns))
raw = spark.createDataFrame(pd.DataFrame(data, columns=columns))

a = datetime.datetime.now()
for col in raw.columns:
    raw = raw.withColumnRenamed(col, f"prefix_{col}")

b = datetime.datetime.now()
for col in raw.columns:
    raw = raw.withColumnRenamed(col, f"prefix_{col}")

c = datetime.datetime.now()
for col in raw.columns:
    raw = raw.withColumnRenamed(col, f"prefix_{col}")

d = datetime.datetime.now()
for col in raw.columns:
    raw = raw.withColumnRenamed(col, f"prefix_{col}")

e = datetime.datetime.now()
for col in raw.columns:
    raw = raw.withColumnRenamed(col, f"prefix_{col}")

f = datetime.datetime.now()
for col in raw.columns:
    raw = raw.withColumnRenamed(col, f"prefix_{col}")

g = datetime.datetime.now()
g-a
datetime.timedelta(seconds=12, microseconds=480021) {code}
{code:java}
import datetime
import numpy as np
import pandas as pd

num_rows = 2
num_columns = 100
data = np.zeros((num_rows, num_columns))
columns = map(str, range(num_columns))
raw = spark.createDataFrame(pd.DataFrame(data, columns=columns))

a = datetime.datetime.now()
raw = DataFrame(raw._jdf.withColumnsRenamed({col: f"prefix_{col}" for col in raw.columns}), spark)
b = datetime.datetime.now()
raw = DataFrame(raw._jdf.withColumnsRenamed({col: f"prefix_{col}" for col in raw.columns}), spark)
c = datetime.datetime.now()
raw = DataFrame(raw._jdf.withColumnsRenamed({col: f"prefix_{col}" for col in raw.columns}), spark)
d = datetime.datetime.now()
raw = DataFrame(raw._jdf.withColumnsRenamed({col: f"prefix_{col}" for col in raw.columns}), spark)
e = datetime.datetime.now()
raw = DataFrame(raw._jdf.withColumnsRenamed({col: f"prefix_{col}" for col in raw.columns}), spark)
f = datetime.datetime.now()
raw = DataFrame(raw._jdf.withColumnsRenamed({col: f"prefix_{col}" for col in raw.columns}), spark)
g = datetime.datetime.now()
g-a
datetime.timedelta(microseconds=632116) {code}
Bug: Add a shutdownhook to CoarseGrained scheduler to avoid dangling resources during abnormal shutdown
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Sep/22 18:36
Affects Version/s: 3.4.0

    Add a shutdown hook in the CoarseGrainedSchedulerBackend to call stop since we've got zombie pods hanging around since the resource tie isn't perfect.
Bug: Add additional tests to StreamingSessionWindowSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Sep/22 06:38
Affects Version/s: 3.3.0

Add complex tests to {{{}StreamingSessionWindowSuite{}}}. Concretely, I created two helper functions,
 * one is called {{{}sessionWindowQueryNestedKey{}}}, which would convert {{sessionId}} from the single word key used in {{sessionWindowQuery}} to a nested column key. For example: {{{}"hello" -> (("hello", "hello"), "hello"){}}}.
 * The other is called {{{}sessionWindowQueryMultiColKey{}}}. It would convert {{sessionId}} from the single word key used in {{sessionWindowQuery}} to two columns. For example: "hello" -> col1: ("hello", "hello"), col2: "hello"

With the two new helper functions, I added more tests for the tests for {{complete mode}} and {{cap gap duration}} ({{{}append{}}} and {{async state}} was not included, because the first two is enough for testing the change I added). The logic of the tests are not changed at all, just the key.

For the aggregation test ({{{}session window - with more aggregation functions{}}}), I added some more functions as well as a UDAF to test, and I tried {{first()}} and {{last()}} function on a nested triple, created using the same method as the above.
Bug: Only set KeyGroupedPartitioning when the referenced column is in the output
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Sep/22 20:19
Affects Version/s: 3.3.0


{code:java}
      sql(s"CREATE TABLE $tbl (id bigint, data string) PARTITIONED BY (id)")
      sql(s"INSERT INTO $tbl VALUES (1, 'a'), (2, 'b'), (3, 'c')")
      checkAnswer(
        spark.table(tbl).select("index", "_partition"),
        Seq(Row(0, "3"), Row(0, "2"), Row(0, "1"))
      )
{code}

failed with 
ScalaTestFailureLocation: org.apache.spark.sql.QueryTest at (QueryTest.scala:226)
org.scalatest.exceptions.TestFailedException: AttributeSet(id#994L) was not empty The optimized logical plan has missing inputs:
RelationV2[index#998, _partition#999] testcat.t
Bug: Upgrade protobuf-python from 4.21.5 to 4.21.6
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Sep/22 16:19
Affects Version/s: 3.4.0

[CVE-2022-1941|https://nvd.nist.gov/vuln/detail/CVE-2022-1941]

[Github|https://github.com/advisories/GHSA-8gq9-2x98-w8hf]
Bug: SharedState to redact secrets when propagating them to HadoopConf
Status: Open | Priority: Minor | Resolution: 
Created: 26/Sep/22 13:29
Affects Version/s: 3.3.0


When SharedState propagates (key, value) pairs from initialConfigs to HadoopConf, it logs the values at debug.

If the config contained secrets (cloud credentials, etc) the log will contain them.

The org.apache.hadoop.conf.ConfigRedactor class will redact values of all keys matching a patten in "hadoop.security.sensitive-config-keys"; this is configured by default to be


{code}
  "secret$",
  "password$",
  "ssl.keystore.pass$",
  "fs.s3.*[Ss]ecret.?[Kk]ey",
  "fs.s3a.*.server-side-encryption.key",
  "fs.s3a.encryption.algorithm",
  "fs.s3a.encryption.key",
  "fs.azure\\.account.key.*",
  "credential$",
  "oauth.*secret",
  "oauth.*password",
  "oauth.*token",
	"hadoop.security.sensitive-config-keys"
{code}

...And it may be extended in site configs/future hadoop releases

Spark should be using the redactor for log hygiene/security
Bug: Add PushProjectionThroughLimit for Optimizer
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Sep/22 09:22
Affects Version/s: 3.4.0

h4. It took a long time to fetch out, still running after 20 minutes...

when run as follow code in spark-shell:

spark.sql("select * from xxx where event_day = '20220919' limit 1").show()

[!https://user-images.githubusercontent.com/15246973/191211875-c29c3bae-1339-414b-84bc-2195545b8c35.png|width=557!|https://user-images.githubusercontent.com/15246973/191211875-c29c3bae-1339-414b-84bc-2195545b8c35.png]
[!https://user-images.githubusercontent.com/15246973/191212244-22108810-dd66-46bd-bea7-a7dab70a1a06.png|width=1419!|https://user-images.githubusercontent.com/15246973/191212244-22108810-dd66-46bd-bea7-a7dab70a1a06.png]
Bug: SparkSQLEnvSuite failed to clean the `spark_derby` directory after execution
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Sep/22 06:36
Affects Version/s: 3.4.0

run 
{code:java}
mvn clean install  -Phive-thriftserver -pl sql/hive-thriftserver -Dtest=none -DwildcardSuites=org.apache.spark.sql.hive.thriftserver.SparkSQLEnvSuite

git status {code}
The ` sql/hive-thriftserver/spark_derby/` directory will be found