av9ash/GitBugs huggingface dataset:
Bug: Spark submit fails if we provide extraJavaOption which contains Xmx  as substring
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 23/Apr/20 10:43
Affects Version/s: 2.4.0

Spark submit doesn't allow Xmx anywhere in the spark.driver.extraJavaOptions
 For eg:
{code:java}
bin\spark-submit --class org.apache.spark.examples.SparkPi --master local[*] --conf "spark.driver.extraJavaOptions=-DmyKey=MyValueContainsXmx" examples\jars\spark-examples_2.11-2.4.4.jar
Error: Not allowed to specify max heap(Xmx) memory settings through java options (was -DmyKey=MyValueContainsXmx). Use the corresponding --driver-memory or spark.driver.memory configuration instead.{code}
https://github.com/apache/spark/blob/v2.4.4/launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java#L102

[https://github.com/apache/spark/blob/master/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java#L302]

Can we update the above condition to check more specific for eg -Xmx
Bug: Kerberos: Spark UGI credentials are not getting passed down to Hive
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 22/Apr/20 02:26
Affects Version/s: 2.4.4

I'm using Spark-2.4, I have a Kerberos enabled cluster where I'm trying to run a query via the {{spark-sql}} shell.

The simplified setup basically looks like this: spark-sql shell running on one host in a Yarn cluster -> external hive-metastore running one host -> S3 to store table data.

When I launch the {{spark-sql}} shell with DEBUG logging enabled, this is what I see in the logs:
{code:java}
> bin/spark-sql --proxy-user proxy_user 

...
DEBUG HiveDelegationTokenProvider: Getting Hive delegation token for proxy_user against hive/_HOST@REALM.COM at thrift://hive-metastore:9083 

DEBUG UserGroupInformation: PrivilegedAction as:spark/spark_host@REALM.COM (auth:KERBEROS) from:org.apache.spark.deploy.security.HiveDelegationTokenProvider.doAsRealUser(HiveDelegationTokenProvider.scala:130){code}
This means that Spark made a call to fetch the delegation token from the Hive metastore and then added it to the list of credentials for the UGI. [This is the piece of code|https://github.com/apache/spark/blob/branch-2.4/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala#L129] that does that. I also verified in the metastore logs that the {{get_delegation_token()}} call was being made.

Now when I run a simple query like {{create table test_table (id int) location "s3://some/prefix";}} I get hit with an AWS credentials error. I modified the hive metastore code and added this right before the file system in Hadoop is initialized ([org/apache/hadoop/hive/metastore/Warehouse.java|#L116]):
{code:java}
 public static FileSystem getFs(Path f, Configuration conf) throws MetaException {
    try {
      UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
      LOG.info("UGI information: " + ugi);
      Collection<Token<? extends TokenIdentifier>> tokens = ugi.getCredentials().getAllTokens();
      for(Token token : tokens) {
        LOG.info(token);
      }
    } catch (IOException e) {
      e.printStackTrace();
    }
...
{code}
In the metastore logs, this does print the correct UGI information:
{code:java}
UGI information: proxy_user (auth:PROXY) via hive/hive-metastore@REALM.COM (auth:KERBEROS){code}
but there are no tokens present in the UGI. Looks like [Spark code|https://github.com/apache/spark/blob/branch-2.4/core/src/main/scala/org/apache/spark/deploy/security/HiveDelegationTokenProvider.scala#L101] adds it with the alias {{hive.server2.delegation.token}} but I don't see it in the UGI. This makes me suspect that somehow the UGI scope is isolated and not being shared between spark-sql and hive metastore. How do I go about solving this? Any help will be really appreciated!
Bug: sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner() method not found during spark-submit
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 23/Apr/20 14:43
Affects Version/s: 2.4.5

I am trying to run the following Spark submit from a VM using Yarn cluster mode.
 ./spark-submit --master yarn --deploy-mode client test_spark_yarn.py

The VM has java version 11 and spark-2.4.5 while the yarn cluster java 8 and spark-2.4.0. I am getting the error below:

!error.PNG!
Bug: Remove millennium, century, decade from  trunc/date_trunc fucntions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Apr/20 10:01
Affects Version/s: 3.0.0

Same as #SPARK-31507, millennium, century, and decade are not commonly used in most modern platforms.

for example
Negative:
https://docs.snowflake.com/en/sql-reference/functions-date-time.html#supported-date-and-time-parts
https://prestodb.io/docs/current/functions/datetime.html#date_trunc
https://teradata.github.io/presto/docs/148t/functions/datetime.html#date_trunc
https://www.oracletutorial.com/oracle-date-functions/oracle-trunc/

Positive:
https://docs.aws.amazon.com/redshift/latest/dg/r_Dateparts_for_datetime_functions.html
https://www.postgresql.org/docs/9.1/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC
Bug: Backport SPARK-27676       InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 23/Apr/20 20:12
Affects Version/s: 2.4.6

Backport SPARK-27676       InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles

cc [~joshrosen] I think backporting this has been asked in the original ticket, do you have any objections?
Bug: Backport SPARK-25407       Allow nested access for non-existent field for Parquet file when nested pruning is enabled
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 23/Apr/20 20:02
Affects Version/s: 2.4.6

Consider backporting SPARK-25407       Allow nested access for non-existent field for Parquet file when nested pruning is enabled to 2.4.6
Bug: Show the correct alias in schema for expression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Apr/20 07:09
Affects Version/s: 3.1.0

Spark SQL exists some function no elegant implementation alias.
For example: BitwiseCount override the sql method

{code:java}
override def sql: String = s"bit_count(${child.sql})" 
{code}

I don't think it's elegant enough.
Because `Expression` gives the following definitions.

{code:java}
  def sql: String = {
    val childrenSQL = children.map(_.sql).mkString(", ")
    s"$prettyName($childrenSQL)"
  }
{code}

By this definition, BitwiseCount should override `prettyName` method.
Bug: R installation in Github Actions is being failed
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 28/Apr/20 02:21
Affects Version/s: 2.4.5

Currently, R installation seems being failed as below:

{code}
Get:61 https://dl.bintray.com/sbt/debian  Packages [4174 B]
Get:62 http://ppa.launchpad.net/apt-fast/stable/ubuntu bionic/main amd64 Packages [532 B]
Get:63 http://ppa.launchpad.net/git-core/ppa/ubuntu bionic/main amd64 Packages [3036 B]
Get:64 http://ppa.launchpad.net/ondrej/php/ubuntu bionic/main amd64 Packages [52.0 kB]
Get:65 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu bionic/main amd64 Packages [33.9 kB]
Get:66 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu bionic/main Translation-en [10.1 kB]
Reading package lists...
E: The repository 'https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ Release' does not have a Release file.
##[error]Process completed with exit code 100.
{code}
Bug: Legacy parser incorrectly interprets pre-Gregorian dates
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Apr/20 00:05
Affects Version/s: 3.0.0

With CSV:
{noformat}
scala> sql("set spark.sql.legacy.timeParserPolicy=LEGACY")
res0: org.apache.spark.sql.DataFrame = [key: string, value: string]

scala> val seq = Seq("0002-01-01", "1000-01-01", "1500-01-01", "1800-01-01").map(x => s"$x,$x")
seq: Seq[String] = List(0002-01-01,0002-01-01, 1000-01-01,1000-01-01, 1500-01-01,1500-01-01, 1800-01-01,1800-01-01)

scala> val ds = seq.toDF("value").as[String]
ds: org.apache.spark.sql.Dataset[String] = [value: string]

scala> spark.read.schema("expected STRING, actual DATE").csv(ds).show
+----------+----------+
|  expected|    actual|
+----------+----------+
|0002-01-01|0001-12-30|
|1000-01-01|1000-01-06|
|1500-01-01|1500-01-10|
|1800-01-01|1800-01-01|
+----------+----------+

scala> 
{noformat}
Similarly, with JSON:
{noformat}
scala> sql("set spark.sql.legacy.timeParserPolicy=LEGACY")
res0: org.apache.spark.sql.DataFrame = [key: string, value: string]

scala> val seq = Seq("0002-01-01", "1000-01-01", "1500-01-01", "1800-01-01").map { x =>
  s"""{"expected": "$x", "actual": "$x"}"""
}

     |      | seq: Seq[String] = List({"expected": "0002-01-01", "actual": "0002-01-01"}, {"expected": "1000-01-01", "actual": "1000-01-01"}, {"expected": "1500-01-01", "actual": "1500-01-01"}, {"expected": "1800-01-01", "actual": "1800-01-01"})

scala> 
scala> val ds = seq.toDF("value").as[String]
ds: org.apache.spark.sql.Dataset[String] = [value: string]

scala> spark.read.schema("expected STRING, actual DATE").json(ds).show
+----------+----------+
|  expected|    actual|
+----------+----------+
|0002-01-01|0001-12-30|
|1000-01-01|1000-01-06|
|1500-01-01|1500-01-10|
|1800-01-01|1800-01-01|
+----------+----------+

scala> 
{noformat}
Bug: An optimized K-Means based on DenseMatrix and GEMM
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Apr/20 02:08
Affects Version/s: 3.1.0

The main computations in K-Means are calculating distances between individual points and center points. Currently K-Means implementation is vector-based which can't take advantage of optimized native BLAS libraries.

When the original points are represented as dense vectors, our approach is to modify the original input data structures to a DenseMatrix-based one by grouping several points together. The original distance calculations can be translated into a Matrix multiplication then optimized native GEMM routines (Intel MKL, OpenBLAS etc.) can be used. This approach can also work with sparse vectors despite having larger memory consumption when translating sparse vectors to dense matrix.

Our preliminary benchmark shows this DenseMatrix+GEMM approach can boost the training performance by *3.5x* with Intel MKL, looks very promising!

To minimize end user impact, proposed changes are to use config parameters to control if turn on this implementation without modifying public interfaces. Parameter rowsPerMatrix is used to control how many points are grouped together to build a DenseMatrix. An example:

$ spark-submit --master $SPARK_MASTER \

    --conf "spark.ml.kmeans.matrixImplementation.enabled=true" \

    --conf "spark.ml.kmeans.matrixImplementation.rowsPerMatrix=5000" \

    --class org.apache.spark.examples.ml.KMeansExample 

Several code changes are made in "spark.ml" namespace as we think "spark.mllib" is in maintenance mode, some are duplications from spark.mllib for using private definitions in the same package: 
 - Modified: KMeans.scala, DatasetUtils.scala

 - Added: KMeansMatrixImpl.scala

 - Duplications: DistanceMeasure.scala, LocalKMeans.scala

If this general idea is accepted by community, we are willing to contribute our code to upstream and polish the implementation according to feedbacks and produce benchmarks.
Bug: Support Z-order curve expression
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Apr/20 02:05
Affects Version/s: 3.1.0

Z-ordering is a technique that allows you to map multidimensional data to a single dimension.  We can use this feature to improve query performance. 

More details:
https://en.wikipedia.org/wiki/Z-order_curve
https://aws.amazon.com/blogs/database/z-order-indexing-for-multifaceted-queries-in-amazon-dynamodb-part-1/
https://aws.amazon.com/blogs/aws/quickly-filter-data-in-amazon-redshift-using-interleaved-sorting/
https://docs.oracle.com/database/121/DWHSG/attcluster.htm#DWHSG-GUID-7B007A3C-53C2-4437-9E71-9ECECF8B4FAB

Benchmark result:
These. 2 tables ordered and z-ordered  by AUCT_END_DT, AUCT_START_DT.
Filter on the AUCT_START_DT column:
 !zorder.png! 
 !lexicalorder.png! 
Filter on the auct_end_dt column:
 !zorder-AUCT_END_DT.png! 
 !lexicalorder-AUCT_END_DT.png!
Bug: Kafka connector does not retry in case of RetriableException
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 15/Apr/20 13:47
Affects Version/s: 2.4.5

The Spark DStream API uses spark-streaming-kafka-0-10 to talk to Kafka. The method in the connector code that's responsible for committing offset, commitAll, calls commitAsync in Kafka client to commit the offsets. commitAsync tries to find the group coordinator and sends the commits in case of success, or throws a RetriableCommitFailedException in case of failure and doesn't retry. This behavior was introduced in KAFKA-4034. The reason for not attempting retry was written there as: "we don't want recursive retries which can cause offset commits to arrive out of order". From the Spark side though, we should be able to retry when running into a RetriableException. The issue of potentially committing offsets out of order can be addressed by keeping a monotonically increasing sequence number every time a commit happens and including this number in the callback function of commitAsync.
Bug: Recommend user to disable delay scheduling for barrier taskset
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 21/Apr/20 13:38
Affects Version/s: 2.4.0

Currently, barrier taskset can only be scheduled when all tasks are launched at the same time. As a result, delay scheduling becomes the biggest obstacle for the barrier taskset  to get scheduled. And the application can be easily fail with barrier taskset when delay scheduling is on. So, probably, we should recommend user to disable delay scheduling while using barrier execution.

 

BTW, we could still support delay scheduling with barrier taskset if we find a way to resolve SPARK-24818
Bug: createSparkUser lost user's non-Hadoop credentials
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 24/Apr/20 05:38
Affects Version/s: 2.4.4

See current *[createSparkUser|https://github.com/apache/spark/blob/263f04db865920d9c10251517b00a1b477b58ff1/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L66-L76]*:
{code:java}
   def createSparkUser(): UserGroupInformation = {
    val user = Utils.getCurrentUserName()
    logDebug("creating UGI for user: " + user)
    val ugi = UserGroupInformation.createRemoteUser(user)
    transferCredentials(UserGroupInformation.getCurrentUser(), ugi)
    ugi
  }

  def transferCredentials(source: UserGroupInformation, dest: UserGroupInformation): Unit = {
    dest.addCredentials(source.getCredentials())
  }

  def getCurrentUserName(): String = {
    Option(System.getenv("SPARK_USER"))
      .getOrElse(UserGroupInformation.getCurrentUser().getShortUserName())
  }
{code}
The *transferCredentials* func can only transfer Hadoop creds such as Delegation Tokens.
 However, other creds stored in UGI.subject.getPrivateCredentials, will be lost here, such as:
 # Non-Hadoop creds:
 Such as, [Kafka creds |https://github.com/apache/kafka/blob/f3c8bff311b0e4c4d0e316ac949fe4491f9b107f/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/OAuthBearerLoginModule.java#L395]
 # Newly supported or 3rd party supported Hadoop creds:
 Such as to support OAuth/JWT token authn on Hadoop, we need to store the OAuth/JWT token into UGI.subject.getPrivateCredentials. However, these tokens are not supposed to be managed by Hadoop Credentials (currently it is only for Hadoop secret keys and delegation tokens)

Another issue is that the *SPARK_USER* only gets the UserGroupInformation.getCurrentUser().getShortUserName() of the user, which may lost the user's fully qualified user name. We should better use the *getUserName* to get fully qualified user name in our client side, which is aligned to *[HADOOP_PROXY_USER|https://github.com/apache/hadoop/blob/30ef8d0f1a1463931fe581a46c739dad4c8260e4/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L716-L720]*.

Related to https://issues.apache.org/jira/browse/SPARK-1051
Bug: No call to destroy() for filter in SparkHistory
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 03/Apr/20 13:00
Affects Version/s: 2.4.5

Adding  UI filter AuthenticationFilter (from Hadoop) causes Spark application to never end, due to threads created in this class not interrupted.

*To reproduce*

Start a local spark context with hadoop-auth 3.1.0 
{{spark.ui.enabled=true}}
{{spark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter}}
{{#and all required ldap props}}
{{spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.param.ldap.*=...}}

*What's happening :*

In [AuthenticationFilter's |https://github.com/apache/hadoop/blob/branch-3.1/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java] init we have the following chain:

{{(line.178) initializeSecretProvider(filterConfig);}}
{{(}}{{line.209) secretProvider = constructSecretProvider(...)}}
{{(}}{{line 237) provider.init(config, ctx, validity);}}

If no config is specified provider will be [RolloverSignerSecretProvider |https://github.com/apache/hadoop/blob/branch-3.1/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/RolloverSignerSecretProvider.java]which will (line 95) start a new thread via
{{scheduler = Executors.newSingleThreadScheduledExecutor();}}

The created thread will be stopped in destroy() method (line 106).

*Unfortunately, this destroy() method is not called* when SparkHistory is closed, leaving threads running.

 

This ticket is not here to address the particular case of Hadoop's authentication filter, but to ensure that any Filter added in spark.ui will have its destroy() method called.
Bug: Proposal to add Proximity Measure in Random Forest
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 02/Apr/20 21:59
Affects Version/s: 2.4.5

h3. Background

The RandomForest model does not provide proximity measure as described in [Breiman|[https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm]]. There are many important use cases of proximity:
 - more accurate replacement for missing data

 - identify outliers

 - clustering or multi-dimensional scaling

 - compute the proximities of test set in the training set

 - unsupervised learning

Performance and storage concerns are among reasons that proximities are not computed and kept during prediction, as mentioned in [https://dzone.com/articles/classification-using-random-forest-with-spark-20.|https://dzone.com/articles/classification-using-random-forest-with-spark-20]
h3. Proposal

RF in Spark is optimized for massive scalability on large-scale dataset where the number of data points, features and trees can be very big. Even with optimized storage, proximity requires O(NxT) memory, and it may still not fit in memory: where N is number of data points and T is number of trees in the forest.

We propose to add a column in the prediction output to return the node-id (or hash) of the terminal node for each sample data point.

The required changes on the current RF implementation will not increase the computation and storage by significant amounts. And it will leave the possibility open for computing some form of proximity after prediction. It us up to the users how to use the extra column of node-ids. Without this, currently there is no work around to compute proximity measure.
h4. Experiment on Spark 2.3.1 and 2.4.5

In one prototype, we output the terminal node id for each prediction from RandomForestClassificationModel. And then we use Spark’s LSHModel to cluster prediction results by terminal node ids. The performance of the whole pipeline was reasonable for the size of our dataset.
h3. References
 * L. Breiman. Manual on setting up, using, and understanding random forests v3.1, 2002. [https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm]
 * [https://dzone.com/articles/classification-using-random-forest-with-spark-20]
Bug: Unable to write  data into hive table using Spark via Hive JDBC driver Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 30/Apr/20 05:29
Affects Version/s: 2.3.1

I'm trying to wrire data into hive table using a JDBC connection to Hive. Unfortunately, when I write data that resides in every column I get the following error:

org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:36 cannot recognize input near '.' 'aname' 'TEXT' in column type
 at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:255)
 at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:241)

1)  On Hive create a simple table,its name is "test"，it have three column(aname,score,banji),their type both are "String"

2)important code:

object HiveDialect extends JdbcDialect {
 override def canHandle(url: String): Boolean = url.startsWith("jdbc:hive2")|| url.contains("hive2")
 override def quoteIdentifier(colName: String): String = s"$colName"
 }

-------------------------------------------------------------------

object callOffRun {
 def main(args: Array[String]): Unit = {

val spark = SparkSession.builder().enableHiveSupport().getOrCreate()

JdbcDialects.registerDialect(HiveDialect)
 val props = new Properties()
 props.put("driver","org.apache.hive.jdbc.HiveDriver")
 props.put("user","username")
 props.put("password","password")
 props.put("fetchsize","20")

val table=spark.read.jdbc("jdbc:hive2://xxxxxxxx:10000","test",props)

table.write.jdbc("jdbc:hive2://xxxxxxxx:10000", "resulttable", props)

}

}

3)spark-submit ,After running,When table write，it have error 

4)table.count() have result 

5) i try some method to write data into table,They all reported the same error
Bug: Pyspark memory and cores calculation doesn't account for task cpus
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 14/Apr/20 14:00
Affects Version/s: 2.4.5

during changes for stage level scheduling, I discovered a possible issue that the calculation for splitting pyspark memory up doesn't take into account the spark.task.cpus setting.

Discussion here: [https://github.com/apache/spark/pull/28085#discussion_r407573038]

See PythonRunner.scala:

[https://github.com/apache/spark/blob/6b88d136deb99afd9363b208fd6fe5684fe8c3b8/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala#L90]