av9ash/GitBugs huggingface dataset:
Bug: Failure of InSet.sql for UTF8String collection
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Apr/20 09:59
Affects Version/s: 2.4.5

The InSet expression works on collections of internal Catalyst's types. We can see this in the optimization when In is replaced by InSet, and In's collection is evaluated to internal Catalyst's values: [https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala#L253-L254]
{code:scala}
        if (newList.length > SQLConf.get.optimizerInSetConversionThreshold) {
          val hSet = newList.map(e => e.eval(EmptyRow))
          InSet(v, HashSet() ++ hSet)
        }
{code}
The code existed before the optimization https://github.com/apache/spark/pull/25754 that made another wrong assumption about collection types.

If InSet accepts only internal Catalyst's types, the following code shouldn't fail:
{code:scala}
InSet(Literal("a"), Set("a", "b").map(UTF8String.fromString)).sql
{code}
but it fails with the exception:
{code}
Unsupported literal type class org.apache.spark.unsafe.types.UTF8String a
java.lang.RuntimeException: Unsupported literal type class org.apache.spark.unsafe.types.UTF8String a
	at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:88)
	at org.apache.spark.sql.catalyst.expressions.InSet.$anonfun$sql$2(predicates.scala:522)
{code}
Bug: Wrong result of isInCollection for large collections
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Apr/20 11:34
Affects Version/s: 3.0.0

If the size of a collection passed to isInCollection is bigger than spark.sql.optimizer.inSetConversionThreshold, the method can return wrong results for some inputs. For example:
{code:scala}
    val set = (0 to 20).map(_.toString).toSet
    val data = Seq("1").toDF("x")
    println(set.contains("1"))
    data.select($"x".isInCollection(set).as("isInCollection")).show()
{code}
{code}
true
+--------------+
|isInCollection|
+--------------+
|         false|
+--------------+
{code}
Bug: Failure on pushing down filters with java.time.LocalDate values in ORC
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/20 17:46
Affects Version/s: 3.0.0

When spark.sql.datetime.java8API.enabled is set to true, filters pushed down with java.time.LocalDate values to ORC datasource fails with the exception:
{code}
Wrong value class java.time.LocalDate for DATE.EQUALS leaf
java.lang.IllegalArgumentException: Wrong value class java.time.LocalDate for DATE.EQUALS leaf
	at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.checkLiteralType(SearchArgumentImpl.java:192)
	at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.<init>(SearchArgumentImpl.java:75)
	at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$BuilderImpl.equals(SearchArgumentImpl.java:352)
	at org.apache.spark.sql.execution.datasources.orc.OrcFilters$.buildLeafSearchArgument(OrcFilters.scala:229)
{code}
Bug: Add SQL Rest API Documentation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Apr/20 00:47
Affects Version/s: 3.1.0

SQL Rest API exposes query execution metrics as Public API. Its documentation will be useful for end-users. 
{code:java}
/applications/[app-id]/sql

1- A list of all queries for a given application.
2- ?details=[true|false (default)] lists metric details in addition to queries details.
3- ?offset=[offset]&length=[len] lists queries in the given range.{code}
{code:java}
/applications/[app-id]/sql/[execution-id]

1- Details for the given query.
2- ?details=[true|false (default)] lists metric details in addition to given query details.{code}
Bug: Use agg column in Having clause behave different with column type 
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 03/Apr/20 06:08
Affects Version/s: 2.4.0

{code:java}
```

test("xxxxxxxx") {
    Seq(
      (1, 3),
      (2, 3),
      (3, 6),
      (4, 7),
      (5, 9),
      (6, 9)
    ).toDF("a", "b").createOrReplaceTempView("testData")

    val x = sql(
      """
        | SELECT b, sum(a) as a
        | FROM testData
        | GROUP BY b
        | HAVING sum(a) > 3
      """.stripMargin)

    x.explain()
    x.show()
  }

[info] - xxxxxxxx *** FAILED *** (508 milliseconds)
[info]   org.apache.spark.sql.AnalysisException: Resolved attribute(s) a#184 missing from a#180,b#181 in operator !Aggregate [b#181], [b#181, sum(cast(a#180 as double)) AS a#184, sum(a#184) AS sum(a#184)#188]. Attribute(s) with the same name appear in the operation: a. Please check if the right attribute(s) are used.;;
[info] Project [b#181, a#184]
[info] +- Filter (sum(a#184)#188 > cast(3 as double))
[info]    +- !Aggregate [b#181], [b#181, sum(cast(a#180 as double)) AS a#184, sum(a#184) AS sum(a#184)#188]
[info]       +- SubqueryAlias `testdata`
[info]          +- Project [_1#177 AS a#180, _2#178 AS b#181]
[info]             +- LocalRelation [_1#177, _2#178]
```
```
test("xxxxxxxx") {
    Seq(
      ("1", "3"),
      ("2", "3"),
      ("3", "6"),
      ("4", "7"),
      ("5", "9"),
      ("6", "9")
    ).toDF("a", "b").createOrReplaceTempView("testData")

    val x = sql(
      """
        | SELECT b, sum(a) as a
        | FROM testData
        | GROUP BY b
        | HAVING sum(a) > 3
      """.stripMargin)

    x.explain()
    x.show()
  }


== Physical Plan ==
*(2) Project [b#181, a#184L]
+- *(2) Filter (isnotnull(sum(cast(a#180 as bigint))#197L) && (sum(cast(a#180 as bigint))#197L > 3))
   +- *(2) HashAggregate(keys=[b#181], functions=[sum(cast(a#180 as bigint))])
      +- Exchange hashpartitioning(b#181, 5)
         +- *(1) HashAggregate(keys=[b#181], functions=[partial_sum(cast(a#180 as bigint))])
            +- *(1) Project [_1#177 AS a#180, _2#178 AS b#181]
               +- LocalTableScan [_1#177, _2#178]
```{code}
Spend A lot of time I can't find witch analyzer make this different,

When column type is double, it failed.
Bug: HiveThriftServer2Listener update methods fail with unknown operation/session id
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Apr/20 16:33
Affects Version/s: 2.3.4

HiveThriftServer2Listener update methods, such as onSessionClosed and onOperationError throw a NullPointerException (in Spark 3) or a NoSuchElementException (in Spark 2) when the input session/operation id is unknown. In Spark 2, this can cause control flow issues with the caller of the listener. In Spark 3, the listener is called by a ListenerBus which catches the exception, but it would still be nicer if an invalid update is logged and does not throw an exception.
Bug: Try assigning tasks to existing executors by which required resources in ResourceProfile are satisfied
Status: Open | Priority: Major | Resolution: 
Created: 13/Apr/20 12:41
Affects Version/s: 3.1.0

By the change in [PR|https://github.com/apache/spark/pull/27773] of SPARK-29154, submitted tasks are scheduled onto executors only if resource profile IDs strictly match. As a result Spark always starts new executors for customized ResourceProfiles.

This limitation makes working with process-local jobs unfriendly. E.g. Task cores has been increased from 1 to 4 in a new stage, and executor has 8 slots, it is expected that 2 new tasks can be run on the existing executor but Spark starts new executors for new ResourceProfile. The behavior is unnecessary.
Bug: spark.kubernetes.driver.podTemplateFile Configuration not used by the job
Status: Open | Priority: Major | Resolution: 
Created: 18/Apr/20 20:55
Affects Version/s: 3.0.0

Spark 3.0 - Running Spark Submit as below and point to a MinKube cluster
{code:java}
bin/spark-submit \
 --master k8s://https://192.168.99.102:8443 \
 --deploy-mode cluster \
 --name spark-pi \
 --class org.apache.spark.examples.SparkPi \
 --conf spark.kubernetes.driver.podTemplateFile=../driver_1E.template \
 --conf spark.kubernetes.executor.podTemplateFile=../executor.template \
 --conf spark.kubernetes.container.image=spark:spark3 \
 local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0-preview2.jar 1
{code}
 

Spark Binaries - spark-3.0.0-preview2-bin-hadoop2.7.tgz

Driver Template - 
{code:java}
apiVersion: v1
kind: Pod
metadata:
  labels:
    spark-app-id: my-custom-id
  annotations:
    spark-driver-cpu: 1
    spark-driver-mem: 1
    spark-executor-cpu: 1
    spark-executor-mem: 1
    spark-executor-count: 1
spec:
  schedulerName: spark-scheduler{code}
 Executor Template

 
{code:java}
apiVersion: v1
kind: Pod
metadata:
  labels:
    spark-app-id: my-custom-id
spec:
  schedulerName: spark-scheduler{code}
Kubernetes Pods Launched - Two Executor Pods were launched which was default
{code:java}
spark-pi-e608e7718f11cc69-driver   1/1     Running     0          10s
spark-pi-e608e7718f11cc69-exec-1   1/1     Running     0          5s
spark-pi-e608e7718f11cc69-exec-2   1/1     Running     0          5s{code}
Bug: Being able to not populate Hadoop classpath
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Apr/20 21:01
Affects Version/s: 2.4.5

Spark Yarn client will populate hadoop classpath from `yarn.application.classpath` and ``mapreduce.application.classpath`. However, for Spark with embedded hadoop build, it will result jar conflicts because spark distribution can contain different version of hadoop jars.

We are adding a new Yarn configuration to not populate hadoop classpath from  `yarn.application.classpath` and ``mapreduce.application.classpath`.
Bug: AM starts with initial fetched tokens in any attempt
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Apr/20 01:16
Affects Version/s: 3.0.0

The issue is only occurred in yarn-cluster mode.

Submitter will obtain delegation tokens for yarn-cluster mode, and add these credentials to the launch context. AM will be launched with these credentials, and AM and driver are able to leverage these tokens.

In Yarn cluster mode, driver is launched in AM, which in turn initializes token manager (while initializing SparkContext) and obtain delegation tokens (+ schedule to renew) if both principal and keytab are available.

That said, even we provide principal and keytab to run application with yarn-cluster mode, AM always starts with initial tokens from launch context until token manager runs and obtains delegation tokens.

So there's a "gap", and if user codes (driver) access to external system with delegation tokens (e.g. HDFS) before initializing SparkContext, it cannot leverage the tokens token manager will obtain. It will make the application fail if AM is killed "after" the initial tokens are expired and relaunched.
Bug: If shutdownhook is added with priority Integer.MIN_VALUE, it's supposed to be called the last, but it gets called before other positive priority shutdownhook
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Apr/20 03:27
Affects Version/s: 1.6.3

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala

Since shutdownHookManager use below method to do the comparison. 
override def compareTo(other: SparkShutdownHook): Int = {
    other.priority - priority
  }

Which will cause :
(Int)(25 - Integer.MIN_VALUE) < 0
Then the shutdownhook with Integer.Min_VALUE would not be called the last.
Bug: Expose hashFuncVersion property in HashingTF
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Apr/20 00:01
Affects Version/s: 3.0.0

Expose hashFuncVersion property in HashingTF

Some third-party library such as mleap need to access it.
See background description here:
https://github.com/combust/mleap/pull/665#issuecomment-621258623
Bug: Barrier stage can hang if only partial tasks launched
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/20 13:40
Affects Version/s: 2.4.0

The issue can be reproduced by following test:

 
{code:java}
initLocalClusterSparkContext(2)
val rdd0 = sc.parallelize(Seq(0, 1, 2, 3), 2)
val dep = new OneToOneDependency[Int](rdd0)
val rdd = new MyRDD(sc, 2, List(dep), Seq(Seq("executor_h_0"),Seq("executor_h_0")))
rdd.barrier().mapPartitions { iter =>
  BarrierTaskContext.get().barrier()
  iter
}.collect()
{code}
Bug: Spark sql cli should allow unescaped quote mark in quoted string
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Apr/20 10:31
Affects Version/s: 3.0.0

spark-sql> select "'";
spark-sql> select '"';

In Spark parser if we pass a text of `select "'";`, there will be ParserCancellationException, which will be handled by PredictionMode.LL. By dropping `;` correctly we can avoid that retry.
Bug: Fix test issue with table named `test` in hive/SQLQuerySuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Apr/20 08:18
Affects Version/s: 3.1.0

If we add ut in hive/SQLQuerySuite and use table named `test`. We may meet these exceptions.
{code:java}
 org.apache.spark.sql.AnalysisException: Inserting into an RDD-based table is not allowed.;;
[info] 'InsertIntoTable Project [_1#1403 AS key#1406, _2#1404 AS value#1407], Map(name -> Some(n1)), true, false
[info] +- Project [col1#3850]
[info]    +- LocalRelation [col1#3850]
{code}


{code:java}
org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException: Table or view 'test' already exists in database 'default';
[info]   at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$doCreateTable$1.apply$mcV$sp(HiveExternalCatalog.scala:226)
[info]   at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$doCreateTable$1.apply(HiveExternalCatalog.scala:216)
[info]   at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$doCreateTable$1.apply(HiveExternalCatalog.scala:216)
[info]   at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
[info]   at org.apache.spark.sql.hive.HiveExternalCatalog.doCreateTable(HiveExternalCatalog.scala:216)
{code}
Bug: Returning complex types in Pandas UDF
Status: Open | Priority: Major | Resolution: 
Created: 07/Apr/20 10:34
Affects Version/s: 3.0.0

I would like to return a complex type in an GROUPED_AGG operation:

{code:python}
window_overlap_schema = t.StructType([
     t.StructField("counts", t.ArrayType(t.LongType())),
     t.StructField("starts", t.ArrayType(t.LongType())),
     t.StructField("ends", t.ArrayType(t.LongType())),
])

@f.pandas_udf(window_overlap_schema, f.PandasUDFType.GROUPED_AGG)
def spark_window_overlap([...]):
    [...]
{code}


However, I get the following error when trying to run this:

{code:python}
NotImplementedError: Invalid returnType with grouped aggregate Pandas UDFs: StructType(List(StructField(counts,ArrayType(LongType,true),true),StructField(starts,ArrayType(LongType,true),true),StructField(ends,ArrayType(LongType,true),true))) is not supported
{code}
Bug: When using the insert overwrite directory syntax, if the target path is an existing file, the final run result is incorrect
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Apr/20 13:04
Affects Version/s: 2.4.5

When using the insert overwrite directory syntax, if the target path is an existing file, the final operation result is incorrect.
At present, Spark will not delete the existing files. After the calculation is completed, one of the result files will be renamed to the result path.
This is different from hive's behavior. Hive will delete the existing target file.
Bug: collect_set() of BinaryType returns duplicate elements
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Apr/20 19:16
Affects Version/s: 2.0.2

The collect_set() aggregate function should produce a set of distinct elements. When the column argument's type is BinayType this is not the case.

 

Example:

{{import org.apache.spark.sql.functions._}}
 {{import org.apache.spark.sql.expressions.Window}}

{{case class R(id: String, value: String, bytes: Array[Byte])}}
 {{def makeR(id: String, value: String) = R(id, value, value.getBytes)}}
 {{val df = Seq(makeR("a", "dog"), makeR("a", "cat"), makeR("a", "cat"), makeR("b", "fish")).toDF()}}

 

{{// In the example below "bytesSet" erroneously has duplicates but "stringSet" does not (as expected).}}

{{df.agg(collect_set('value) as "stringSet", collect_set('bytes) as "byteSet").show(truncate=false)}}

 

{{// The same problem is displayed when using window functions.}}
 {{val win = Window.partitionBy('id).rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)}}
 {{val result = df.select(}}
  collect_set('value).over(win) as "stringSet",
  collect_set('bytes).over(win) as "bytesSet"
 {{)}}
 {{.select('stringSet, 'bytesSet, size('stringSet) as "stringSetSize", size('bytesSet) as "bytesSetSize")}}
 {{.show()}}
Bug: Add new configuration to make sure temporary directory cleaned
Status: In Progress | Priority: Major | Resolution: 
Created: 04/Apr/20 15:13
Affects Version/s: 3.1.0

In InsertIntoHiveTable and InsertIntoHiveDirCommand, we use deleteExternalTmpPath to clean temporary directories after Job committed and cancel deleteOnExit if succeeded. But sometimes (e.g., when speculative task is enabled), temporary directories may be left uncleaned. This is happened if there are still some tasks running after we called deleteExternalTmpPath. Thus it maybe necessary to keep deleteOnExit, even if temporary directory has already deleted, to make sure the temporary directories cleaned.
Bug: Add try function support
Status: In Progress | Priority: Major | Resolution: 
Created: 03/Apr/20 07:26
Affects Version/s: 3.1.0


{code:java}
Evaluate an expression and handle certain types of execution errors by returning NULL.
In cases where it is preferable that queries produce NULL instead of failing when corrupt or invalid data is encountered, the TRY function may be useful especially when ANSI mode is on and the users need null-tolerant on certain columns or outputs.

AnalysisExceptions will not handle by this, typically errors handled by TRY function are: 

      * Division by zero,
      * Invalid casting,
      * Numeric value out of range,
      * e.t.c
{code}
Bug: Reading from S3 (Structured Streaming Bucket) Fails after Compaction
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 28/Apr/20 19:09
Affects Version/s: 2.4.5

I have a S3 bucket which has data streamed (Parquet format) to it by Spark Structured Streaming Framework from Kafka. Periodically I try to run compaction on this bucket (a separate Spark Job), and on successful compaction delete the non compacted (parquet) files. After which I am getting following error on Spark jobs which read from that bucket:
 *Caused by: java.io.FileNotFoundException: No such file or directory: s3a://spark-kafka-poc/intermediate/part-00000-05ff7893-8a13-4dcd-aeed-3f0d4b5d1691-c000.gz.parquet*

How do we run *_c_ompaction on Structured Streaming S3 bucket_s*. Also I need to delete the un-compacted files after successful compaction to save space.
Bug: Flaky test suite org.apache.spark.sql.hive.thriftserver.CliSuite
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 24/Apr/20 14:18
Affects Version/s: 3.1.0

The test org.apache.spark.sql.hive.thriftserver.CliSuite fails very often, for example:
* https://github.com/apache/spark/pull/28328#issuecomment-618992335
The error message:
{code}
org.apache.spark.sql.hive.thriftserver.CliSuite.SPARK-11188 Analysis error reporting
Caused by: sbt.ForkMain$ForkError: java.lang.RuntimeException: Failed with error line 'Exception in thread "main" org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;'
	at org.apache.spark.sql.hive.thriftserver.CliSuite.$anonfun$runCliWithin$4(CliSuite.scala:138)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.sql.hive.thriftserver.CliSuite.captureOutput$1(CliSuite.scala:135)
	at org.apache.spark.sql.hive.thriftserver.CliSuite.$anonfun$runCliWithin$6(CliSuite.scala:152)
	at org.apache.spark.sql.hive.thriftserver.CliSuite.$anonfun$runCliWithin$6$adapted(CliSuite.scala:152)
	at scala.sys.process.BasicIO$.readFully$1(BasicIO.scala:188)
	at scala.sys.process.BasicIO$.$anonfun$processFully$1$adapted(BasicIO.scala:192)
	at org.apache.spark.sql.test.ProcessTestUtils$ProcessOutputCapturer.run(ProcessTestUtils.scala:30)
{code}
* https://github.com/apache/spark/pull/28261#issuecomment-618950225
* https://github.com/apache/spark/pull/28261#issuecomment-618950225
* https://github.com/apache/spark/pull/27617#issuecomment-614318644
Bug: Unable to return Hive data into Spark via Hive JDBC driver Caused by:  org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED
Status: Closed | Priority: Major | Resolution: Fixed
Created: 27/Apr/20 13:41
Affects Version/s: 2.3.1

I'm trying to fetch back data in Spark SQL using a JDBC connection to Hive. Unfortunately, when I try to query data that resides in every column I get the following error:

Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException [Error 10004]: Line 1:7 Invalid table alias or column reference 'test.aname': (possible column names are: aname, score, banji)
 at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
 at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:199)

1)  On Hive create a simple table,its name is "test"，it have three column(aname,score,banji),their type both are "String"

2)important code:

object HiveDialect extends JdbcDialect

{ override def canHandle(url: String): Boolean = url.startsWith("jdbc:hive2")|| url.contains("hive2")                                                                                                                    override def quoteIdentifier(colName: String): String = s"`$colName`" }

-------------------------------------------------------------------

object callOffRun {
 def main(args: Array[String]): Unit =

{ val spark = SparkSession.builder().enableHiveSupport().getOrCreate() JdbcDialects.registerDialect(HiveDialect)                                                                                 val props = new Properties()                                                                                                          props.put("driver","org.apache.hive.jdbc.HiveDriver")                                  props.put("user","username")                                                         props.put("password","password")                                                           props.put("fetchsize","20")                                                                                                     val table=spark.read .jdbc("jdbc:hive2://xxxxxxxx:10000","test",props)                          table.show() }

}

3)spark-submit ,After running,it have error

Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException [Error 10004]: Line 1:7 Invalid table alias or column reference 'test.aname': (possible column names are: aname, score, banji)

4)table.count() have result 

5) I try some method to print result,They all reported the same error