av9ash/GitBugs huggingface dataset:
Bug: KinesisInitialPosition interface should be public
Status: Open | Priority: Minor | Resolution: 
Created: 16/Aug/21 08:29
Affects Version/s: 3.1.2

In `org.apache.spark.streaming.kinesis` there's a Java interface `KinesisInitialPosition` - for some reason it's not public, so can't be used.

The reason it needs to be used is simple: it is the parent of `KinesisInitialPositions.Latest` and `KinesisInitialPositions.TrimHorizon`. Without using it, it's impossible to pass instances of `Latest` and `TrimHorizon` around without calling them by name.

The workaround is to use the enum `InitialPositionInStream` instead, and a factory method `fromKinesisInitialPosition(..)` - but it doesn't seem like the correct pattern, because it belongs to Amazon Kinesis Client library and not Spark (it's currently the only import in my class which is outside of `org.apache.spark`).

The change is trivial, so it's mainly a matter of why *not* to do it.

KinesisInitialPositions: https://github.com/apache/spark/blob/master/external/kinesis-asl/src/main/java/org/apache/spark/streaming/kinesis/KinesisInitialPositions.java
KinesisInitialPosition: https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/InitialPositionInStream.java
Bug: Improve consistency processing floating point special literals
Status: In Progress | Priority: Minor | Resolution: 
Created: 08/Aug/21 22:35
Affects Version/s: 3.3.0

Special literal in floating point are not consistent between cast and json expressions

 
{code:java}
scala> spark.sql("SELECT CAST('+Inf' as Double)").show
+--------------------+                                                        
|CAST(+Inf AS DOUBLE)|
+--------------------+
|            Infinity|
+--------------------+
{code}
 
{code:java}
scala> val schema =  StructType(StructField("a", DoubleType) :: Nil)

scala> Seq("""{"a" : "+Inf"}""").toDF("col1").select(from_json(col("col1"),schema)).show

+---------------+
|from_json(col1)|
+---------------+
|         {null}|
+---------------+

scala> Seq("""{"a" : "+Inf"}""").toDF("col").withColumn("col", from_json(col("col"), StructType.fromDDL("a DOUBLE"))).write.json("/tmp/jsontests12345")
scala> spark.read.schema(StructType(Seq(StructField("col",schema)))).json("/tmp/jsontests12345").show

+------+
|   col|
+------+
|{null}|
+------+

{code}
Bug: Add RowToColumnConverter for BinaryType
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Aug/21 00:51
Affects Version/s: 3.3.0

currently, we have RowToColumnConverter for all data types except BinaryType


{code:java}
  private def getConverterForType(dataType: DataType, nullable: Boolean): TypeConverter = {
    val core = dataType match {
      case BooleanType => BooleanConverter
      case ByteType => ByteConverter
      case ShortType => ShortConverter
      case IntegerType | DateType => IntConverter
      case FloatType => FloatConverter
      case LongType | TimestampType => LongConverter
      case DoubleType => DoubleConverter
      case StringType => StringConverter
      case CalendarIntervalType => CalendarConverter
      case at: ArrayType => ArrayConverter(getConverterForType(at.elementType, at.containsNull))
      case st: StructType => new StructConverter(st.fields.map(
        (f) => getConverterForType(f.dataType, f.nullable)))
      case dt: DecimalType => new DecimalConverter(dt)
      case mt: MapType => MapConverter(getConverterForType(mt.keyType, nullable = false),
        getConverterForType(mt.valueType, mt.valueContainsNull))
      case unknown => throw QueryExecutionErrors.unsupportedDataTypeError(unknown.toString)
    }
{code}
Bug: MinHashLSH.approxSimilarityJoin should not required inputCol if output exist
Status: Open | Priority: Minor | Resolution: 
Created: 09/Aug/21 11:02
Affects Version/s: 3.1.1

Refer to documents and example code in MinHashLSH 
 [https://spark.apache.org/docs/latest/ml-features#minhash-for-jaccard-distance]

The example written that:

We could avoid computing hashes by passing in the already-transformed dataset, e.g. `model.approxSimilarityJoin(transformedA, transformedB, 0.6)`

However, inputCol still required in transformedA and transformedB even if they already have outputCol.

A code that should work but it doesn't

 

 
{code:java}
from pyspark.ml.feature import MinHashLSH
 from pyspark.ml.linalg import Vectors
 from pyspark.sql.functions import col
dataA = [(0, Vectors.sparse(6, [0, 1, 2], [1.0, 1.0, 1.0]),),
 (1, Vectors.sparse(6, [2, 3, 4], [1.0, 1.0, 1.0]),),
 (2, Vectors.sparse(6, [0, 2, 4], [1.0, 1.0, 1.0]),)]
 dfA = spark.createDataFrame(dataA, ["id", "features"])
dataB = [(3, Vectors.sparse(6, [1, 3, 5], [1.0, 1.0, 1.0]),),
 (4, Vectors.sparse(6, [2, 3, 5], [1.0, 1.0, 1.0]),),
 (5, Vectors.sparse(6, [1, 2, 4], [1.0, 1.0, 1.0]),)]
 dfB = spark.createDataFrame(dataB, ["id", "features"])
key = Vectors.sparse(6, [1, 3], [1.0, 1.0])
mh = MinHashLSH(inputCol="features", outputCol="hashes", numHashTables=5)
 model = mh.fit(dfA)
transformedA = model.transform(dfA).select("id", "hashes")
 transformedB = model.transform(dfB).select("id", "hashes")
model.approxSimilarityJoin(transformedA, transformedB, 0.6, distCol="JaccardDistance")\
 .select(col("datasetA.id").alias("idA"),
 col("datasetB.id").alias("idB"),
 col("JaccardDistance")).show()
{code}
As in the code I give, I discard columns `features` but keep column `hashes` which is output data. 
 approxSimilarityJoin should only work on `hashes` (the outputCol), which is exist and ignore the lack of `features` (the inputCol).

Be able to transform the data beforehand and remove inputCol can make input data much smaller and prevent confusion about the tip "_We could avoid computing hashes by passing in the already-transformed dataset_".
Bug: Not push down partition filter to ORCScan for DSv2
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Aug/21 01:13
Affects Version/s: 3.3.0

Seems to me that partition filter is only used for partition pruning and shouldn't be pushed down to ORCScan. We don't push down partition filter to ORCScan in DSv1, and we don't push down partition filter for parquet in both DSv1 and DSv2.
Bug: Add resource profile to Spark pods labels
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 04/Aug/21 17:57
Affects Version/s: 3.3.0

In SPARK-36058 I added SPARK_RESOURCE_PROFILE_ID_LABEL, dongjoon suggested that this could be a useful improvement on its own (e.g. for situations like wanting to inspect or collect metrics by resource profile).
Bug: Upgrade json4s to 4.0.3
Status: In Progress | Priority: Minor | Resolution: 
Created: 04/Aug/21 04:52
Affects Version/s: 3.3.0

json4s v4 was released and 173 bug fixes and improvements are applied after v3.7.0-M11 which Spark currently depends on.
Some improvements seem to be about Scala 2.13 and Scala 3.
Bug: Revert the change that accepts negative mapId in ShuffleBlockId
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Aug/21 03:00
Affects Version/s: 3.2.0

With SPARK-32922, we added a change that {{ShuffleBlockId}} can have a negative mapId. This was to support push-based shuffle where {{-1}} as mapId indicated a push-merged block. However with SPARK-32923, a different type of {{BlockId}} was introduced - {{ShuffleMergedId}}, but reverting the change to {{ShuffleBlockId}} was missed.
Bug: Remove noisy footer from the summary table for metrics
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Aug/21 13:05
Affects Version/s: 3.3.0

In the WebUI, some tables are implemented using DataTables (https://datatables.net/).
By default, tables created using DataTables shows footer which says `Showing x to y of z entries`, which is helpful for some tables if table entries can grow
But the summary table for metrics in StagePage cannot grow so it's a little bit noisy.
Bug: Use Type match to simplify CatalystTypeConverter.toCatalyst
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 12/Aug/21 11:10
Affects Version/s: 3.3.0

CatalystTypeConverter.toCatalyst uses the following methods for type conversion
{code:java}
if (maybeScalaValue.isInstanceOf[Option[ScalaInputType]]) {
  val opt = maybeScalaValue.asInstanceOf[Option[ScalaInputType]]
{code}
 

 can use Type match to simplify this pattern
Bug: Optimise speed and memory with Pyspark when create DataFrame (with patch)
Status: Open | Priority: Trivial | Resolution: 
Created: 26/Aug/21 11:27
Affects Version/s: 3.1.2

The Python method {{SparkSession._createFromLocal()}} start to the data, and create a list if it's not an instance of list. But it is necessary only if the scheme is not present.
{quote}# make sure data could consumed multiple times
 if not isinstance(data, list):
  data = list(data)
{quote}
If you use {{createDataFrame(data=_a_generator_,...)}}, all the datas were save in memory in a list, then convert to a row in memory, then convert to buffer in pickle format, etc.

Two lists were present at the same time in memory. The list created by _createFromLocal() and the list created later with
{quote}# convert python objects to sql data
data = [schema.toInternal(row) for row in data]
{quote}
The purpose of using a generator is to reduce the memory footprint when the data are dynamically build.
{quote}def _createFromLocal(self, data, schema):
  """
  Create an RDD for DataFrame from a list or pandas.DataFrame, returns
  the RDD and schema.
  """

  if schema is None or isinstance(schema, (list, tuple)):
    *# make sure data could consumed multiple times*
    *if inspect.isgeneratorfunction(data):* 
      *data = list(data)*
    struct = self._inferSchemaFromList(data, names=schema)
    converter = _create_converter(struct)
    data = map(converter, data)
    if isinstance(schema, (list, tuple)):
      for i, name in enumerate(schema):
        struct.fields[i].name = name
        struct.names[i] = name
      schema = struct

    elif not isinstance(schema, StructType):
      raise TypeError("schema should be StructType or list or None, but got: %s" % schema)

  # convert python objects to sql data
  data = [schema.toInternal(row) for row in data]
  return self._sc.parallelize(data), schema{quote}
Then, it is interesting to use a generator.

 
{quote}The patch:

diff --git a/python/pyspark/sql/session.py b/python/pyspark/sql/session.py
index 57c680fd04..0dba590451 100644
--- a/python/pyspark/sql/session.py
+++ b/python/pyspark/sql/session.py
@@ -15,6 +15,7 @@
 # limitations under the License.
 #
 
+import inspect
 import sys
 import warnings
 from functools import reduce
@@ -504,11 +505,11 @@ class SparkSession(SparkConversionMixin):
 Create an RDD for DataFrame from a list or pandas.DataFrame, returns
 the RDD and schema.
 """
- # make sure data could consumed multiple times
- if not isinstance(data, list):
- data = list(data)
 
 if schema is None or isinstance(schema, (list, tuple)):
+ # make sure data could consumed multiple times
+ if inspect.isgeneratorfunction(data): # PPR
+ data = list(data)
 struct = self._inferSchemaFromList(data, names=schema)
 converter = _create_converter(struct)
 data = map(converter, data)
{quote}
Bug: Update the MAVEN_OPTS in Spark build docs
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 22/Aug/21 19:06
Affects Version/s: 3.2.0

As Jacek Laskowski pointed out in the dev list, there is StackOverflowError if compiling Spark with the current MAVEN_OPTS in Spark documentation.
We should update it with `-Xss64m` to avoid it.
Bug: Ivy skips looking for source and doc pom
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 08/Aug/21 12:02
Affects Version/s: 3.3.0

Because SPARK-35863 Upgrade Ivy to 2.5.0, it supports skip searching the source and doc pom, but the remote repo will still be queried at present.

 

org.apache.ivy.plugins.parser.m2.PomModuleDescriptorParser#addSourcesAndJavadocArtifactsIfPresent
{code:java}
boolean sourcesLookup = !"false"
        .equals(ivySettings.getVariable("ivy.maven.lookup.sources"));
boolean javadocLookup = !"false"
        .equals(ivySettings.getVariable("ivy.maven.lookup.javadoc"));
if (!sourcesLookup && !javadocLookup) {
    Message.debug("Sources and javadocs lookup disabled");
    return;
}
{code}
Bug: Missing spark.redaction.string.regex property
Status: In Progress | Priority: Trivial | Resolution: 
Created: 13/Aug/21 22:30
Affects Version/s: 3.1.2

The property *spark.redaction.string.regex* is missing in [Runtime Environment|https://spark.apache.org/docs/3.1.2/configuration.html#runtime-environment] properties table but referred by *spark.sql.redaction.string.regex* description as its default value
Bug: Propagation cause when UDF reflection fails
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 20/Aug/21 07:12
Affects Version/s: 3.1.2

Now when UDF reflection fails, InvocationTargetException is thrown, but it is not a specific exception.
{code:java}
Error in query: No handler for Hive UDF 'XXX': java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
{code}
Bug: modify exit executor log logic
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 12/Aug/21 06:09
Affects Version/s: 3.1.2

When exit executor with system code 0, coarseGrainedExecutorBackend will print ERROR log.
That doesn't make sense, because it seems to me that executor's normal decommission is not caused by an exception.

So modify exit-executor log logic.Select info type or Error type by judging argument code.
Bug: Improve SQL syntax for MERGE
Status: Open | Priority: Trivial | Resolution: 
Created: 10/Aug/21 18:42
Affects Version/s: 3.1.2

Existing SQL syntax for *MEGRE* (see Delta Lake examples [here|https://docs.delta.io/latest/delta-update.html#upsert-into-a-table-using-merge] and [here|https://docs.microsoft.com/en-us/azure/databricks/spark/latest/spark-sql/language-manual/delta-merge-into]) could be improved by adding an alternative for {{<merge_condition>}}

*Main assumption*
 In common cases target and source tables have the same column names used in {{<merge_condition>}} as merge keys, for example:
{code:sql}
ON target.key1 = source.key1 AND target.key2 = source.key2{code}
It would be more convenient to use a syntax similar to:
{code:sql}
ON COLUMNS (key1, key2)
-- or
ON MATCHING (key1, key2)
{code}
The same approach is used for [JOIN|https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-join.html] where {{join_criteria}} syntax is
{code:sql}
ON boolean_expression | USING ( column_name [ , ... ] )
{code}
*Improvement proposal*
 Syntax
{code:sql}
MERGE INTO target_table_identifier [AS target_alias]
USING source_table_identifier [<time_travel_version>] [AS source_alias]
ON { <merge_condition> | COLUMNS ( column_name [ , ... ] ) }
[ WHEN MATCHED [ AND <condition> ] THEN <matched_action> ]
[ WHEN MATCHED [ AND <condition> ] THEN <matched_action> ]
[ WHEN NOT MATCHED [ AND <condition> ]  THEN <not_matched_action> ]
{code}
Example
{code:sql}
MERGE INTO target
USING source
ON COLUMNS (key1, key2)
WHEN MATCHED THEN
    UPDATE SET *
WHEN NOT MATCHED THEN
    INSERT *
{code}
Bug: Replace SessionState.close with SessionState.detachSession
Status: In Progress | Priority: Trivial | Resolution: 
Created: 03/Aug/21 08:52
Affects Version/s: 3.2.0

https://issues.apache.org/jira/browse/SPARK-35286 replace SessionState.start with SessionState.setCurrentSessionState, but SessionState.close will create a HiveMetaStoreClient , connect to the Hive Meta Store Server, and then load all functions
Bug: Join loses records for cached datasets
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 22/Sep/23 17:13
Affects Version/s: 3.4.1

we observed this issue on spark 3.4.1 but it is also present on 3.5.0. it is not present on spark 3.3.1.

it only shows up in distributed environment. i cannot replicate in unit test. however i did get it to show up on hadoop cluster, kubernetes, and on databricks 13.3

the issue is that records are dropped when two cached dataframes are joined. it seems in spark 3.4.1 in queryplan some Exchanges are dropped as an optimization while in spark 3.3.1 these Exhanges are still present. it seems to be an issue with AQE with canChangeCachedPlanOutputPartitioning=true.

to reproduce on distributed cluster these settings needed are:
{code:java}
spark.sql.adaptive.advisoryPartitionSizeInBytes 33554432
spark.sql.adaptive.coalescePartitions.parallelismFirst false
spark.sql.adaptive.enabled true
spark.sql.optimizer.canChangeCachedPlanOutputPartitioning true {code}
code using scala to reproduce is:
{code:java}
import java.util.UUID
import org.apache.spark.sql.functions.col

import spark.implicits._

val data = (1 to 1000000).toDS().map(i => UUID.randomUUID().toString).persist()

val left = data.map(k => (k, 1))
val right = data.map(k => (k, k)) // if i change this to k => (k, 1) it works!
println("number of left " + left.count())
println("number of right " + right.count())
println("number of (left join right) " +
  left.toDF("key", "value1").join(right.toDF("key", "value2"), "key").count()
)

val left1 = left
  .toDF("key", "value1")
  .repartition(col("key")) // comment out this line to make it work
  .persist()
println("number of left1 " + left1.count())

val right1 = right
  .toDF("key", "value2")
  .repartition(col("key")) // comment out this line to make it work
  .persist()
println("number of right1 " + right1.count())

println("number of (left1 join right1) " +  left1.join(right1, "key").count()) // this gives incorrect result{code}
this produces the following output:
{code:java}
number of left 1000000
number of right 1000000
number of (left join right) 1000000
number of left1 1000000
number of right1 1000000
number of (left1 join right1) 859531 {code}
note that the last number (the incorrect one) actually varies depending on settings and cluster size etc.
Bug: SPARK-43183 broke various tests in 3rd party streaming data sources
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 01/Sep/23 06:40
Affects Version/s: 3.5.0

SPARK-43183 made a change regarding behavior to the StreamingQueryListener as well as StreamingQuery API, while the intention was more about introducing the change in the former one.

I just got some reports from 3rd party data sources that they encountered innocent test failures during upgrade to 3.5.0 (RC), and it looks like they are mostly concerned about the behavioral change for StreamingQuery API. (It's lot less convenient for developers to rely on streaming query listener for testing.)

That said, to avoid 3rd party ecosystem be bugging with new behavior, we should probably revert the behavior for StreamingQuery API at least.
Bug: Optimize hive patition filter when the comparision dataType not match
Status: Open | Priority: Critical | Resolution: 
Created: 30/Sep/23 09:47
Affects Version/s: 3.1.1

Suppose we have a partitioned table `table_pt` with partition colum `dt` which is StringType and the table metadata is managed by Hive Metastore, if we filter partition by dt = '123', this filter can be pushed down to data source directly, but if the filter condition is number, e.g. dt = 123, Spark will not known which partition should be pushed down. Thus in the process of physical plan optimization, Spark will pull all of that table's partition meta data to client side, to decide which partition filter should be push down to the data source. This is poor of performance if the table has thousands of partitions and increasing the risk of hive metastore oom.
Bug: Pyspark DecisionTreeClassifier: results and tree structure in spark3 very different from that of the spark2 version on the same data and with the same hyperparameters.
Status: Open | Priority: Critical | Resolution: 
Created: 13/Sep/23 13:39
Affects Version/s: 3.0.0

Hello,
I have an engine running on spark2 using a DecisionTreeClassifier model using the CrossValidator. 

 
{code:java}
dt  = DecisionTreeClassifier(maxBins=10000, seed=0)   
cv_dt_evaluator = BinaryClassificationEvaluator(
            metricName="", 
            rawPredictionCol="probability")

# Create param grid and cross validator for model selection
dt_grid = ParamGridBuilder()\
            .addGrid(
                dt.minInstancesPerNode, [100]
        )\
            .addGrid(
                dt.maxDepth, [10]
        )\
            .build()
cv = CrossValidator(
            estimator=dt, estimatorParamMaps=dt_grid, evaluator=cv_dt_evaluator,
            parallelism=4
            numFolds=4
        ){code}
 

I want to {*}migrate from spark2  to spark3{*}. I've run *DecisionTreeClassifier* on the same data with the same parameter values. But unfortunately my results are {*}completely different, especially in terms of tree structure{*}. I have trees with less depth and fewer splits on spark3. I've tried to read the documentation but I haven't found an answer to my question.

 

Can you help me find a solution to this problem?

Thanks in advance for your help
Bug: Remove System.setSecurityManager usage
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/23 07:11
Affects Version/s: 4.0.0

{code}
$ java -version
openjdk version "21-ea" 2023-09-19
OpenJDK Runtime Environment (build 21-ea+32-2482)
OpenJDK 64-Bit Server VM (build 21-ea+32-2482, mixed mode, sharing)
max spark-3.5.0-bin-hadoop3:$ bin/spark-sql --help
...
CLI options:
Exception in thread "main" java.lang.UnsupportedOperationException: The Security Manager is deprecated and will be removed in a future release
	at java.base/java.lang.System.setSecurityManager(System.java:429)
	at org.apache.spark.deploy.SparkSubmitArguments.getSqlShellOptions(SparkSubmitArguments.scala:623)
	at org.apache.spark.deploy.SparkSubmitArguments.$anonfun$printUsageAndExit$6(SparkSubmitArguments.scala:593)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.logInfo(SparkSubmit.scala:1112)
	at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.logInfo(SparkSubmit.scala:1104)
	at org.apache.spark.deploy.SparkSubmitArguments.printUsageAndExit(SparkSubmitArguments.scala:593)
	at org.apache.spark.deploy.SparkSubmitArguments.handle(SparkSubmitArguments.scala:448)
	at org.apache.spark.launcher.SparkSubmitOptionParser.parse(SparkSubmitOptionParser.java:174)
	at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:111)
	at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:1103)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:1103)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
Bug: Minimizing calls to HiveMetaStore layer for getting partitions,  when tables are repeated
Status: Open | Priority: Major | Resolution: 
Created: 28/Sep/23 21:00
Affects Version/s: 3.5.0

In the rule PruneFileSourcePartitions where the CatalogFileIndex gets converted to InMemoryFileIndex,  the HMS calls can get very expensive if :
1) The translated filter string for push down to HMS layer becomes empty ,  resulting in fetching of all partitions and same table is referenced multiple times in the query. 
2) Or just in case same table is referenced multiple times in the query with different partition filters.
In such cases current code would result in multiple calls to HMS layer. 
This can be avoided by grouping the tables based on CatalogFileIndex and passing a common minimum filter ( filter1 || filter2) and getting a base PrunedInmemoryFileIndex which can become a basis for each of the specific table.

Opened following PR for ticket:
[SPARK-45373-PR|https://github.com/apache/spark/pull/49155]