av9ash/GitBugs huggingface dataset:
Bug: Enable DataFramePlotMatplotlibTests.test_area_plot_stacked_false for pandas 2.0.0.
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 22/May/23 06:38
Affects Version/s: 3.5.0

Enable DataFramePlotMatplotlibTests.test_area_plot_stacked_false for pandas 2.0.0.
Bug: Enable ExponentialMovingLike.mean with Spark Connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/May/23 05:53
Affects Version/s: 3.5.0

Repro:
{code:java}
pser = pd.Series([1, 2, 3], index=np.random.rand(3), name="a")
psser = ps.from_pandas(pser)
psser.ewm(com=0.2).mean() {code}
Bug: LEFT JOIN is treated as INNER JOIN when being in a middle of double join
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 02/May/23 12:45
Affects Version/s: 3.4.0

Consider query like

 
{code:java}
SELECT ss_item_sk
       FROM   store_sales
              LEFT OUTER JOIN store_returns
                           ON ( sr_item_sk = ss_item_sk ),
              reason
       WHERE  sr_reason_sk = r_reason_sk
              AND r_reason_desc = 'reason 38'{code}
 

Spark generates following plan:

 
{code:java}
AdaptiveSparkPlan isFinalPlan=false
+- Project [ss_item_sk#2]
   +- BroadcastHashJoin [sr_reason_sk#458], [r_reason_sk#734], Inner, BuildRight, false
      :- Project [ss_item_sk#2, sr_reason_sk#458]
      :  +- BroadcastHashJoin [ss_item_sk#2], [sr_item_sk#452], Inner, BuildRight, false
      :     :- FileScan parquet [ss_item_sk#2] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/home/leonid/tpcds-spark-data-no-padding/store_sales], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<ss_item_sk:int>
      :     +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [id=#7227]
      :        +- Filter (isnotnull(sr_item_sk#452) AND isnotnull(sr_reason_sk#458))
      :           +- FileScan parquet [sr_item_sk#452,sr_reason_sk#458] Batched: true, DataFilters: [isnotnull(sr_item_sk#452), isnotnull(sr_reason_sk#458)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/home/leonid/tpcds-spark-data-no-padding/store_returns], PartitionFilters: [], PushedFilters: [IsNotNull(sr_item_sk), IsNotNull(sr_reason_sk)], ReadSchema: struct<sr_item_sk:int,sr_reason_sk:int>
      +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [id=#7231]
         +- Project [r_reason_sk#734]
            +- Filter ((isnotnull(r_reason_desc#736) AND (r_reason_desc#736 = reason 38)) AND isnotnull(r_reason_sk#734))
               +- FileScan parquet [r_reason_sk#734,r_reason_desc#736] Batched: true, DataFilters: [isnotnull(r_reason_desc#736), (r_reason_desc#736 = reason 38), isnotnull(r_reason_sk#734)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/home/leonid/tpcds-spark-data-no-padding/reason], PartitionFilters: [], PushedFilters: [IsNotNull(r_reason_desc), EqualTo(r_reason_desc,reason 38), IsNotNull(r_reason_sk)], ReadSchema: struct<r_reason_sk:int,r_reason_desc:string>
{code}
Bug: Support registration of an Arrow Python UDF 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/May/23 21:09
Affects Version/s: 3.5.0

Currently, when users register an Arrow-optimized Python UDF, it will be registered as a pickled Python UDF and thus, executed without Arrow optimization. 
We should support Arrow-optimized Python UDFs registration and execute them with Arrow optimization.
Bug: Introduce `SQL_ARROW_BATCHED_UDF` EvalType for Arrow Python UDFs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/May/23 18:34
Affects Version/s: 3.5.0

We are about to improve nested non-atomic input/output support of an Arrow-optimized Python UDF.

However, currently, it shares the same EvalType with a pickled Python UDF, but the same implementation with a Pandas UDF.

Introducing an EvalType enables isolating the changes to Arrow-optimized Python UDFs.
Bug: Incorrect result of MINUS in spark sql.
Status: Open | Priority: Major | Resolution: 
Created: 23/May/23 14:35
Affects Version/s: 2.4.8

sql("""
with va as (
  select '123' id, 'a' name
   union all
  select '123' id, 'b' name
)
select '123' id, 'a' name from va t where t.name = 'a'
 minus 
select '123' id, 'a' name from va s where s.name = 'b'
""").show
+---+----+
| id|name|
+---+----+
|123|   a|
+---+----+

which is expected to be empty result set.
Bug: Enable DataFrameSlowParityTests.test_udt
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 22/May/23 05:22
Affects Version/s: 3.5.0

Repro:
{code:java}
sparse_values = {0: 0.1, 1: 1.1}
sparse_vector = SparseVector(len(sparse_values), sparse_values)
pdf = pd.DataFrame({"a": [sparse_vector], "b": [10]})
psdf = ps.from_pandas(pdf) {code}
Bug: Allow parameter markers in DDL (again)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/May/23 17:02
Affects Version/s: 3.5.0

When we introduced parameter markers initially, we allowed them n any SQL statement.
Subsequently, we have limited their use to DML and queries because that aligns better with the industry and we saw no immediate use for broader support.

However, we have introduced the IDENTIFIER() clause, which allows templating table-, and function-identifiers in DDL statements. 
To exploit this, we need parameter markers as argument:

spark.sql("CREATE TABLE IDENTIFIER(:tableName) (c1 INT)", args = Map("tableName" -> "mytable")
Bug: Partition filter condition should pushed down to metastore query if it is equivalence Predicate
Status: Resolved | Priority: Major | Resolution: Workaround
Created: 29/May/23 14:01
Affects Version/s: 3.4.0

Typically, hive partition fields are created as string types.
{code:java}
```
CREATE TABLE if not exists test_tb (
id int
)
PARTITIONED BY (dt string)
```{code}
However, cast data conversions are often introduced inadvertently during use. For example
{code:java}
```
select * from test_tb where dt=20230505;
```{code}

it will prevent the condition `dt=20230505` from being pushed down into the metastore , because `20230505` is an IntegralType. And resulting in a request for all partitions. However, in the case of equivalent predicates, partition field pushdown should be supported.

This can affect execution performance in cases where the data table has very many partitions.
Bug: Enable DataFrameSlowParityTests.test_eval
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/May/23 05:15
Affects Version/s: 3.5.0

Repro:
{code:java}
pdf = pd.DataFrame({"A": range(1, 6), "B": range(10, 0, -2)})
psdf = ps.from_pandas(pdf)
pdf.eval("B = A + B // (100 + 200) * (500 - B) - 10.5")
psdf.eval("B = A + B // (100 + 200) * (500 - B) - 10.5") {code}
Bug: Spark connect: improve handling of long running API - awaitTermination
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/May/23 22:10
Affects Version/s: 3.5.0

Currently, long running APIs like awaitTermination() keep running on the server even if the client has disconnected. We need to improve that. The implementation should periodically check if the client side is disconnected and release the resources.
Bug: Add working directory into classpath on the driver in K8S cluster mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/May/23 14:29
Affects Version/s: 3.4.0

In Yarn cluster modes, the passed files/jars are able to be accessed in the classloader. Looks like this is not the case in Kubernetes cluster mode.

After SPARK-33782, it places spark.files, spark.jars and spark.files under the current working directory on the driver in K8S cluster mode. but the spark.files and spark.jars seems are not accessible by the classloader.

 

we need to add the current working directory into classpath.
Bug: Add API `copyLocalFileToHadoopFS`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/May/23 05:01
Affects Version/s: 3.5.0

In new distributed spark ML module (designed to support spark connect and support local inference)

We need to save ML model to hadoop file system using custom binary file format, the reason is:
 * We often submit a spark application to spark cluster for running the training model job, we need to save trained model to hadoop file system before the spark application completes.
 * But we want to support local model inference, that means if we save the model by current spark DataFrame writer (e.g. parquet format), when loading model we have to rely on the spark service. But we hope we can load model without spark service. So we want the model being saved as the original binary format that our ML code can handle.

 

So we need to add an API like `copyLocalFileToHadoopFS`,

The implementation of `copyLocalFileToHadoopFS` could be:

 

(1) call `add_artifact` API to upload local file to spark driver (spark connect already support this)

(2) implement a pyspark (spark connect client) API: `copy_artifact_to_hadoop_fs`, the API sends a command to spark driver to request upload the artifact file to hadoop FS, *we need to design a spark connect protobuf command message for this part.* In spark driver side, when spark connect server received the request, it gets `sparkContext.hadoopConf` and then using Hadoop FileSystem API to upload file to Hadoop FS.

(3) call `copy_artifact_to_hadoop_fs` API to upload artifact file to Hadoop FS.
Bug: Avro to Support custom decimal type backed by Long
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/May/23 17:52
Affects Version/s: 3.5.0

Right now, Avro only allows Decimal logical type in fixed and array types. However, there is a requirement from users to represent decimal in long type. It is to support represent currency (for money). The request is to support a customized decimal type backed by long.
Bug: Migrate deprecated createOrReplace to serverSideApply
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/May/23 13:00
Affects Version/s: 3.5.0

 

 

{{public interface CreateOrReplaceable<T> extends Replaceable<T> {}}

{{  /**}}
{{   * Creates a provided resource in a Kubernetes Cluster. If creation}}
{{   * fails with a HTTP_CONFLICT, it tries to replace resource.}}
{{   *}}
{{   * @return created item returned in kubernetes api response}}
{{   *}}
{{   * @deprecated please use \{@link ServerSideApplicable#serverSideApply()} or attempt a create and edit/patch operation.}}
{{   */}}
{{  @Deprecated}}
{{  T createOrReplace();}}

 

{{  /**}}
{{   * Creates an item}}
{{   *}}
{{   * @return the item from the api server}}
{{   */}}
{{  T create();}}
{{}}}
Bug: Spark application hangs when YarnAllocator adds running executors after processing completed containers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/23 15:35
Affects Version/s: 3.4.0

I see application hangs when containers are preempted immediately after allocation as follows.
{code:java}
23/05/14 09:11:33 INFO YarnAllocator: Launching container container_e3812_1684033797982_57865_01_000382 on host hdc42-mcc10-01-0910-4207-015-tess0028.stratus.rno.ebay.com for executor with ID 277 for ResourceProfile Id 0 
23/05/14 09:11:33 WARN YarnAllocator: Cannot find executorId for container: container_e3812_1684033797982_57865_01_000382
23/05/14 09:11:33 INFO YarnAllocator: Completed container container_e3812_1684033797982_57865_01_000382 (state: COMPLETE, exit status: -102)
23/05/14 09:11:33 INFO YarnAllocator: Container container_e3812_1684033797982_57865_01_000382 was preempted.{code}
Note the warning log where YarnAllocator cannot find executorId for the container when processing completed containers. The only plausible cause is YarnAllocator added the running executor after processing completed containers. The former happens in a separate thread after executor launch.

YarnAllocator believes there are still running executors, although they are already lost due to preemption. Hence, the application hangs without any running executors.
Bug: Add spark DataFrame binary file format writer
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 22/May/23 12:07
Affects Version/s: 3.5.0

In new distributed spark ML module (designed to support spark connect and support local inference)

We need to save ML model to hadoop file system using custom binary file format, the reason is:
 * We often submit a spark application to spark cluster for running the training model job, we need to save trained model to hadoop file system before the spark application completes.
 * But we want to support local model inference, that means if we save the model by current spark DataFrame writer (e.g. parquet format), when loading model we have to rely on the spark service. But we hope we can load model without spark service. So we want the model being saved as the original binary format that our ML code can handle.

We already have reader API of "binaryFile" format, we need to add a writer API:

{*}Writer API{*}:

Supposing we have a dataframe with schema:

[file_path: String, content: binary],

we can save the dataframe to a hadoop path, each row we will save it as a file under the hadoop path, the saved file path is \{hadoop path}/\{file_path}, "file_path" can be a multiple part path.
Bug: SerializerHelper.deserializeFromChunkedBuffer leaks deserialization streams
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/May/23 12:59
Affects Version/s: 3.4.0

The method SerializerHelper.deserializeFromChunkedBuffer leaks serializations stream. This can lead to huge performance regressions when using kryo serializer as the spark application can become bottlenecked on the driver creating expensive kryo objects that are then leaked as part of the deserialization stream
Bug: Add support for df[|] suggest column
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/May/23 00:02
Affects Version/s: 3.5.0

We don't have a keys method for pyspark dataframe, even though we support df['column name']

And when we want to get a suggestion for {{{}df[|]{}}}, we can't get anything. So this could be an improvement for autocomplete engine that uses {{keys()}} to generate autocomplete suggestions (e.g. IPython kernel, Databricks Notebooks) will suggest column names on the completion df[|]

However, for those using IPython, they don't check keys, or say it only support keys for pandas & numpy & dict type. So we need to add _ipython_key_completions_ to our object to make it work.
Bug: Basic estimator / transformer / model / evaluator interfaces and basic transformer / evaluator implementation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/23 23:58
Affects Version/s: 3.5.0

* Define basic interfaces of Evaluator / Transformer / Model / Evaluator, these interfaces are designed to support both spark connect and legacy spark mode, and are designed to support train / transform / evaluate over either spark dataframe or pandas dataframe
 * Implement a feature transformer {{{}MaxAbsScaler{}}}, {{ScalerScaler}}
 * Implement a regressor evaluator {{RegressorEvaluator}} that supports MSE and R2 metric evaluation
Bug: spark cluster deploy mode cannot initialize metastore java.sql.SQLException: No suitable driver found for jdbc:mysql
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 29/May/23 12:19
Affects Version/s: 3.4.0

1. Test with JDK 11 + SPARK340
object BugHS {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf()
    conf.set("javax.jdo.option.ConnectionURL","jdbc:mysql://mysql:3306/hive_ms_spark3?useSSL=false")
    conf.set("javax.jdo.option.ConnectionDriverName","com.mysql.jdbc.Driver")
    conf.set("javax.jdo.option.ConnectionUserName","**")
    conf.set("javax.jdo.option.ConnectionPassword","**")
    conf.set("spark.sql.hive.thriftServer.singleSession","false")
    conf.set("spark.sql.warehouse.dir","hdfs://hadoop/warehouse_spark3")
    import org.apache.spark.sql.SparkSession
    val spark = SparkSession
      .builder()
      .appName("Test").config(conf).enableHiveSupport()
      .getOrCreate()
    HiveThriftServer2.startWithContext(spark.sqlContext)
    spark.sql("create table IF NOT EXISTS test2 (id int) USING parquet")
  }
}
2. Submit in cluster mode
   a. spark_config.properties 
            spark.master=spark://master:6066
            spark.jars=hdfs://hadoop/tmp/test_bug/mysql-connector-java-5.1.47.jar
            spark.master.rest.enabled=true
   b. spark-submit2.cmd --deploy-mode cluster  --properties-file spark_config.properties  --class com.test.BugHS "hdfs://hadoop/tmp/test_bug/bug_classloader.jar"  

3.  Meet "No suitable driver found" exception, caused by classloader is different for driver in spark.jars and metastore jar in JDK 11
java.sql.SQLException: Unable to open a test connection to the given database. JDBC url = jdbc:mysql://mysql:3306/hive_ms_spark3?useSSL=false, username = root. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
java.sql.SQLException: No suitable driver found for jdbc:mysql://mysql:3306/hive_ms_spark3?useSSL=false
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:702)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189)
	at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
	at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416)
	at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:120)
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:483)
	at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:297)
	at jdk.internal.reflect.GeneratedConstructorAccessor77.newInstance(Unknown Source)
Bug: Implement changelog checkpointing for RocksDB state store
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/May/23 06:38
Affects Version/s: 3.4.0

We have identified state checkpointing latency as one of the major performance bottlenecks for stateful streaming queries. Currently, RocksDB state store pauses the RocksDB instances to upload a snapshot to the cloud when committing a batch, which is heavy weight and has unpredictable performance.

In order to reduce the checkpoint duration and end to end latency, we propose to
1. During state commit, make the state of a microbatch durable by syncing the changelog instead of the state snapshot to the checkpoint directory.
2. Upload snapshot in the background to enable changelog purging and faster failure recovery.

In this way, we allow the RocksDB instance to run uninterruptibly, which improves RocksDB operation performance. This also dramatically reduces the commit time and batch duration because we are uploading a smaller amount of data during state commit. With this change, stateful query with RocksDB state store will have lower and more predictable latency.
Bug: Spark cannot construct the DecimalType in CatalystTypeConverters.convertToCatalyst() API
Status: Open | Priority: Major | Resolution: 
Created: 26/May/23 09:11
Affects Version/s: 3.2.2

 

When constructing the DecimalType in CatalystTypeConverters.convertToCatalyst(), spark throw following exception:

 
{code:java}
Decimal scale (18) cannot be greater than precision (1).
org.apache.spark.sql.AnalysisException: Decimal scale (18) cannot be greater than precision (1).
        at org.apache.spark.sql.errors.QueryCompilationErrors$.decimalCannotGreaterThanPrecisionError(QueryCompilationErrors.scala:1671)
        at org.apache.spark.sql.types.DecimalType.<init>(DecimalType.scala:48)
        at org.apache.spark.sql.catalyst.CatalystTypeConverters$.convertToCatalyst(CatalystTypeConverters.scala:518)
        at org.apache.spark.sql.DataFrameFunctionsSuite.$anonfun$new$712(DataFrameFunctionsSuite.scala:3714){code}
 

 

This issue can be reproduced by the following case:

 
{code:java}
  val expression = Literal.default(DecimalType.SYSTEM_DEFAULT)
  val schema = StructType(
    StructField("a", IntegerType, nullable = true) :: Nil)
  val empData = Seq(Row(1))
  val df = spark.createDataFrame(spark.sparkContext.parallelize(empData), schema)
  val resultDF = df.select(Column(expression))
  val result = resultDF.collect().head.get(0)
  CatalystTypeConverters.convertToCatalyst(result)

{code}
 

It seems that the reason for the failure is that the value of precision is not set when the Decimal.toJavaBigDecimal() method is called. However, Java BigDecimal only provides an interface for modifying scale and does not provide an interface for modifying Precision.
Bug: Avro to Name union type members after types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/May/23 20:56
Affects Version/s: 3.3.2

Spark converts Avro union types into record types, where each member of the union type corresponds to a field in the record type. The current behaviour is to name the record fields "member0", "member1", etc, for each member of the union type. We propose having the option to instead use the member type name.

The purpose of this is twofold:
 # To allow adding or removing types to the union without affecting the record names of other member types. If the new or removed type is not ordered last, then existing queries referencing "member2" may need to be rewritten to reference "member1" or "member3".
 # Referencing the type name in the query is more readable than referencing "member0".

For example, our system produces an avro schema from a Java type structure where subtyping maps to union types whose members are ordered lexicographically. Adding a subtype can therefore easily result in all references to "member2" needing to be updated to "member3".
Bug: Improve suggested candidates for unresolved attribute
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/May/23 20:13
Affects Version/s: 3.4.0

It is related to [https://github.com/apache/spark/pull/41038] . Given a column list cols
 # group by qualifiers, if only one group, strip qualifier and only keep the column name
 # in each group,  group by the qualifiers excluding table name, if only one group, strip qualifiers excluding table name, only keep table name + column name
 # sort the result by similarity with the base string
Bug: DataSource V2: Allow representing updates as deletes and inserts
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/May/23 15:31
Affects Version/s: 3.5.0

It may be beneficial for data sources to represent updates as deletes and inserts for delta-based implementations. Specifically, it may be helpful to properly distribute and order records on write. Remember that delete records have only row ID and metadata attributes set. Update records have data, row ID, metadata attributes set. Insert records have only data attributes set.

For instance, a data source may rely on a metadata column _row_id (synthetic internally generated) to identify the row and is partitioned by bucket(product_id). Splitting updates into inserts and deletes would allow data sources to cluster all update and insert records for the same partition into a single task. Otherwise, the clustering key for updates and inserts will be different (updates have _row_id set). This is critical to reduce the number of generated files.
Bug: Add check to see if column name is legal for __dir__
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/May/23 20:55
Affects Version/s: 3.5.0

In SPARK-43270, we add a support for the autocomplete suggest for {{df.|}} , which also suggest column_name for dataframe. 

However, we found out later that some dataframe can have column which has illegal variable name (e.g: name?1, name 1, 2name etc.) These variable name should be filtered out and this will be consistent with pandas behavior now