av9ash/GitBugs huggingface dataset:
Bug: RocksDB State Store can accumulate unbounded native memory
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 23/Apr/23 18:39
Affects Version/s: 3.3.2

We noticed in one of our production stateful streaming jobs using RocksDB that an executor with 20g of heap was using around 40g of resident memory. I noticed a single RocksDB instance was using around 150 MiB of memory, and only 5 MiB or so of this was from the write batch (which is now cleared after committing).

After reading about RocksDB memory usage (this link was helpful: [https://github.com/EighteenZi/rocksdb_wiki/blob/master/Memory-usage-in-RocksDB.md]) I realized a lot of this was likely the "Index and Filters" memory usage. This job is doing a streaming duplicate with a lot of unique keys so it makes sense these block usages would be high. The problem is that, because as it is now the underlying RocksDB instance stays open on an executor as long as that executor is assigned that stateful partition (to be reused across batches). So a single executor can accumulate a large number of RocksDB instances open at once, each using a certain amount of native memory. In the worst case, a single executor could need to keep open every single partitions' RocksDB instance at once. 

There are a couple ways you can control the amount of memory used, such as limiting the max open files, or adding the option to use the block cache for the indices and filters, but neither of these solve the underlying problem of accumulating native memory from multiple partitions on an executor.

The real fix needs to be a mechanism and option to close the underlying RocksDB instance at the end of each task, so you have the option to only ever have one RocksDB instance open at a time, thus having predictable memory usage no matter the size of your data or number of shuffle partitions. 

We are running this on Spark 3.3, but just kicked off a test to see if things are any different in Spark 3.4.
Bug: Implement __dir__() in pyspark.sql.dataframe.DataFrame to include columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Apr/23 20:45
Affects Version/s: 3.5.0

Currently, Given {{df.|}} , the databricks notebook will only suggest the method of dataframe(see attached Screenshot of databricks notebook),

{{However, df.column_name}} is also legal and runnable 

Hence we should override the parent __{{{}dir__{}}}{{{}(){}}} method on Python {{DataFrame}} class to include column names. And the benefit of this is engine that uses {{dir()}} to generate autocomplete suggestions (e.g. IPython kernel, Databricks Notebooks) will suggest column names on the completion {{df.|}}
Bug: Upgrade FasterXML jackson to 2.15.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Apr/23 13:34
Affects Version/s: 3.5.0

* #390: (yaml) Upgrade to Snakeyaml 2.0 (resolves [CVE-2022-1471|https://nvd.nist.gov/vuln/detail/CVE-2022-1471])
 (contributed by @pjfannin
Bug: predict_batch_udf with scalar input fails when batch size consists of a single value
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Apr/23 20:31
Affects Version/s: 3.4.0

This is related to SPARK-42250.  For scalar inputs, the predict_batch_udf will fail if the batch size is 1:
{code:java}
import numpy as np
from pyspark.ml.functions import predict_batch_udf
from pyspark.sql.types import DoubleType

df = spark.createDataFrame([[1.0],[2.0]], schema=["a"])

def make_predict_fn():
    def predict(inputs):
        return inputs
    return predict

identity = predict_batch_udf(make_predict_fn, return_type=DoubleType(), batch_size=1)
preds = df.withColumn("preds", identity("a")).collect()
{code}
fails with:
{code:java}
  File "/.../spark/python/pyspark/worker.py", line 869, in main
    process()
  File "/.../spark/python/pyspark/worker.py", line 861, in process
    serializer.dump_stream(out_iter, outfile)
  File "/.../spark/python/pyspark/sql/pandas/serializers.py", line 354, in dump_stream
    return ArrowStreamSerializer.dump_stream(self, init_stream_yield_batches(), stream)
  File "/.../spark/python/pyspark/sql/pandas/serializers.py", line 86, in dump_stream
    for batch in iterator:
  File "/.../spark/python/pyspark/sql/pandas/serializers.py", line 347, in init_stream_yield_batches
    for series in iterator:
  File "/.../spark/python/pyspark/worker.py", line 555, in func
    for result_batch, result_type in result_iter:
  File "/.../spark/python/pyspark/ml/functions.py", line 818, in predict
    yield _validate_and_transform_prediction_result(
  File "/.../spark/python/pyspark/ml/functions.py", line 339, in _validate_and_transform_prediction_result
    if len(preds_array) != num_input_rows:
TypeError: len() of unsized object
{code}
Bug: Improve scalar subquery logic plan when result are literal
Status: Open | Priority: Major | Resolution: 
Created: 27/Apr/23 12:43
Affects Version/s: 3.5.0

When use scalar subquery, sometimes we can get result before physical plan execute. 

Like `select (select (count(1)) is null from r where a = c ) from l where a < 4`.

The result alway be false. So we can skip unnecessary aggregate.
Bug: Connect JVM client REPL not correctly shut down if killed
Status: Open | Priority: Major | Resolution: 
Created: 25/Apr/23 20:45
Affects Version/s: 3.5.0

How to reproduce:
 # Start a scala client `./connector/connect/bin/spark-connect-scala-client`
 # in another terminal, kill the process `kill <pid>`
 # Back to the client terminal, you can't see anything you type, but the command still works

 

 
{code:java}
Spark session available as 'spark'.
   _____                  __      ______                            __
  / ___/____  ____ ______/ /__   / ____/___  ____  ____  ___  _____/ /_
  \__ \/ __ \/ __ `/ ___/ //_/  / /   / __ \/ __ \/ __ \/ _ \/ ___/ __/
 ___/ / /_/ / /_/ / /  / ,<    / /___/ /_/ / / / / / / /  __/ /__/ /_
/____/ .___/\__,_/_/  /_/|_|   \____/\____/_/ /_/_/ /_/\___/\___/\__/
    /_/


@ wei.liu:~/oss-spark$ CONTRIBUTING.md  appveyor.yml  conf                        examples            logs         resource-managers                    target
LICENSE          artifacts     connector                   graphx              mllib        sbin                                 tools
LICENSE-binary   assembly      core                        hadoop-cloud        mllib-local  scalastyle-config.xml
NOTICE           bin           data                        hs_err_pid9062.log  pom.xml      scalastyle-on-compile.generated.xml
NOTICE-binary    binder        dependency-reduced-pom.xml  launcher            project      spark-warehouse
R                build         dev                         licenses            python       sql
README.md        common        docs                        licenses-binary     repl         streaming
wei.liu:~/oss-spark$ wei.liu:~/oss-spark$ wei.liu:~/oss-spark$ 

{code}
 

I ran 'ls' above, and clicked return multiple times
Bug: Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
Status: Open | Priority: Major | Resolution: 
Created: 25/Apr/23 09:16
Affects Version/s: 3.3.0

Java version: 1.8.0_331, Apache Maven 3.8.4

I run next steps:
 # git clone [https://github.com/apache/spark.git]
 # git checkout -b v3.3.0 3.3.0
 #  mvn clean install -DskipTests
 # copy hive-site.xml to examples/src/main/resources/
 # execute TPC-H Q6 

 
{code:java}
public static void main(String[] args) throws InterruptedException {
        SparkConf sparkConf = new SparkConf()
                .setAppName("demo")
                .setMaster("local[1]")
                ;        
         SparkSession sparkSession = SparkSession.builder()
                .config(sparkConf)
                .enableHiveSupport()
                .getOrCreate();
        sparkSession.sql("use local_tpch_sf10_uncompressed_etl");        
        sparkSession.sql(TPCH.SQL6).show();
} {code}
 

 

get the error info:

Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
    at org.apache.spark.util.io.ChunkedByteBufferOutputStream.toChunkedByteBuffer(ChunkedByteBufferOutputStream.scala:115)
    at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:325)
    at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:140)
    at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:95)
    at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
    at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:75)
    at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1529)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.buildReaderWithPartitionValues(ParquetFileFormat.scala:235)
    at org.apache.spark.sql.execution.FileSourceScanExec.inputRDD$lzycompute(DataSourceScanExec.scala:457)
    at org.apache.spark.sql.execution.FileSourceScanExec.inputRDD(DataSourceScanExec.scala:448)
    at org.apache.spark.sql.execution.FileSourceScanExec.doExecuteColumnar(DataSourceScanExec.scala:547)
    at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeColumnar$1(SparkPlan.scala:221)
    at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:232)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:229)
    at org.apache.spark.sql.execution.SparkPlan.executeColumnar(SparkPlan.scala:217)
Bug: Fix "Could not initialise class ammonite..." error when using filter
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/23 16:07
Affects Version/s: 3.5.0

When
{code:java}
spark.range(10).filter(n => n % 2 == 0).collectAsList()`{code}
 is run in the ammonite REPL (Spark Connect), the following error is thrown:
{noformat}
io.grpc.StatusRuntimeException: UNKNOWN: ammonite/repl/ReplBridge$
  io.grpc.Status.asRuntimeException(Status.java:535)
  io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:660)
  org.apache.spark.sql.connect.client.SparkResult.org$apache$spark$sql$connect$client$SparkResult$$processResponses(SparkResult.scala:62)
  org.apache.spark.sql.connect.client.SparkResult.length(SparkResult.scala:114)
  org.apache.spark.sql.connect.client.SparkResult.toArray(SparkResult.scala:131)
  org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2687)
  org.apache.spark.sql.Dataset.withResult(Dataset.scala:3088)
  org.apache.spark.sql.Dataset.collect(Dataset.scala:2686)
  org.apache.spark.sql.Dataset.collectAsList(Dataset.scala:2700)
  ammonite.$sess.cmd0$.<init>(cmd0.sc:1)
  ammonite.$sess.cmd0$.<clinit>(cmd0.sc){noformat}
Bug: PySpark UDF supports python package dependencies
Status: Open | Priority: Major | Resolution: 
Created: 26/Apr/23 03:33
Affects Version/s: 3.5.0

h3. Requirements

 

Make the pyspark UDF support annotating python dependencies and when executing UDF, the UDF worker creates a new python environment with provided python dependencies.
h3. Motivation

 

We have two major cases:

 
 * For spark connect case, the client python environment is very likely to be different with pyspark server side python environment, this causes user's UDF function execution failure in pyspark server side.
 * Some machine learning third-party library (e.g. MLflow) requires pyspark UDF supporting  dependencies, because in ML cases, we need to run model inference by pyspark UDF in the exactly the same python environment that trains the model. Currently MLflow supports it by creating a child python process in pyspark UDF worker, and redirecting all UDF input data to the child python process to run model inference, this way it causes significant overhead, if pyspark UDF support builtin python dependency management then we don't need such poorly performing approach.

 
h3. Proposed API

```

@pandas_udf("string", pip_requirements=...)

```

`pip_requirements` argument means either an iterable of pip requirement strings (e.g. ``["scikit-learn", "-r /path/to/req2.txt", "-c /path/to/constraints.txt"]``) or the string path to a pip requirements file path on the local filesystem (e.g. ``"/path/to/requirements.txt"``) represents the pip requirements for the python UDF.
Bug: ReplE2ESuite consistently fails with JDK 17
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Apr/23 16:16
Affects Version/s: 3.5.0

[[Comment|https://github.com/apache/spark/pull/40675#discussion_r1174696470] from [~gurwls223]]

This test consistently fails with JDK 17:
{code:java}
[info] ReplE2ESuite:
[info] - Simple query *** FAILED *** (10 seconds, 4 milliseconds)
[info] java.lang.RuntimeException: REPL Timed out while running command: 
[info] spark.sql("select 1").collect()
[info] 
[info] Console output: 
[info] Error output: Compiling (synthetic)/ammonite/predef/ArgsPredef.sc
[info] at org.apache.spark.sql.application.ReplE2ESuite.runCommandsInShell(ReplE2ESuite.scala:87)
[info] at org.apache.spark.sql.application.ReplE2ESuite.$anonfun$new$1(ReplE2ESuite.scala:102)
[info] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info] at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info] at org.scalatest.Transformer.apply(Transformer.scala:22)
[info] at org.scalatest.Transformer.apply(Transformer.scala:20)
[info] at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
[info] at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
[info] at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
[info] at org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1564)
[info] at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224){code}

[https://github.com/apache/spark/actions/runs/4780630672/jobs/8498505928#step:9:4647]
[https://github.com/apache/spark/actions/runs/4774942961/jobs/8488946907]
[https://github.com/apache/spark/actions/runs/4769162286/jobs/8479293802]
[https://github.com/apache/spark/actions/runs/4759278349/jobs/8458399201]
[https://github.com/apache/spark/actions/runs/4748319019/jobs/8434392414]
Bug: _metadata.file_path returns unescaled URLs
Status: Open | Priority: Major | Resolution: 
Created: 25/Apr/23 15:52
Affects Version/s: 3.4.0

As part of https://issues.apache.org/jira/browse/SPARK-41970 we changed the encoding of the string returned by `_metadata.file_path` from url-encoded to hadoop-path encoded (i.e. not encoded).

 

This ticket is to undo that behavior change.
Bug: DSL expressions fail on attribute with special characters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Apr/23 13:11
Affects Version/s: 3.4.0

Expressions on implicitly converted attributes fail if the attributes have names containing special characters. They fail even if the attributes are backtick-quoted:
{code:java}
scala> import org.apache.spark.sql.catalyst.dsl.expressions._
import org.apache.spark.sql.catalyst.dsl.expressions._

scala> "`slashed/col`".attr
res0: org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute = 'slashed/col

scala> "`slashed/col`".attr.asc
org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input '/' expecting {<EOF>, '.', '-'}(line 1, pos 7)

== SQL ==
slashed/col
-------^^^
{code}
Bug: Define extractors for file-constant metadata columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Apr/23 03:43
Affects Version/s: 3.4.0

File-source constant metadata columns are often derived indirectly from file-level metadata values rather than exposing those values directly. For example, {{_metadata.file_name}} is currently hard-coded in {{FileFormat.updateMetadataInternalRow}} as:

 
{code:java}
UTF8String.fromString(filePath.getName){code}
 

We should add support for metadata extractors, functions that map from {{PartitionedFile}} to {{{}Literal{}}}, so that we can express such columns in a generic way instead of hard-coding them.

We can't just add them to the metadata map because then they have to be pre-computed even if it turns out the query does not select that field.
Bug: Codegen error when full outer join's bound condition has multiple references to the same stream-side column
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Apr/23 22:33
Affects Version/s: 3.3.2

Example # 1 (sort merge join):
{noformat}
create or replace temp view v1 as
select * from values
(1, 1),
(2, 2),
(3, 1)
as v1(key, value);

create or replace temp view v2 as
select * from values
(1, 22, 22),
(3, -1, -1),
(7, null, null)
as v2(a, b, c);

select *
from v1
full outer join v2
on key = a
and value > b
and value > c;
{noformat}
The join's generated code causes the following compilation error:
{noformat}
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 277, Column 9: Redefinition of local variable "smj_isNull_7"
{noformat}
Example #2 (shuffle hash join):
{noformat}
select /*+ SHUFFLE_HASH(v2) */ *
from v1
full outer join v2
on key = a
and value > b
and value > c;
{noformat}
The shuffle hash join's generated code causes the following compilation error:
{noformat}
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 174, Column 5: Redefinition of local variable "shj_value_1" 
{noformat}
With default configuration, both queries end up succeeding, since Spark falls back to running each query with whole-stage codegen disabled.

The issue happens only when the join's bound condition refers to the same stream-side column more than once.
Bug: Remove null_counts from info()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Apr/23 20:13
Affects Version/s: 3.4.0

df.info() is broken now. 
It prints 

TypeError                                 Traceback (most recent call last)
Cell In[12], line 1
----> 1 F05.info()

File /opt/spark/python/pyspark/pandas/frame.py:12167, in DataFrame.info(self, verbose, buf, max_cols, null_counts)
  12163     count_func = self.count
  12164     self.count = (  # type: ignore[assignment]
  12165         lambda: count_func()._to_pandas()  # type: ignore[assignment, misc, union-attr]
  12166     )
> 12167     return pd.DataFrame.info(
  12168         self,  # type: ignore[arg-type]
  12169         verbose=verbose,
  12170         buf=buf,
  12171         max_cols=max_cols,
  12172         memory_usage=False,
  12173         null_counts=null_counts,
  12174     )
  12175 finally:
  12176     del self._data

TypeError: DataFrame.info() got an unexpected keyword argument 'null_counts'
Bug: Spark connect's user agent validations are too restrictive
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/23 12:43
Affects Version/s: 3.4.0

The current restriction on allowed charset and length are too restrictive

 

https://github.com/apache/spark/blob/cac6f58318bb84d532f02d245a50d3c66daa3e4b/python/pyspark/sql/connect/client.py#L274-L275
Bug: INSERT INTO REPLACE statement can't support WHERE with bool_expression
Status: Closed | Priority: Major | Resolution: Invalid
Created: 20/Apr/23 15:39
Affects Version/s: 3.4.0

{code:java}
sql("CREATE TABLE persons (name string,address String,ssn int) USING parquet")
sql("CREATE TABLE persons2 (name string,address String,ssn int) USING parquet")
sql("INSERT INTO TABLE persons VALUES " +
"('Dora Williams', '134 Forest Ave, Menlo Park', 123456789)," +
"('Eddie Davis','245 Market St, Milpitas',345678901)")

sql("INSERT INTO TABLE persons2 VALUES ('Ashua Hill', '456 Erica Ct, Cupertino', 432795921)")
sql("INSERT INTO persons REPLACE WHERE ssn =  123456789 SELECT * FROM persons2")
sql("SELECT * FROM persons").show(){code}

When use `INSERT INTO table REPLACE WHERE`, only support `WHERE TRUE` at now. `WHERE ssn = 123456789` or `WHERE FALSE` both not support.

!image-2023-04-20-23-40-25-212.png|width=795,height=152!
Bug: Support "ESCAPE BY" in SparkScriptTransformationExec
Status: Open | Priority: Major | Resolution: 
Created: 20/Apr/23 13:23
Affects Version/s: 3.2.0

If I don't `set spark.sql.catalogImplementation=hive`, I can't use "SELECT TRANSFORM" with "ESCAPE BY". Although HiveScriptTransform also doesn't implement ESCAPE BY, I can use RowFormatSerde to achieve this ability.

 

In fact, HiveScriptTransform doesn't need to connect to Hive Metastore. I can use reflection to forcibly call HiveScriptTransformationExec without connecting to Hive Metastore, and it can work properly. Maybe HiveScriptTransform can be more generic.
Bug: Move update event on idleness in streaming query listener to separate callback method
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/23 09:45
Affects Version/s: 3.5.0

People has been having a lot of confusions about update event on idleness; it’s not only the matter of understanding but also comes up with various types of complaints. For example, since we give the latest batch ID for update event on idleness, if the listener implementation blindly performs upsert based on batch ID, they are in risk to lose metrics.

This also complicates the logic because we have to memorize the execution for the previous batch, which is arguably not necessary.

Because of this, we’d be better to move the idle event out of progress update event and have separate callback method for this.
Bug: Spark may  use a column other than the actual specified partitioning column for partitioning, for Hive format tables
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 12/Apr/23 22:11
Affects Version/s: 3.3.1

The class org.apache.spark.sql.catalyst.catalog.HiveTableRelation has its output method implemented as 
  // The partition column should always appear after data columns.
  override def output: Seq[AttributeReference] = dataCols ++ partitionCols

But the DataWriting commands of spark like InsertIntoHiveDirCommand, expect that the output from HiveTableRelation is in the order in which the columns are actually defined in the DDL.

As a result, multiple mismatch scenarios can happen like:
1) data type casting exception being thrown , even though the data frame being inserted has schema which is identical to what is used for creating ddl.
              OR
2) Wrong column being used for partitioning , if the datatypes are same or cast-able, like date type and long

will be creating a PR with the bug test