av9ash/GitBugs huggingface dataset:
Bug: Mark KryoRegistrator as DeveloperApi interface
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Aug/24 22:17
Affects Version/s: 4.0.0

Trait org.apache.spark.serializer.KryoRegistrator is a public interface because it is exposed via config "spark.kryo.registrator" since version 0.5.0. It should have an annotation to describe its stability.
Bug: Increase `Gradle` JVM memory to `4g` like Spark repo
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Aug/24 02:33
Affects Version/s: kubernetes-operator-0.1.0

{code:java}
> Task :spark-operator:compileTestJava
Note: /Users/dongjoon/APACHE/spark-kubernetes-operator/spark-operator/src/test/java/org/apache/spark/k8s/operator/metrics/healthcheck/SentinelManagerTest.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
OpenJDK 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
[1.372s][warning][codecache] CodeCache is full. Compiler has been disabled.
OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
[1.372s][warning][codecache] Try increasing the code cache size using -XX:ReservedCodeCacheSize=
CodeCache: size=2944Kb used=2943Kb max_used=2943Kb free=0Kb
 bounds [0x0000000105004000, 0x00000001052e4000, 0x00000001052e4000]
 total_blobs=1102 nmethods=464 adapters=554
 compilation: disabled (not enough contiguous free space left)
              stopped_count=1, restarted_count=0
 full_count=1 {code}
Bug: Improve readability of log4j console log output
Status: Open | Priority: Major | Resolution: 
Created: 07/Aug/24 17:34
Affects Version/s: 4.0.0

Prior to this update, the OSS Spark logs were difficult to interpret. The logs followed a JSON output format which is not optimal for human consumption:

 
{code:java}
{"ts":"2024-07-26T18:45:17.712Z","level":"INFO","msg":"Running Spark version 4.0.0-SNAPSHOT","context":{"spark_version":"4.0.0-SNAPSHOT"},"logger":"SparkContext"}{"ts":"2024-07-26T18:45:17.715Z","level":"INFO","msg":"OS info Mac OS X, 14.4.1, aarch64","context":{"os_arch":"aarch64","os_name":"Mac OS X","os_version":"14.4.1"},"logger":"SparkContext"}{"ts":"2024-07-26T18:45:17.716Z","level":"INFO","msg":"Java version 17.0.11","context":{"java_version":"17.0.11"},"logger":"SparkContext"}{"ts":"2024-07-26T18:45:17.761Z","level":"WARN","msg":"Unable to load native-hadoop library for your platform... using builtin-java classes where applicable","logger":"NativeCodeLoader"}{"ts":"2024-07-26T18:45:17.783Z","level":"INFO","msg":"==============================================================","logger":"ResourceUtils"}{"ts":"2024-07-26T18:45:17.783Z","level":"INFO","msg":"No custom resources configured for spark.driver.","logger":"ResourceUtils"}{"ts":"2024-07-26T18:45:17.783Z","level":"INFO","msg":"==============================================================","logger":"ResourceUtils"}{"ts":"2024-07-26T18:45:17.784Z","level":"INFO","msg":"Submitted application: Spark Pi","context":{"app_name":"Spark Pi"},"logger":"SparkContext"}...{"ts":"2024-07-26T18:45:18.036Z","level":"INFO","msg":"Start Jetty 0.0.0.0:4040 for SparkUI","context":{"host":"0.0.0.0","port":"4040","server_name":"SparkUI"},"logger":"JettyUtils"}{"ts":"2024-07-26T18:45:18.044Z","level":"INFO","msg":"jetty-11.0.20; built: 2024-01-29T21:04:22.394Z; git: 922f8dc188f7011e60d0361de585fd4ac4d63064; jvm 17.0.11+9-LTS","logger":"Server"}{"ts":"2024-07-26T18:45:18.054Z","level":"INFO","msg":"Started Server@22c75c01{STARTING}[11.0.20,sto=30000] @1114ms","logger":"Server"} {code}
 

 

This issue updates the default `log4j.properties.template` with the following improvements for console logging format:
 * Use PatternLayout for improved human readability
 * Color-code log levels to simplify logging output
 * Visually partition the threadName and contextInfo for easy interpretation
Bug: Support type promotion/widening for Avro readers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Aug/24 18:50
Affects Version/s: 4.0.0

Similar to the ask in https://issues.apache.org/jira/browse/SPARK-40876, we should be able to enable type promotion/widening for Avro readers - i.e. we should be able to read back Avro data encoded in a narrower type using a wider Spark schema type.

We should support the following:
 * Int -> Long
 * Float -> Double
 * Int -> Double
 * Date -> TimestampNTZ (Timestamp without timezone only as a date has no timezone information)
 * Decimal with higher precision (already supported in non-vectorized reader)
Bug: Make MemoryConsumer thread-safe
Status: Open | Priority: Major | Resolution: 
Created: 07/Aug/24 00:52
Affects Version/s: 3.5.1

TaskMemoryManager is thread-safe however MemoryConsumer is not.

Thinking of multiple threads are allocating memory in the same task (although it's known not to be ideal regarding Spark's one-thread-per-task model), it leads to practice that developer should add a lock in the caller code of MemoryConsumer. However since it's common that one memory consumer spills another, using lock in user code could easily causes ABBA dead lock, e.g., consumer A locks it self to acquire memory from task memory manager, while consumer B locks task memory manager then spills A.

It's easy to make MemoryConsumer thread-safe. AFAICS the only change needed so far is to make accesses of field `used` atomic.
Bug: Improve the logging behavior when Spark loaded zero result from Kafka after poll timeout
Status: Open | Priority: Major | Resolution: 
Created: 06/Aug/24 13:36
Affects Version/s: 3.3.0

When using spark structured streaming to read data from Kafka, the poll request can time out if the producer is writing data to the topic in transaction mode but not commit the transaction for a long time.

 

When this happened, the `r.isEmpty` as `True` but the offsetAfterPoll changed. This will not thrown any error or warning logs. This makes users have no idea why the related spark task delay for `pollTimeoutMs`. Here is the related [code|https://github.com/apache/spark/blob/v3.3.0/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/KafkaDataConsumer.scala#L85-L101]:

 

 
{code:java}
    if (r.isEmpty) {
      // We cannot fetch anything after `poll`. Two possible cases:
      // - `offset` is out of range so that Kafka returns nothing. `OffsetOutOfRangeException` will
      //   be thrown.
      // - Cannot fetch any data before timeout. `TimeoutException` will be thrown.
      // - Fetched something but all of them are not invisible. This is a valid case and let the
      //   caller handles this.
      if (offset < range.earliest || offset >= range.latest) {
        throw new OffsetOutOfRangeException(
          Map(topicPartition -> java.lang.Long.valueOf(offset)).asJava)
      } else if (offset == offsetAfterPoll) {
        throw new TimeoutException(
          s"Cannot fetch record for offset $offset in $pollTimeoutMs milliseconds")
      }
    }{code}
 

 

To improve the user experience for troubleshooting when the poll get zero result before timeout but offsetAfterPoll changed, there should add some warning or error level logs:

 

A sample looks like the following:

 
{code:java}
    if (r.isEmpty) {
      // We cannot fetch anything after `poll`. Two possible cases:
      // - `offset` is out of range so that Kafka returns nothing. `OffsetOutOfRangeException` will
      //   be thrown.
      // - Cannot fetch any data before timeout. `TimeoutException` will be thrown.
      // - Fetched something but all of them are invisible. This is a valid case and let the
      //   caller handles this. Although this is a valid case, it is necessary to add warning level logs to indicate zero data returned before timeout.
      if (offset < range.earliest || offset >= range.latest) {
        throw new OffsetOutOfRangeException(
          Map(topicPartition -> java.lang.Long.valueOf(offset)).asJava)
      } else if (offset == offsetAfterPoll) {
        throw new TimeoutException(
          s"Cannot fetch record for offset $offset in $pollTimeoutMs milliseconds")
      } else {
        logWarning(s"After taking $pollTimeoutMs milliseconds polling, ZERO record fetched while offset changed from $offset to $offsetAfterPoll.")
      }
} {code}
 

 

By the way, fix the typo from `Fetched something but all of them are not invisible` to `Fetched something but all of them are invisible`.
Bug: Involve more info from the analysis of logical plan to the graph ui in sql tab
Status: Open | Priority: Major | Resolution: 
Created: 06/Aug/24 05:59
Affects Version/s: 3.5.3

Currently we can't get the stats from sql tab to get more useful info in the sql tab, which only includes the physical plan metrics.

So we need a way to involve more info into the tab, which can give a readable in the sql tab, which can help the engineer analysis the plan of whole life cycle.

We can take the below scenario as an example:

!image-2024-08-06-15-42-09-729.png!

We can see there is a new row in the Metric graph: sizeInBytes, which is a stats metric from logical plan, which means we will find this can check whether the  rule applied right or not, or whether the rule works as expected.
Bug: Memory Used in Spark UI using the Custom spark listener
Status: Open | Priority: Major | Resolution: 
Created: 03/Aug/24 12:16
Affects Version/s: 3.3.1

Hi, I was trying to write a custom spark listener to get the metrics which are available in Spark UI. I'm kinda lost at memory used part of the executors tab. How is it calculated when i tried to take sum of all on heap and off heap gave a very large number compared to the one in UI. Please guide me here.
Bug: Remove bouncycastle-related test dependencies from hive-thriftserver module
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Aug/24 03:28
Affects Version/s: 4.0.0

After SPARK-49066 merged, other than `OrcEncryptionSuite`, the test cases for writing Orc data no longer require the use of `FakeKeyProvider`. As a result, `hive-thriftserver` no longer needs these test dependencies.
Bug: Spark jobs failing with UnknownHostException on executors if driver readiness timeout elapsed
Status: Open | Priority: Major | Resolution: 
Created: 01/Aug/24 10:34
Affects Version/s: 3.5.0

We have seen cases where Spark jobs would fail to run in case ExecutorPodsAllocator times out while waiting for the driver pod to get to the READY status. If that happens, we have seen 2 possible scenarios leading to the same result (executors failing with an UnknownHostException trying to resolve the k8s service for spark driver and the job failing because the maximum number of executor failures was reached):
 * Kubernetes service not getting created (confirmed that with the k8s service created event/metric available in grafana)
 * Kubernetes service being there but still not being able to resolve the hostname in the executors (maybe the service being fully available only when driver pod got ready and executors tried to resolve the hostname prior to that)

The particular part of the code under question is [https://github.com/apache/spark/blob/v3.5.0/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala#L130] 
{code:java}
    driverPod.foreach { pod =>
      // Wait until the driver pod is ready before starting executors, as the headless service won't
      // be resolvable by DNS until the driver pod is ready.
      Utils.tryLogNonFatalError {
        kubernetesClient
          .pods()
          .inNamespace(namespace)
          .withName(pod.getMetadata.getName)
          .waitUntilReady(driverPodReadinessTimeout, TimeUnit.SECONDS)
      }
    } {code}
Interestingly enough the comment says wait until the driver pod otherwise the service will not be resolvable by executors, but we still let the run to continue.

Also worth mentioning the documentation around such readiness timeout config ([https://github.com/apache/spark/blob/v3.5.0/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala#L454])
{code:java}
  val KUBERNETES_ALLOCATION_DRIVER_READINESS_TIMEOUT =
    ConfigBuilder("spark.kubernetes.allocation.driver.readinessTimeout")
      .doc("Time to wait for driver pod to get ready before creating executor pods. This wait " +
        "only happens on application start. If timeout happens, executor pods will still be " +
        "created.")
      .version("3.1.3")
      .timeConf(TimeUnit.SECONDS)
      .checkValue(value => value > 0, "Allocation driver readiness timeout must be a positive "
        + "time value.")
      .createWithDefaultString("1s") {code}
Please note the "If timeout happens, executor pods will still be created", which conflicts (at least in my head) with the code comment on the await we have for the driver pod.

The question would be, is this intended behaviour? Looks like a bug, maybe we should check before creating the executors once again whether driver pod is ready and otherwise fail the job?

For now trying to mitigate by increasing the readiness timeout value as a bandaid fix.
Bug: How to restart failed spark stream job from the failure point
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Aug/24 19:10
Affects Version/s: 3.5.0

I am setting up a ETL process using pyspark. My input is a kafka stream and i am writing output to multiple sink (one into kafka and another into cloud storage). I am writing checkpoints on the cloud storage. The issue i am facing is that, whenever my application is getting failed due to some reason and when i am restarting my application then, my pyspark application is again reprocessing some (not all) of the input stream data causing data redundancy. Is there any way i can avoid this. I am using spark 3.5.0 and python 3.11. Below are some of my application code:

Spark Session :

spark = SparkSession \
.builder \
.appName("ETL") \
.config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.2.2") \
.config('spark.hadoop.fs.s3a.aws.credentials.provider', 'org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider')\
.config('spark.driver.extraJavaOptions', '-Duser.timezone=GMT') \
.config('spark.executor.extraJavaOptions', '-Duser.timezone=GMT') \
.config('spark.sql.session.timeZone', 'UTC') \
.config('spark.hadoop.fs.s3a.buffer.dir', '/tmp,/mnt/tmp') \
.config('spark.hadoop.fs.s3a.fast.upload.buffer', 'bytebuffer') \
.config('spark.hadoop.fs.s3a.fast.upload.active.blocks', 1) \
.config('spark.streaming.backpressure.enabled', True) \
.config("spark.redis.host",conf["nosql-host"]) \
.config("spark.redis.port",conf["nosql-port"]) \
.config("spark.redis.db",conf["nosql-db"]) \
.config("spark.redis.auth", __REDIS_CREDENTIAL__) \
.getOrCreate()  

 

Kafka Read Stream :

streamDF = (spark \
.readStream \
.format("kafka") \
.option("kafka.bootstrap.servers", kafka_bootstrap_server_consumer) \
.option("subscribe", kafka_topic_name) \
.option("mode", "PERMISSIVE") \
.option("startingOffsets", "earliest").option("failOnDataLoss", "false") \
.load().withColumn('fixedValue', fn.expr("substring(value, 6, length(value)-5)")).select('fixedValue'))

 

Write Stream to multiple sinks :

write_stream = extractionDF \
.writeStream \
.trigger(processingTime='2 seconds') \
.outputMode("append") \
.foreachBatch(lambda df,epochId: write_to_multiple_sinks(df, epochId,processed_cloud_storage_path,kafka_bootstrap_server_producer)) \
.option("truncate", "false").option("checkpointLocation", cloud_storage_path)\
.start()

write_to_multiple_sinks Function :

def write_to_multiple_sinks(dataframe: DataFrame, epochId,cloud_storage_path, kafka_bootstrap_server):
dataframe = dataframe.cache()
druidDF = dataframe.select(druidSchema())

druidDF.selectExpr(producerTopic,"to_json(struct(*)) AS value").write\
.format("kafka")\
.option("kafka.bootstrap.servers", kafka_bootstrap_server).save()

processedDF = dataframe.select(processedSchema())
processedDF.write.format("csv").mode("append").option("sep","^").option("compression","gzip").option("path", cloud_storage_path).save()
Bug: Deduplicate Scala UDF handling in the SparkConnectPlanner
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Aug/24 19:53
Affects Version/s: 4.0.0

SparkConnectPlanner.transformScalaUDF and SparkConnectPlanner.transformScalaFunction do duplicate work. transformScalaFunction should create the UDF, and transformScalaUDF should invoke it.
Bug: Additional JDBC dialects not registered in executor
Status: Open | Priority: Minor | Resolution: 
Created: 20/Aug/24 13:56
Affects Version/s: 3.5.0

While working on a Spark application, I created a custom JDBC dialect and registered before creating a {{SparkSession}}. Spark did not use the dialect's {{JdbcSQLQueryBuilder}}, however, so I investigated further. I discovered that {{JDBCRDD}} [repeats the dialect resolution|https://github.com/apache/spark/blame/fe2174d8caaf6f9474319a8d729a2f537038b4c1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala#L177], and this runs on the executor.
Bug: Add OpenTelemetryPush Sink
Status: Reopened | Priority: Minor | Resolution: 
Created: 14/Aug/24 22:18
Affects Version/s: 4.0.0

From OpenTelemetry side, they have added the support from a pull model: [https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/apachesparkreceiver]. However, in my case, the internal team only accept the push model. Therefore,  OpenTelemetryPushSink and OpenTelemetryPushReporter will be helpful in such scenarios.
Bug: Spark Connect ExecuteThreadRunner promise will always complete with success.
Status: Closed | Priority: Minor | Resolution: Abandoned
Created: 28/Aug/24 18:01
Affects Version/s: 4.0.0

{code:java}
private class ExecutionThread(onCompletionPromise: Promise[Unit])
    extends Thread(s"SparkConnectExecuteThread_opId=${executeHolder.operationId}") {
  override def run(): Unit = {
    try {
      execute()
      onCompletionPromise.success(())
    } catch {
      case NonFatal(e) =>
        onCompletionPromise.failure(e)
    }
  }
}{code}
 

execute method end with ErrorUtils.handleError() function call.  if any excetion throw. it will not catch by promise. is it better to catch real exceptions with promises instead of？  if wants. i will submit this change.
Bug: Size based partition creation during kafka read
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Aug/24 09:04
Affects Version/s: 4.0.0

Currently Spark + kafka structured streaming provides *minPartitions* config to create more number of partitions than kafka has. This is helpful to increase parallelism but this value is can not be changed dynamically. 

It would be better to dynamically increase spark partitions based on input size, if input size is high create more partitions. We can take *avg msg size* and *maxBytesPerPartition* as input and dynamically create partitions to handle varying loads.
Bug: Univocity parser handles ArrayIndexOutOfBounds exception
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Aug/24 09:18
Affects Version/s: 3.4.3

The current implementation of `UnivocityParser` throws `ArrayIndexOutOfBounds` exception when parsing a csv record with more columns than set in options as maximum. This case was reproduced in the `UnivocityParserSuite`.
Bug: Cache Table with Parameter markers returns wrong error
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Aug/24 22:23
Affects Version/s: 4.0.0

When investigating the OSS structure of code it was found that `CacheTableAsSelect` when used with parameter markers in the select part of the query fails with `UNBOUND_SQL_PARAMETER` even though logically it should fail with `UNSUPPORTED_FEATURE.PARAMETER_MARKER_IN_UNEXPECTED_STATEMENT`. The reason for the second error is that `CacheTableAsSelect` creates a temporary view which should follow rules for parameter markers as views does.
Bug: Add `OpenContainers` Annotations in docker images
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Aug/24 04:04
Affects Version/s: 4.0.0

h3. BEFORE
{code:java}
$ docker inspect apache/spark:3.5.2 | jq '.[0].Config.Labels'
{
  "org.opencontainers.image.ref.name": "ubuntu",
  "org.opencontainers.image.version": "20.04"
} {code}
h3. AFTER
{code:java}
$ NO_MANUAL=1 ./dev/make-distribution.sh -Pkubernetes
$ bin/docker-image-tool.sh -t SPARK-49243 -n build
$ docker inspect spark:SPARK-49243 | jq '.[0].Config.Labels'
{
  "org.opencontainers.image.authors": "Apache Spark project <dev@spark.apache.org>",
  "org.opencontainers.image.licenses": "Apache-2.0",
  "org.opencontainers.image.ref.name": "Apache Spark Scala/Java Image",
  "org.opencontainers.image.version": ""
} {code}
Bug: Performance improvement for window rangeBetween then Filter queries
Status: Open | Priority: Minor | Resolution: 
Created: 30/Aug/24 10:42
Affects Version/s: 4.0.0

There is a performance improvement opportunity for a special query where rangeBetween window function is followed by a filter expression.

 

Consider the following query:

 
{code:java}
import org.apache.spark.sql.expressions.Window
val df = spark.range(10).withColumnRenamed("id", "day_index").withColumn("sales", lit(10))
df.write.parquet("sales")
 
val w = Window.partitionBy().orderBy("day_index").rangeBetween(-1, 0)
val res = spark.read.parquet("sales").withColumn("moving_average_sales", mean("sales").over(w)).filter("day_index = 9")
res.explain
{code}
 
{code:java}
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- Filter (isnotnull(day_index#74L) AND (day_index#74L = 9))
   +- Window [avg(sales#75) windowspecdefinition(day_index#74L ASC NULLS FIRST, specifiedwindowframe(RangeFrame, -1, currentrow$())) AS moving_average_sales#78], [day_index#74L ASC NULLS FIRST]
      +- Sort [day_index#74L ASC NULLS FIRST], false, 0
         +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=185]
            +- FileScan parquet [day_index#74L,sales#75] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/home/koray/Desktop/libs/spark/sales], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<day_index:bigint,sales:int>{code}
 

In this case, we have filtered for day_index = 9. Since the window is rangeBetween(-1, 0), we can filter the input dataframe for only day_index IS IN (8, 9). In general, Window orderBy column and the filter column should be the same for this to work. 

 

This is especially useful for calculating lags, moving averages for specific dates (today for example).

 

Do you think it is possible/doable to inject the filter
{code:java}
day_index IS IN (8, 9) {code}
so that the filter will be pushed down? Is this something we can do with Catalyst with a reasonable effort?

 

[~gurwls223] [~cloud_fan] tagging to get your opinion on this.
Bug: ProtobufFunctionsSuite test case should not be environment-dependent
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Aug/24 19:22
Affects Version/s: 4.0.0

During debugging CI failure in https://github.com/apache/spark/pull/47843, we found that `ProtobufFunctionsSuite`'s test case `SPARK-49121: from_protobuf and to_protobuf SQL functions` is environment-dependent.

In the test, it checks the start and stop indices of SQL text fragment but the fragment length depends on the repo name of the author of a PR.