av9ash/GitBugs huggingface dataset:
Bug: Skipped stages are inconsitent on UI
Status: Open | Priority: Minor | Resolution: 
Created: 19/Nov/24 15:07
Affects Version/s: 4.0.0

The following snipett produces inconsitent UI state:
{noformat}
import scala.collection.mutable
import scala.concurrent.{ExecutionContext, Future, Await}
import scala.concurrent.duration.Duration

val rdd = sc.parallelize(1 to 1000, 100).repartition(10)
rdd.count

val futures = mutable.ListBuffer.empty[Future[Unit]]
implicit val xc = ExecutionContext.fromExecutorService(java.util.concurrent.Executors.newFixedThreadPool(5))
futures += Future { rdd.count }
futures += Future { rdd.count }
futures += Future { rdd.count }
Await.result(Future.sequence(futures), Duration.Inf)
{noformat}

The jobs page doesn't show the skipped tasks oj job 2 and job 3 properly:
!Screenshot 2024-11-19 at 16.06.48.png!


The job 1 page looks good:
!Screenshot 2024-11-19 at 16.07.06.png!


But job 2 and 3 pages are inconsistent in DAG and skipped stages list:
!Screenshot 2024-11-19 at 16.07.20.png!


The issue affects only the UI, the result of the jobs are correct.
Without parallel submission of jobs the UI is consistent.
Bug: Make Spark's label text colorful
Status: In Progress | Priority: Minor | Resolution: 
Created: 27/Nov/24 09:16
Affects Version/s: 4.0.0

As we have seen in other code (eg: `iceberg`) repositories, GitHub can {color:#ff0000}color{color} the PR label as follows:

[https://github.com/apache/iceberg/pulls]

!image-2024-11-27-17-17-48-394.png|width=653,height=624!

Do we have to add this feature to Spark's repository?

 

The advantage is that it makes it easier for code reviewers to find the tags they need to pay attention to, which are eye-catching.

 

We can update the color of the label through the following url:

 [https://github.com/apache/spark/labels]

 

{color:#00875a}*Label Dyeing Record*{color}
||Label Name||Color||
|CORE|#0E8A16|
|DOCS|#79CE54|
|PYTHON|#FBCA04|
|SQL|#E99695|
|CONNECT|#1D76DB|
Bug: The staged table interface does not return metrics
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Nov/24 21:07
Affects Version/s: 3.5.3

When a change is made to a table using the [StagedTable|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/StagedTable.java] interface, no metrics are available for the change like how many rows have been added.
Bug: Spark App executes unintended commands
Status: Open | Priority: Minor | Resolution: 
Created: 08/Nov/24 07:01
Affects Version/s: 3.5.3

Spark commits a task and Yarn executes the command directly using bash -c, if the commit contains ``` it will execute the statement directly, this has some impact on Yarn, does Saprk need to intercept this case

hadoop ContainerExecutor.java
{code:java}
ContainerLaunch.ShellScriptBuilder sb =
    ContainerLaunch.ShellScriptBuilder.create();
...
sb.command(command);

PrintStream pout = null;
try {
  pout = new PrintStream(out, false, "UTF-8");
  sb.write(pout);
} finally {
  if (out != null) {
    out.close();
  }
}{code}
Bug: accessing a MySQL database through a proxy that has a set timeout, an error occurs when using df.write.jdbc
Status: Open | Priority: Minor | Resolution: 
Created: 25/Nov/24 08:32
Affects Version/s: 3.1.0

When accessing a MySQL database through a proxy that has a set timeout, an error occurs when using {{{}df.write.mode(SaveMode.Append).jdbc(url, table, properties){}}}.

error msg :  !图片1.png!
 

The reason for this error is that the JdbcRelationProvider created a connection on the driver side, but did not close it after use; instead, it was closed after the saveTable method was called. At this point, Nginx detected that there was no data transfer on the connection and directly closed the connection, leading to an error when calling conn.close(): "no more data to read from socket".
Bug: javaoptions execution method.
Status: Open | Priority: Minor | Resolution: 
Created: 19/Nov/24 09:00
Affects Version/s: 3.5.3

Executing javaOptions as bash -c has an execution problem that causes the commands in `` to be executed

After committing to Yarn, the commands in `` are executed, when in reality I only want to execute them after the JVM crashes.
Bug: Set max value for spark.sql.autoBroadcastJoinThreshold to 8gb if more than 8gb is passed
Status: Open | Priority: Minor | Resolution: 
Created: 19/Nov/24 08:26
Affects Version/s: 3.5.3

The default value for spark.sql.autoBroadcastJoinThreshold is 10m. But user can override it during spark-submit. User can provide any value, there is no validation.

for ex: user can pass 16g. During the execution, we validate if the data size is <8g ([code refernece|[https://github.com/apache/spark/blob/d39f5ab99f67ce959b4379ecc3d6e262c10146cf/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala#L229]).] 

If the data size is >8g, the job fails with an exception "
Cannot broadcast the table that is larger than...".
 
Instead of failing at runtime, the improvement suggested in this Jira is to set the autoBroadcastJoinThreshold to 8GB if the passed value is >8GB.
Bug: bash -c execute command risk.
Status: Open | Priority: Minor | Resolution: 
Created: 18/Nov/24 12:04
Affects Version/s: 3.5.3

bash -c executing user input is risky, does Spark need to be guarded? For example, the execution of a spark app in yarn mode executes input directly with bash
YarnSparkHadoopUtil.scala

 
{code:java}
/**
 * Escapes a string for inclusion in a command line executed by Yarn. Yarn executes commands
 * using either
 *
 * (Unix-based) `bash -c "command arg1 arg2"` and that means plain quoting doesn't really work.
 * The argument is enclosed in single quotes and some key characters are escaped.
 *
 * (Windows-based) part of a .cmd file in which case windows escaping for each argument must be
 * applied. Windows is quite lenient, however it is usually Java that causes trouble, needing to
 * distinguish between arguments starting with '-' and class names. If arguments are surrounded
 * by ' java takes the following string as is, hence an argument is mistakenly taken as a class
 * name which happens to start with a '-'. The way to avoid this, is to surround nothing with
 * a ', but instead with a ".
 *
 * @param arg A single argument.
 * @return Argument quoted for execution via Yarn's generated shell script.
 */
def escapeForShell(arg: String): String = {
  if (arg != null) {
    if (Utils.isWindows) {
      YarnCommandBuilderUtils.quoteForBatchScript(arg)
    } else {
      val escaped = new StringBuilder("'")
      arg.foreach {
        case '$' => escaped.append("\\$")
        case '"' => escaped.append("\\\"")
        case '\'' => escaped.append("'\\''")
        case c => escaped.append(c)
      }
      escaped.append("'").toString()
    }
  } else {
    arg
  }
} {code}
Bug: DirectByteBufferOutputStream may crash if used after close
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Nov/24 23:29
Affects Version/s: 4.0.0

`DirectByteBufferOutputStream#close()` calls `StorageUtils.dispose()` to free its direct byte buffer. This puts the object into an unspecified and dangerous state after being closed, and can cause unpredictable JVM crashes if it the object is used after close.

We can make this safer by modifying `close()` to place the object into a known-closed state, and modify all methods to assert not closed.
Bug: Use Spark 3.4.4 instead of 3.0.1 in `test_install_spark`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Nov/24 05:09
Affects Version/s: 3.5.3

[https://github.com/apache/spark/actions/runs/11623974780/job/32371883850]

 
{code:java}
urllib.error.URLError: <urlopen error [Errno 110] Connection timed out>
ERROR
test_package_name (pyspark.tests.test_install_spark.SparkInstallationTestCase) ... Trying to download Spark spark-3.0.1 from [https://dlcdn.apache.org/, https://archive.apache.org/dist, https://dist.apache.org/repos/dist/release]
Downloading spark-3.0.1 for Hadoop hadoop3.2 from:
- https://dlcdn.apache.org//spark/spark-3.0.1/spark-3.0.1-bin-hadoop3.2.tgz
Failed to download spark-3.0.1 for Hadoop hadoop3.2 from https://dlcdn.apache.org//spark/spark-3.0.1/spark-3.0.1-bin-hadoop3.2.tgz:
Downloading spark-3.0.1 for Hadoop hadoop3.2 from:
- https://archive.apache.org/dist/spark/spark-3.0.1/spark-3.0.1-bin-hadoop3.2.tgz
Failed to download spark-3.0.1 for Hadoop hadoop3.2 from https://archive.apache.org/dist/spark/spark-3.0.1/spark-3.0.1-bin-hadoop3.2.tgz:
Downloading spark-3.0.1 for Hadoop hadoop3.2 from:
- https://dist.apache.org/repos/dist/release/spark/spark-3.0.1/spark-3.0.1-bin-hadoop3.2.tgz
Failed to download spark-3.0.1 for Hadoop hadoop3.2 from https://dist.apache.org/repos/dist/release/spark/spark-3.0.1/spark-3.0.1-bin-hadoop3.2.tgz:
ok {code}
Bug: PySpark Pipe Method
Status: Open | Priority: Trivial | Resolution: 
Created: 24/Nov/24 21:39
Affects Version/s: 3.5.3

Implement pipe method for classic PySpark DataFrame. Similar to the transform method, this method can be called directly on a DataFrame to perform custom transformations functions. However, unlike the current transform method which requires one call per custom transformation, pipe can accept an ambiguous number of transformations and chain them together on the user's behalf.
Bug: The accumulator of ArrayAggregate should copy the intermediate result if string, struct, array, or map
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 25/May/22 22:27
Affects Version/s: 3.0.3

The accumulator of ArrayAggregate should copy the intermediate result if string, struct, array, or map.

{code:scala}
import org.apache.spark.sql.functions._

val reverse = udf((s: String) => s.reverse)

val df = Seq(Array("abc", "def")).toDF("array")
val testArray = df.withColumn(
  "agg",
  aggregate(
    col("array"),
    array().cast("array<string>"),
    (acc, s) => concat(acc, array(reverse(s)))))

aggArray.show(truncate=false)
{code}

should be:

{code}
+----------+----------+
|array     |agg       |
+----------+----------+
|[abc, def]|[cba, fed]|
+----------+----------+
{code}

but:

{code}
+----------+----------+
|array     |agg       |
+----------+----------+
|[abc, def]|[fed, fed]|
+----------+----------+
{code}
Bug: Spark tasks stuck forever due to deadlock between TaskMemoryManager and UnsafeExternalSorter
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 25/May/22 03:05
Affects Version/s: 3.0.0

We are seems this deadlock between {{TaskMemoryManager}} and {{UnsafeExternalSorter}} pretty often on our workload. Sometime, the retry is successful but sometimes we have to do hacky ways to break the deadlocks such as turning down the worker machines explicitly. 

Below is the thread dump from the Spark UI showing the deadlock :

!DeadlockSparkTasks.png!

 

I believe there was a related Jira on the similar deadlock between the same threads and it was resolved. 
https://issues.apache.org/jira/browse/SPARK-27338
Bug: Make unionByName null-filling behavior work with Array/Array of Structs columns
Status: Open | Priority: Major | Resolution: 
Created: 02/May/22 21:00
Affects Version/s: 3.1.0

https://issues.apache.org/jira/browse/SPARK-29358 added support for unionByName to work when the two datasets didn't necessarily have the same schema, but it does not work with nested array columns.

This has been solved for Structs. https://issues.apache.org/jira/browse/SPARK-32376
Bug: Support returning a table or set of rows in CREATE FUNCTION
Status: Open | Priority: Major | Resolution: 
Created: 21/May/22 10:25
Affects Version/s: 3.3.0

The Databricks version supports create a function which returns a result set:
_"Creates a SQL scalar or table function that takes a set of arguments and returns a scalar value or a set of rows."_
Such functions are very useful as the user can pass a list of parameters which can be used in the function to create a dynamic result set (a query which will use the passed parameters). 

See the example from databricks:
{_}"{_}{_}-- Produce all weekdays between two dates > CREATE FUNCTION weekdays(start DATE, end DATE) RETURNS TABLE(day_of_week STRING, day DATE) RETURN SELECT extract(DAYOFWEEK_ISO FROM day), day FROM (SELECT sequence(weekdays.start, weekdays.end)) AS T(days) LATERAL VIEW explode(days) AS day WHERE extract(DAYOFWEEK_ISO FROM day) BETWEEN 1 AND 5;"{_}
Bug: Pyspark failed to serialize dates before 1970 in windows
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/May/22 08:17
Affects Version/s: 3.0.1

h3. Fix problems with pyspark in Windows
 # Fixed datetime conversion to timestamp before 1970;
 # Fixed datetime conversion when timestamp is negative;
 # Adding a test script.

h3. Pyspark has problems serializing pre-1970 times in Windows

An exception occurs when executing the following code under Windows:
{code:java}
rdd = sc.parallelize([('a', datetime(1957, 1, 9, 0, 0)),
                      ('b', datetime(2014, 1, 27, 0, 0))])
df = spark.createDataFrame(rdd, ["id", "date"])

df.show()
df.printSchema()

print(df.collect()){code}
{code:java}
  File "...\spark\python\lib\pyspark.zip\pyspark\sql\types.py", line 195, in toInternal
    else time.mktime(dt.timetuple()))
OverflowError: mktime argument out of range

at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:503)
   at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:638)
   at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:621)
   at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:456)
   at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
   at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:489)
   at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
   at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
   at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
   at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
   at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729)
   at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:340)
   at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
   at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
   at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
   at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
   at org.apache.spark.scheduler.Task.run(Task.scala:127)
   at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
   at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   ... 1 more {code}
_*and*_
{code:java}
File ...\spark\python\lib\pyspark.zip\pyspark\sql\types.py, in fromInternal:
Line 207:   return datetime.datetime.fromtimestamp(ts // 1000000).replace(microsecond=ts % 1000000)

OSError: [Errno 22] Invalid argument {code}
Bug: JDBCTable support properties
Status: In Progress | Priority: Major | Resolution: 
Created: 10/May/22 08:37
Affects Version/s: 3.3.0


{code:java}
         >
         > desc formatted jdbc.test.people;
NAME	string
ID	int

# Partitioning
Not partitioned

# Detailed Table Information
Name	test.people
Table Properties	[]
Time taken: 0.048 seconds, Fetched 9 row(s)
{code}
Bug: Speed up Timestamp type inference of legacy format in JSON/CSV data source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/May/22 02:51
Affects Version/s: 3.4.0

The optimization of {{DefaultTimestampFormatter}} has been implemented in [#36562|https://github.com/apache/spark/pull/36562] , this ticket adds the optimization of legacy format. The basic logic is to prevent the formatter from throwing exceptions, and then use catch to determine whether the parsing is successful.
Bug: Fix remote table location based on database location
Status: Reopened | Priority: Major | Resolution: 
Created: 17/May/22 03:25
Affects Version/s: 2.2.0

We have HDFS and Hive on cluster A. We have Spark on cluster B and need to read data from cluster A. The table location is incorrect:
{noformat}
spark-sql> desc formatted  default.test_table;
fas_acct_id         	decimal(18,0)
fas_acct_cd         	string
cmpny_cd            	string
entity_id           	string
cre_date            	date
cre_user            	string
upd_date            	timestamp
upd_user            	string

# Detailed Table Information
Database             default
Table               	test_table
Type                	EXTERNAL
Provider            	parquet
Statistics          	25310025737 bytes
Location            	/user/hive/warehouse/test_table
Serde Library       	org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
InputFormat         	org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
OutputFormat        	org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
Storage Properties  	[compression=snappy]

spark-sql> desc database default;
Namespace Name	default
Comment
Location	viewfs://clusterA/user/hive/warehouse/
Owner     hive_dba
{noformat}

The correct table location should be viewfs://clusterA/user/hive/warehouse/test_table.
Bug: When we look at spark UI or History, we can see the failed tasks first
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 11/May/22 18:39
Affects Version/s: 3.2.0

When a task fails, users are more concerned about the causes of failed tasks and failed tasks. The Current Spark UI and History are sorted according to "Index" rather than "Errors". When a large number of tasks are sorted, you need to wait a certain period for tasks to be sorted. In order to find the cause of Errors for failed tasks, we can improve the user experience by specifying sorting by the "Errors" column at the beginning
Bug: Merge PromotePrecision and CheckOverflow into decimal binary arithmetic
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/May/22 08:28
Affects Version/s: 3.4.0

Fix the bug of `TypeCoercion`, for example:

{code:java}
  SELECT CAST(1 AS DECIMAL(28, 2))
  UNION ALL
  SELECT CAST(1 AS DECIMAL(18, 2)) / CAST(1 AS DECIMAL(18, 2));
{code}

The union result data type is not correct according to the formula:

|| Operation || Result Precision || Result Scale ||
|e1 union e2 | max(s1, s2) + max(p1-s1, p2-s2) | max(s1, s2) |


{code:java}
-- before
-- query schema
decimal(28,2)
-- query output
1.00
1.00

-- after
-- query schema
decimal(38,20)
-- query output
1.00000000000000000000
1.00000000000000000000
{code}
Bug: Cannot refer to nested CTE within a nested CTE in a subquery.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/May/22 17:39
Affects Version/s: 3.2.1

The following query fails with {color:#ff0000}Table or view not found: cte1;{color}
{code:java}
set spark.sql.legacy.ctePrecedencePolicy=CORRECTED;
with
cte1 as (select 1)
select * from (
  with
    cte2 as (select * from cte1)
    select * from cte2
); {code}
Or Spark 3.1.1 it returns 1 as expected.

This is related to SPARK-38404, but different, since the query fails with Spark built from 'master' (commit 17b85ff9). The [PR #36146|https://github.com/apache/spark/pull/36146] therefore does not fix this issue.
Bug: Create table in overwrite mode fails when interrupted
Status: Open | Priority: Major | Resolution: 
Created: 31/May/22 13:21
Affects Version/s: 3.1.1

When you attempt to rerun an Apache Spark write operation by cancelling the currently running job, the following error occurs:
{code:java}
Error: org.apache.spark.sql.AnalysisException: Cannot create the managed table('`testdb`.` testtable`').
The associated location ('dbfs:/user/hive/warehouse/testdb.db/metastore_cache_ testtable) already exists.;{code}
This problem can occur if:
 * The cluster is terminated while a write operation is in progress.
 * A temporary network issue occurs.
 * The job is interrupted.

You can reproduce the problem by following these steps:

1. Create a DataFrame:
{code:java}
val df = spark.range(1000){code}
2. Write the DataFrame to a location in overwrite mode:
{code:java}
df.write.mode(SaveMode.Overwrite).saveAsTable("testdb.testtable"){code}
3. Cancel the command while it is executing.

4. Re-run the {{write}} command.
Bug: Makes DPP support the pruning side has Union
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/22 05:16
Affects Version/s: 3.4.0

Support the following case:

{noformat}
SELECT f.store_id,
       f.date_id,
       s.state_province
FROM (SELECT 4 AS store_id,
               date_id,
               product_id
      FROM   fact_sk
      WHERE  date_id >= 1300
      UNION ALL
      SELECT   store_id,
               date_id,
               product_id
      FROM   fact_stats
      WHERE  date_id <= 1000) f
JOIN dim_store s
ON f.store_id = s.store_id
WHERE s.country IN ('US', 'NL')
{noformat}
Bug: ps.read_csv ignore double quotes.
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 26/May/22 13:13
Affects Version/s: 3.4.0

This one is coming from user@spark.org mail list tittle "Complexity with the data" and also on [SO|https://stackoverflow.com/questions/72389385/how-to-load-complex-data-using-pyspark] 

Add a notebook and the sample data, where this error is tested. 


Test data :

Some years,"If your job title needs additional context, please clarify here:","If ""Other,"" please indicate the currency here: "
5-7 years,"I started as the Marketing Coordinator, and was given the ""Associate Product Manager"" title as a promotion. My duties remained mostly the same and include graphic design work, marketing, and product management.",
8 - 10 years,equivalent to Assistant Registrar,
2 - 4 years,"I manage our fundraising department, primarily overseeing our direct mail, planned giving, and grant writing programs. ",
Bug: Silent change in regexp_replace's handling of empty strings
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/22 15:24
Affects Version/s: 3.1.2

Hi, we just upgraded from 3.0.2 to 3.1.2 and noticed a silent behavior change that a) seems incorrect, and b) is undocumented in the [migration guide|https://spark.apache.org/docs/latest/sql-migration-guide.html]:

{code:title=3.0.2}
scala> val df = spark.sql("SELECT '' AS col")
df: org.apache.spark.sql.DataFrame = [col: string]

scala> df.withColumn("replaced", regexp_replace(col("col"), "^$", "<empty>")).show
+---+--------+
|col|replaced|
+---+--------+
|   | <empty>|
+---+--------+
{code}

{code:title=3.1.2}
scala> val df = spark.sql("SELECT '' AS col")
df: org.apache.spark.sql.DataFrame = [col: string]

scala> df.withColumn("replaced", regexp_replace(col("col"), "^$", "<empty>")).show
+---+--------+
|col|replaced|
+---+--------+
|   |        |
+---+--------+
{code}

Note, the regular expression {{^$}} should match the empty string, but doesn't in version 3.1. E.g. this is the Java behavior:

{code}
scala> "".replaceAll("^$", "<empty>");
res1: String = <empty>
{code}