av9ash/GitBugs huggingface dataset:
Bug: [SPARK-35222] In YARN mode, for better user experience, when Spark is started, not only the AppID is printed, but the Tracking URL is also printed to allow users to better track Spark Job
Status: In Progress | Priority: Major | Resolution: 
Created: 25/Apr/21 10:39
Affects Version/s: 2.4.3

In YARN mode, for better user experience, when Spark is started, not only the AppID is printed, but the Tracking URL is also printed to allow users to better track Spark Job
Bug: YearMonthIntervalType/DayTimeIntervalType 's string value not same in spark default serde and hive serde
Status: Open | Priority: Major | Resolution: 
Created: 25/Apr/21 10:00
Affects Version/s: 3.1.1

YearMonthIntervalType/DayTimeIntervalType 's string value not same in spark default serde and hive serde
Bug: all columns are read even if column pruning applies when spark3.0 read table written by spark2.2
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 22/Apr/21 11:01
Affects Version/s: 3.0.0

Before I address this issue, let me talk about the issue background: The current spark version we use is 2.2, and we plan to migrate to spark3.0 in near future. Before migration, we test some query in both spark2.2 and spark3.0 to check potential issue. The data source table of these query is orc format written by spark2.2.

 

I find that even if column pruning is applied, spark3.0’s native reader will read all columns.

 

Then I do remote debug. In OrcUtils.scala’s requestedColumnIds Method, it will check whether field name is started with “_col”. In my case, field name is started with “_col”, like “_col1”, “_col2”. So pruneCols is not done.  The code is below:

 

if (orcFieldNames.forall(_.startsWith("_col"))) {

  // This is a ORC file written by Hive, no field names in the physical schema, assume the

  // physical schema maps to the data scheme by index.

  _assert_(orcFieldNames.length <= dataSchema.length, "The given data schema " +

    s"*$*{dataSchema.catalogString} has less fields than the actual ORC physical schema, " +

    "no idea which columns were dropped, fail to read.")

  // for ORC file written by Hive, no field names

  // in the physical schema, there is a need to send the

  // entire dataSchema instead of required schema.

  // So pruneCols is not done in this case

  Some(requiredSchema.fieldNames.map { name =>

    val index = dataSchema.fieldIndex(name)

    if (index < orcFieldNames.length) {

      index

    } else {

      -1

    }

  }, false)
 
 Although this code comment explains reason, I still do not understand. This issue only happens in this case: spark3.0 uses native reader to read table written by spark2.2. 

 

In other cases, there is no such issue. I do another 2 tests:

Test1: use spark3.0’s hive reader (running with spark.sql.hive.convertMetastoreOrc=false and spark.sql.orc.impl=hive) to read the same table, it only reads pruned columns.

Test2: use spark3.0 to write a table, then use spark3.0’s native reader to read this new table, it only reads pruned columns.

 

This issue I mentioned is a block we use native reader in spark3.0. Can anyone know further reason or provide solutions?
Bug: `OneVsRest` classifier uses incorrect data type for `rawPrediction` column
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Apr/21 01:29
Affects Version/s: 3.0.0

`OneVsRest` classifier uses an incorrect data type for the `rawPrediction` column.

 Code to reproduce the issue:
{code:java}
from pyspark.ml.classification import LogisticRegression, OneVsRest
from pyspark.ml.linalg import Vectors
from pyspark.sql import SparkSession
from sklearn.datasets import load_iris

spark = SparkSession.builder.getOrCreate()

X, y = load_iris(return_X_y=True)
df = spark.createDataFrame(
 [(Vectors.dense(features), int(label)) for features, label in zip(X, y)], ["features", "label"]
)
train, test = df.randomSplit([0.8, 0.2])
lor = LogisticRegression(maxIter=5)
ovr = OneVsRest(classifier=lor)
ovrModel = ovr.fit(train)
pred = ovrModel.transform(test)

pred.printSchema()
# This prints out:
# root
#  |-- features: vector (nullable = true)
#  |-- label: long (nullable = true)
#  |-- rawPrediction: string (nullable = true)  # <- should not be string
#  |-- prediction: double (nullable = true)

# pred.show()  # this fails because of the incorrect datatype{code}
I ran the code above using GitHub Actiosn:

[https://github.com/harupy/SPARK-35142/pull/1]

 

It looks like the UDF to compute the `rawPrediction` column is generated without specyfing the return type:
 [https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/python/pyspark/ml/classification.py#L3154]
{code:java}
rawPredictionUDF = udf(func)
{code}
Bug: Avoid to recompute the pending speculative tasks in the ExecutorAllocationManager and remove unnecessary code
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Apr/21 03:05
Affects Version/s: 3.0.2

The number of the pending speculative tasks is recomputed in the ExecutorAllocationManager to calculate the maximum number of executors required. While , it only needs to be computed once to improve  performance.
Bug: Scala/Java compatibility issue Re: how to use externalResource in java transformer from Scala Transformer?
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 22/Apr/21 16:22
Affects Version/s: 3.1.1

I am trying to make a custom transformer use an externalResource, as it requires a large table to do the transformation. I'm not super familiar with scala syntax, but from snippets found on the internet I think I've made a proper java implementation. I am running into the following error:

Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Param HardMatchDetector_d95b8f699114__externalResource does not belong to HardMatchDetector_d95b8f699114.
 at scala.Predef$.require(Predef.scala:281)
 at org.apache.spark.ml.param.Params.shouldOwn(params.scala:851)
 at org.apache.spark.ml.param.Params.set(params.scala:727)
 at org.apache.spark.ml.param.Params.set$(params.scala:726)
 at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:41)
 at org.apache.spark.ml.param.Params.set(params.scala:713)
 at org.apache.spark.ml.param.Params.set$(params.scala:712)
 at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:41)
 at HardMatchDetector.setResource(HardMatchDetector.java:45)

 

Code as follows:
{code:java}
public class HardMatchDetector extends Transformer implements DefaultParamsWritable, DefaultParamsReadable, Serializable {
public String inputColumn = "value";
 public String outputColumn = "hardMatches";
 private ExternalResourceParam resourceParam = new ExternalResourceParam(this, "externalResource", "external resource, parquet file with 2 columns, one names and one wordcount");;
 private String uid;
public HardMatchDetector setResource(final ExternalResource value)
{ return (HardMatchDetector)this.set(this.resourceParam, value); }
public HardMatchDetector setResource(final String path)
{ return this.setResource(new ExternalResource(path, ReadAs.TEXT(), new HashMap())); }
@Override
 public String uid()
{ return getUid(); }
private String getUid() {
 if (uid == null)
{ uid = Identifiable$.MODULE$.randomUID("HardMatchDetector"); }
return uid;
 }
@Override
 public Dataset<Row> transform(final Dataset<?> dataset)
{ return dataset; }
@Override
 public StructType transformSchema(StructType schema)
{ return schema.add(DataTypes.createStructField(outputColumn, DataTypes.StringType, true)); }
@Override
 public Transformer copy(ParamMap extra)
{ return new HardMatchDetector(); }
}
public class HardMatcherTest extends AbstractSparkTest
{ @Test 
public void test() 
{ 
var hardMatcher = new HardMatchDetector().setResource(pathName); }
}
{code}
Bug: all columns are read even if column pruning applies when spark3.0 read table written by spark2.2
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 22/Apr/21 11:13
Affects Version/s: 3.0.0

Before I address this issue, let me talk about the issue background: The current spark version we use is 2.2, and we plan to migrate to spark3.0 in near future. Before migration, we test some query in both spark2.2 and spark3.0 to check potential issue. The data source table of these query is orc format written by spark2.2.

 

I find that even if column pruning is applied, spark3.0’s native reader will read all columns.

 

Then I do remote debug. In OrcUtils.scala’s requestedColumnIds Method, it will check whether field name is started with “_col”. In my case, field name is started with “_col”, like “_col1”, “_col2”. So pruneCols is not done.  The code is below:

 

if (orcFieldNames.forall(_.startsWith("_col"))) {

  // This is a ORC file written by Hive, no field names in the physical schema, assume the

  // physical schema maps to the data scheme by index.

  _assert_(orcFieldNames.length <= dataSchema.length, "The given data schema " +

    s"*$*{dataSchema.catalogString} has less fields than the actual ORC physical schema, " +

    "no idea which columns were dropped, fail to read.")

  // for ORC file written by Hive, no field names

  // in the physical schema, there is a need to send the

  // entire dataSchema instead of required schema.

  // So pruneCols is not done in this case

  Some(requiredSchema.fieldNames.map { name =>

    val index = dataSchema.fieldIndex(name)

    if (index < orcFieldNames.length) {

      index

    } else {

      -1

    }

  }, false)
 
 Although this code comment explains reason, I still do not understand. This issue only happens in this case: spark3.0 uses native reader to read table written by spark2.2. 

 

In other cases, there is no such issue. I do another 2 tests:

Test1: use spark3.0’s hive reader (running with spark.sql.hive.convertMetastoreOrc=false and spark.sql.orc.impl=hive) to read the same table, it only reads pruned columns.

Test2: use spark3.0 to write a table, then use spark3.0’s native reader to read this new table, it only reads pruned columns.

 

This issue I mentioned is a block we use native reader in spark3.0. Can anyone know further reason or provide solutions?
Bug: Migrate to transformWithPruning or resolveWithPruning for subquery related rules
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Apr/21 19:28
Affects Version/s: 3.1.0

Migrate transform/resolve functions to transformXxxWithPruning or resolveXxWithPruning, so that we can add tree traversal pruning conditions.

RewriteCorrelatedScalarSubquery will be supported in a separate issue, SPARK-35148.
Bug: Failure on minimal interval literal
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Apr/21 09:52
Affects Version/s: 3.2.0

If the sign '-' inside of interval string, everything is fine after https://github.com/apache/spark/commit/bb5459fb26b9d0d57eadee8b10b7488607eeaeb2:

{code:sql}
spark-sql> SELECT INTERVAL '-178956970-8' YEAR TO MONTH;
-178956970-8
{code}
but the sign outside of interval string is not handled properly:
{code:sql}
spark-sql> SELECT INTERVAL -'178956970-8' YEAR TO MONTH;
Error in query:
Error parsing interval year-month string: integer overflow(line 1, pos 16)

== SQL ==
SELECT INTERVAL -'178956970-8' YEAR TO MONTH
----------------^^^
{code}
Bug: Make graceful shutdown controllable by property
Status: Open | Priority: Major | Resolution: 
Created: 08/Apr/21 22:42
Affects Version/s: 3.1.1

I'm using Spark in embedded mode, i.e. Spark is one of many components in my app.

Currently, when JVM received sigterm, Spark immediately start shutting down.

 

Expected: provide client code with an option to control how and when to shutdown Spark app, i.e. create a config to disable unconditional shutdown by Spark itself.
Bug: Disallow implicit conversion for `String_Column - Date_Column`
Status: In Progress | Priority: Major | Resolution: 
Created: 22/Apr/21 10:31
Affects Version/s: 3.2.0

In Spark 3.2, `String_Column - Date_Column` will cause an analysis exception instead of implicitly converting the first column as Date Type. This is to make it consistent with the behavior of `String_Column - Timestamp_Column`.
Bug: maven autodownload failing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Apr/21 17:26
Affects Version/s: 2.4.7

I attempted to build a fresh clone of Spark using mvn (on two different networks) and got this error:
{noformat}
exec: curl --silent --show-error -L https://www.apache.org/dyn/closer.lua?action=download&filename=/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.
Using `mvn` from path: /tmp/testmvn/spark-mvn-download/build/apache-maven-3.6.3/bin/mvn
build/mvn: line 126: /tmp/testmvn/spark-mvn-download/build/apache-maven-3.6.3/bin/mvn: No such file or directory
{noformat}
if I change the mirror as below, the issue goes away:
{noformat}
-    local APACHE_MIRROR=${APACHE_MIRROR:-'https://www.apache.org/dyn/closer.lua?action=download&filename='}
+    local APACHE_MIRROR=${APACHE_MIRROR:-'https://https://downloads.apache.org'}
{noformat}
Bug: UI progress bar no longer highlights in progress tasks
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Apr/21 21:03
Affects Version/s: 3.1.1

The Spark UI was updated to Bootstrap 4, and during the update the progress bar in the UI was updated to highlight the whole bar once any tasks were in progress, versus highlighting just the number of tasks that were in progress. The was a great visual queue of seeing what percentage of the stage/job was currently being worked on, and it'd be great to get that functionality back.

The change can be found here: https://github.com/apache/spark/pull/27370/files#diff-809c93c57cc59e5fe3c3eb54a24aa96a38147d02323f3e690ae6b5309a3284d2L448
Bug: Add tests for ANSI intervals to HiveThriftBinaryServerSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Apr/21 07:42
Affects Version/s: 3.2.0

Add tests for year-month and day-time intervals to HiveThriftBinaryServerSuite similar to:
# Query Intervals in VIEWs through thrift server
# Support interval type
Bug: Test transferring year-month interval via Hive Thrift server
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/21 09:58
Affects Version/s: 3.2.0

Write a test similar to https://github.com/apache/spark/pull/32121 but for year-month intervals.

Note: this sub-task is blocked by constructing year-month interval column in SQL.
Bug: Add type hints of pyspark.sql.column.* and pyspark.sql.types.*
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 10/Apr/21 02:57
Affects Version/s: 3.1.2

python/pyspark/pandas/spark/functions.py:22: error: Module 'pyspark.sql.column' has no attribute '_to_java_column'
 python/pyspark/pandas/spark/functions.py:22: error: Module 'pyspark.sql.column' has no attribute '_to_seq'
 python/pyspark/pandas/spark/functions.py:22: error: Module 'pyspark.sql.column' has no attribute '_create_column_from_literal'

python/pyspark/pandas/frame.py:6486: error: Module 'pyspark.sql.types' has no attribute '_parse_datatype_string'

 

The error was found when enable the mypy work on pandas intergrations.
Bug: Local mode fails to start cluster due to configuration value escape issue
Status: Open | Priority: Major | Resolution: 
Created: 18/Apr/21 06:07
Affects Version/s: 3.1.1

At least on Windows and perhaps also other systems, running Spark in local mode fails to start a cluster when a configuration key contains a value with "&" (ampersand) in it.

This happens during the "getOrCreate()" call from a Spark session builder.

The reason seems to be incorrect or insufficient escaping since on Windows, the attempt to start a Spark process actually ends up running multiple commands for each occurrence of the "&" character.

On Windows specifically, the correct way to escape "&" would be "^&", but I have not been able yet to figure out exactly where the process is started and how the configuration is passed.
Bug: Initial null value of LiveStage.info can lead to NPE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/21 09:56
Affects Version/s: 3.1.1

The `AppStatusListener.getOrCreateStage` method creates a LiveStage object with the `info` field set to null and right after that set it to a specific StageInfo object. This can lead to a race condition when the `livestages` are read in between those calls. This could then lead to a null pointer exception in, for instance: `activeStages`.
Bug: Parse unit-to-unit interval literals to ANSI intervals
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Apr/21 18:54
Affects Version/s: 3.2.0

Parse unit-to-unit intervals like INTERVAL '1-1' YEAR TO MONTH to either YearMonthIntervalType or DayTimeIntervalType by default. But when spark.sql.legacy.interval.enabled is set to true, parse them to CalendarInterval.
Bug: Migrate CACHE/UNCACHE TABLE to use AnalysisOnlyCommand
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Apr/21 05:08
Affects Version/s: 3.2.0

Now that AnalysisOnlyCommand is introduced, CacheTable/UncacheTable logical plan can extend AnalysisOnlyCommand to clean up the code; currently, there are many places that handle these commands so that the tables in those commands are only analyzed (and not optimized).
Bug: Add column name to SparkUpgradeException about ancient datetime
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Apr/21 18:13
Affects Version/s: 3.2.0

The error message:

{code:java}
org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from Parquet files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set spark.sql.legacy.parquet.datetimeRebaseModeInRead to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during reading. Or set spark.sql.legacy.parquet.datetimeRebaseModeInRead to 'CORRECTED' to read the datetime values as it is.
{code}

doesn't have any clues of which column causes the issue. Need to improve the message and add column name to it.
Bug: The generated data fits the precision of DayTimeIntervalType in spark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Apr/21 13:11
Affects Version/s: 3.2.0

The precision of java.time.Duration is nanosecond, but when it is used as DayTimeIntervalType in Spark, it is microsecond.
At present, the DayTimeIntervalType data generated in the implementation of RandomDataGenerator is accurate to nanosecond, which will cause the DayTimeIntervalType to be converted to long, and then back to DayTimeIntervalType to lose the accuracy, which will cause the test to fail. For example: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137390/testReport/org.apache.spark.sql.hive.execution/HashAggregationQueryWithControlledFallbackSuite/udaf_with_all_data_types/
Bug: Propagate empty relation through Join if join condition is empty
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Apr/21 01:03
Affects Version/s: 3.2.0

{code:scala}
spark.sql("create table t1 using parquet as select id as a, id as b from range(10)")
spark.sql("create table t2 using parquet as select id as a, id as b from range(20)")
spark.sql("select t1.a from t1 left semi join t2").explain(true)
{code}

Current plan:
{noformat}
== Optimized Logical Plan ==
Join LeftSemi
:- Project [a#6L]
:  +- Relation default.t1[a#6L,b#7L] parquet
+- Project
   +- Relation default.t2[a#8L,b#9L] parquet
{noformat}


Excepted plan:
{noformat}
== Optimized Logical Plan ==
Project [a#6L]
+- Relation default.t1[a#6L,b#7L] parquet
{noformat}
Bug: Fix the java.net.BindException when testing with Github Action
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Apr/21 05:11
Affects Version/s: 3.2.0


{noformat}
[info] org.apache.spark.sql.kafka010.producer.InternalKafkaProducerPoolSuite *** ABORTED *** (282 milliseconds)
[info]   java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed after 100 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'sparkDriver' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address.
[info]   at sun.nio.ch.Net.bind0(Native Method)
[info]   at sun.nio.ch.Net.bind(Net.java:461)
[info]   at sun.nio.ch.Net.bind(Net.java:453)
{noformat}
Bug: SparkContext.addJar with an ivy path should not fail with a custom ivySettings file in non-YARN cluster modes
Status: Open | Priority: Major | Resolution: 
Created: 14/Apr/21 15:13
Affects Version/s: 3.2.0

SPARK-33084 introduced support for Ivy paths in sc.addJar or Spark SQL ADD JAR. If we use a custom ivySettings file using spark.jars.ivySettings, it is loaded at https://github.com/apache/spark/blob/b26e7b510bbaee63c4095ab47e75ff2a70e377d7/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L1280. However, this file is only accessible on the client machine. In cluster mode, this file is not available on the driver and so addJar fails. SPARK-34472 provides repro steps for the same.

In SPARK-34472, we decided to restrict the scope of the work to only fix the issue in the YARN resource manager. This fix should also be made to other resource managers which can run in cluster mode.
Bug: A foldable expression could not be replaced by an AttributeReference
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Apr/21 19:37
Affects Version/s: 3.0.0

The PhysicalAggregation pattern can replace a foldable expression with an AttributeReference. This would subsequently fail the checkInputDataTypes() call for certain expressions, e.g., RegExpReplace – `pos` cannot be an AttributeReference.
Bug: Support traversal pruning in transform/resolve functions and their call sites
Status: Open | Priority: Major | Resolution: 
Created: 13/Apr/21 05:40
Affects Version/s: 3.2.0

Transform/resolve functions are called ~280k times per query on average for a TPC-DS query, which are way more than necessary. We can reduce those calls with early exit information and conditions. This [doc|https://docs.google.com/document/d/1SEUhkbo8X-0cYAJFYFDQhxUnKJBz4lLn3u4xR2qfWqk/edit] some evaluation numbers with a prototype.
Bug: Recognize sign before the interval string in literals
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Apr/21 20:03
Affects Version/s: 3.2.0

According to the SQL standard:

{code:java}
<interval literal> ::=
  INTERVAL [ <sign> ] <interval string> <interval qualifier>
<interval string> ::=
  <quote> <unquoted interval string> <quote>
<unquoted interval string> ::=
  [ <sign> ] { <year-month literal> | <day-time literal> }
<sign> ::=
    <plus sign>
  | <minus sign>
{code}
but the parsing fails:

{code:java}
spark-sql> select interval -'1-1' year to month;
Error in query:
mismatched input 'to' expecting {<EOF>, ';'}(line 1, pos 28)

== SQL ==
select interval -'1-1' year to month
----------------------------^^^
{code}