av9ash/GitBugs huggingface dataset:
Bug: Support compact EventLog when there are illegal characters in the path
Status: Closed | Priority: Minor | Resolution: Not A Problem
Created: 26/Mar/22 07:42
Affects Version/s: 2.4.8

When a path has illegal characters, it is escaped.  The file will not be found and compactor  will not work.
 
The path:  "/tmp/app%"
Exception:
java.io.FileNotFoundException: /tmp/app%25/events_7_app.compact.inprogress (No such file or directory)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.(FileOutputStream.java:213)
at java.io.FileOutputStream.(FileOutputStream.java:101)
at org.apache.spark.deploy.history.EventLogFileWriter.initLogFile(EventLogFileWriters.scala:95)
at org.apache.spark.deploy.history.SingleEventLogFileWriter.start(EventLogFileWriters.scala:223)
at org.apache.spark.deploy.history.EventLogFileCompactor.rewrite(EventLogFileCompactor.scala:152)
at org.apache.spark.deploy.history.EventLogFileCompactor.compact(EventLogFileCompactor.scala:94)
Bug: [TESTS] Replace 'abc & Symbol("abc") symbols with $"abc" in tests
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Mar/22 20:01
Affects Version/s: 3.2.1

This ticket is a follow up of SPARK-38351.

 

When building with Scala 2.13 many test classes produce warnings like:
{code:java}
[warn] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:562:11: [deprecation @  | origin= | version=2.13.0] symbol literal is deprecated; use Symbol("d") instead
[warn]           'd.cast("string"),
[warn]           ^
[warn] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:563:11: [deprecation @  | origin= | version=2.13.0] symbol literal is deprecated; use Symbol("e") instead
[warn]           'e.cast("string")).collect())
 {code}
For easier migration to Scala 3.x later it would be good to fix this warnings!

 

Also as suggested by [https://github.com/HeartSaVioR] it would be good to use Spark's $"abc" syntax for columns.
Bug: Validate input dataset of ml.classification
Status: Resolved | Priority: Minor | Resolution: Resolved
Created: 17/Mar/22 10:29
Affects Version/s: 3.4.0

LinearSVC should fail fast if the input dataset contains invalid values.

 
{code:java}
import org.apache.spark.ml.feature._
import org.apache.spark.ml.linalg._
import org.apache.spark.ml.classification._
import org.apache.spark.ml.clustering._
val df = sc.parallelize(Seq(LabeledPoint(1.0, Vectors.dense(1.0, Double.NaN)), LabeledPoint(0.0, Vectors.dense(Double.PositiveInfinity, 2.0)))).toDF()

val svc = new LinearSVC()
val model = svc.fit(df)

scala> model.intercept
res0: Double = NaN

scala> model.coefficients
res1: org.apache.spark.ml.linalg.Vector = [NaN,NaN] {code}
Bug: Simplify Java UT code with Junit `assertThrows`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Mar/22 02:48
Affects Version/s: 3.4.0

There are some code patterns in Java UTs:
{code:java}
 @Test
  public void testAuthReplay() throws Exception {
    try {
      doSomeOperation();
      fail("Should have failed");
    } catch (Exception e) {
      assertTrue(checkException(e));
    }
  }
{code}
or 

 
{code:java}
@Test(expected = SomeException.class)
  public void testAuthReplay() throws Exception {
    try {
      doSomeOperation();
      fail("Should have failed");
    } catch (Exception e) {
      assertTrue(checkException(e));
      throw e;
    }
  } {code}
we can use Junit assertThrows to simplify the similar patterns
Bug: Incorrect DynamicPartitionPruning caused by Literal
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Mar/22 13:28
Affects Version/s: 3.2.0

The return value of Literal.references is an empty AttributeSet, so Literal is mistaken for a partition column.

 

org.apache.spark.sql.execution.dynamicpruning.PartitionPruning#getFilterableTableScan:
{code:java}
val srcInfo: Option[(Expression, LogicalPlan)] = findExpressionAndTrackLineageDown(a, plan)
srcInfo.flatMap {
  case (resExp, l: LogicalRelation) =>
    l.relation match {
      case fs: HadoopFsRelation =>
        val partitionColumns = AttributeSet(
          l.resolve(fs.partitionSchema, fs.sparkSession.sessionState.analyzer.resolver))
        // When resExp is a Literal, Literal is considered a partition column.         
        if (resExp.references.subsetOf(partitionColumns)) {
          return Some(l)
        } else {
          None
        }
      case _ => None
    } {code}
Bug: external top-level directory is problematic for bazel
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Mar/22 13:10
Affects Version/s: 3.2.1

{{external}} is a hardwired special name for top-level directories for [bazel|https://bazel.build/]. This causes all sorts of issues with both native/basic bazel or extensions like [bazel-compile-commands-extractor|https://github.com/hedronvision/bazel-compile-commands-extractor]. Spark forks using bazel to build Spark have to go through hoops to make things work if at all.
Bug: spark-sql can not enable isolatedClientLoader to extend dsv2 catalog when using builtin hiveMetastoreJar
Status: Open | Priority: Minor | Resolution: 
Created: 24/Mar/22 08:20
Affects Version/s: 3.1.2

Hi, all:

I make use of IsolatedClientLoader to enable datasource v2 catalog on hive, It works well on api/spark-shell, while failed on spark-sql cmd.

After dig into source, I found that the SparkSQLCLIDriver(spark-sql) initialize differently by using CliSessionState which will be reused through cli lifecycle.

Thus the IsolatedClientLoader creator in HiveUtils will determine to off isolate because encoutering special global SessionState by that type.In my case, namespaces/tables will not recognized from another hive catalog since a CliSessionState in sparkSession will always be used to connected with.

I notice [SPARK-21428|https://issues.apache.org/jira/browse/SPARK-21428] but think that since the datasource v2 api should be more popular, SparkSQLCLIDriver should also adjust that?

my env:

spark-3.1.2
hadoop-cdh5.13.0
hive-2.3.6
for each v2 catalog set spark.sql.hive.metastore.jars=builtin(we have no auth to deploy jars on target clusters)

Now, for workaround this, we have to deploy jars on hdfs and use 'path' way which cause a significant delay on catalog initialize.

Any help is appreciate, thanks.
Bug: Simplify the code of TreeNode.clone()
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Mar/22 08:04
Affects Version/s: 3.4.0

SPARK-28057 adds {{forceCopy}} to private {{mapChildren}} method in {{TreeNode}} to realize the {{clone()}} method.

After SPARK-34989, the call corresponding to {{forceCopy=false}} is changed to use {{{}withNewChildren{}}}, and {{forceCopy}} always true and the private {{mapChildren}} only used by {{clone()}} method.
Bug: Fix RemoteBlockPushResolverSuite#testWritingPendingBufsIsAbortedImmediatelyDuringComplete
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Mar/22 12:28
Affects Version/s: 3.4.0

`testWritingPendingBufsIsAbortedImmediatelyDuringComplete`  throw an

RuntimeException, but the Exception is not thrown by the expected statement.
Bug: Clean up Junit api usage in scalatest
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Mar/22 02:54
Affects Version/s: 3.4.0

There are some Junit api usage in scalatest, for example

 
{code:java}
  test("SPARK-31422: getMemoryStatus should not fail after BlockManagerMaster stops") {
    val bmm = new BlockManagerMaster(null, null, new SparkConf, true)
    assertTrue(bmm.getMemoryStatus.isEmpty)
  } {code}
`assertTrue` is junit api
Bug: Enrich Avro data source documentation
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Mar/22 22:06
Affects Version/s: 3.2.1

Enrich Avro data source documentation to emphasize the difference between 
*avroSchema* which is an option, and *jsonFormatSchema* which is a parameter for function *from_avro* .

When using {*}from_avro{*}, *avroSchema* option can be set to a compatible and evolved schema, while *jsonFormatSchema* has to be the actual schema. Elsewise, the behavior is undefined.
Bug: Upgrade log4j2 to 2.17.2
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Mar/22 03:10
Affects Version/s: 3.3.0

Log4j 2.17.2 has been released to:
 * Over 50 improvements and fixes to the Log4j 1.x support. Continued testing has shown it is a suitable replacement for Log4j 1.x in most cases.
 * Scripting now requires a system property be specified naming the languages the user wishes to allow. The scripting engine will not load if the property isn't set.
 * By default, the only remote protocol allowed for loading configuration files is HTTPS. Users can specify a system property to allow others or prevent remote loading entirely.
 * Variable resolution has been modified so that only properties defined as properties in the configuration file can be recursive. All other Lookups are now non-recursive. This addresses issues users were having resolving lookups specified in property definitions for use in the RoutingAppender and RollingFileAppender due to restrictions put in place in 2.17.1.
 * Many other fixes and improvements.

 

The change report as follows:

https://logging.apache.org/log4j/2.x/changes-report.html#a2.17.2
Bug: Upgrade netty to 4.1.75
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Mar/22 10:16
Affects Version/s: 3.3.0

The release notes as follows:
 * [https://netty.io/news/2022/03/10/4-1-75-Final.html]

 

There are 2 import change of this version:
 * Reduce the default PooledByteBufAllocator chunk size from 16 MiB to 4 MiB
 * Change default of io.netty.allocator.useCacheForAllThreads to false

 

Both of these changes are to reduce memory-overhead
Bug: Upgarde scala-maven-plugin from 4.4.0 to 4.5.6
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Mar/22 03:32
Affects Version/s: 3.3.0

4.5.6 upgrades zinc dependency to 1.5.8 and cleans up some unnecessary cascading dependencies

 

https://github.com/davidB/scala-maven-plugin/compare/4.4.0...4.5.6
Bug: Remove unnecessary casts between IntegerType and IntDecimal
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Mar/22 01:37
Affects Version/s: 3.2.1

In {{{}NTile{}}}, the number of rows per bucket is computed as {{{}n / buckets{}}}, where {{n}} is the partition size, and {{buckets}} is the argument to {{NTile}} (number of buckets). The code currently casts the arguments to IntDecimal, then casts the result back to IntegerType. This is unnecessary, since it is equivalent to just doing integer division, i.e. {{{}n div buckets{}}}.
Bug: Move usage logging instrumentation util functions from pandas module to pyspark.util module
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Mar/22 17:51
Affects Version/s: 3.2.1

It will be helpful to attach the usage logger to other modules (e.g. sql) besides Pandas but other modules should not depend on Pandas modules to use the instrumentation utils (e.g. _wrap_function, _wrap_property ...). So we need to move usage logging instrumentation util functions from Pandas module to pyspark.util module.
Bug: Dataframe API: withColumn "comment" parameter
Status: Open | Priority: Minor | Resolution: 
Created: 15/Mar/22 10:11
Affects Version/s: 3.2.1

I often find that the right time to document a column in a dataframe is when I create it. It would be nice if withColumn took an optional comment parameter, then one could write e.g.

 
{code:java}
df = df.withColumn("tax", F.col("salary")*F.col("tax_percentage"), comment="The amount of tax payed in dollars.")
{code}
It is possible to do something similiar with alias, but as far as I know the equivalent would be the much more clunky:
{code:java}
df = df.withColumn("tax", F.col("salary")*F.col("tax_percentage")).alias("tax", metadata={"comment": "The amount of tax payed in dollars."}) {code}
Bug: Use more generic Channel type to initialize Netty BootStrap
Status: In Progress | Priority: Minor | Resolution: 
Created: 14/Mar/22 12:07
Affects Version/s: 3.3.0

Spark use {{SocketChannel}} type to initialize {{BootStrap}} now, but there is no need to limit the type to {{{}Socket{}}}, so this pr change the related type definitions from {{SocketChannel}} to {{{}Channel{}}}.
Bug: Introduce a `exists` function for `TreeNode` to eliminate duplicate code pattern
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Mar/22 08:31
Affects Version/s: 3.3.0

There are many duplicate code patterns in Spark code:
{code:java}
treeNode.find(condition).isDefined 
treeNode.find(condition).nonEmpty 
treeNode.find(condition).isEmpty{code}
Maybe we can introduce an `exists` function to simplify the above code patterns.
Bug: Fix always false condition in LogDivertAppender#initLayout 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Mar/22 03:11
Affects Version/s: 3.3.0

 
{code:java}
  private static StringLayout initLayout(OperationLog.LoggingLevel loggingMode) {
   ...
    for (Map.Entry<String, Appender> entry : appenders.entrySet()) {
      Appender ap = entry.getValue();
      if (ap.getClass().equals(ConsoleAppender.class)) {
        Layout l = ap.getLayout();
        if (l.getClass().equals(StringLayout.class)) {
          layout = (StringLayout) l;
          break;
        }
      }
    }
    return getLayout(isVerbose, layout);
  } {code}
 

`l.getClass().equals(StringLayout.class)` is always return false because `StringLayout` is a interface
Bug: Clean up duplicate codes in get method of ColumnarBatchRow/ColumnarRow/MutableColumnarRow
Status: In Progress | Priority: Minor | Resolution: 
Created: 09/Mar/22 04:46
Affects Version/s: 3.3.0

The logic of the `ColumnarBatchRow.get` is very similar to `ColumnarRow.get` and `MutableColumnarRow.get`
Bug: Document config STREAMING_SESSION_WINDOW_MERGE_SESSIONS_IN_LOCAL_PARTITION
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Mar/22 07:51
Affects Version/s: 3.3.0

We have use case the customer faces issue on large shuffle write in session window. The config is hidden, although it is useful for such case. We should document it so end users can find it easily.
Bug: Correct semantic of CheckAnalysis.getDataTypesAreCompatibleFn method
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Mar/22 11:11
Affects Version/s: 3.2.1

Currently, in `CheckAnalysis` method  [getDataTypesAreCompatibleFn |https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala#L606] implemented as:

{code:java}
  private def getDataTypesAreCompatibleFn(plan: LogicalPlan): (DataType, DataType) => Boolean = {
    val isUnion = plan.isInstanceOf[Union]
    if (isUnion) {
      (dt1: DataType, dt2: DataType) =>
        !DataType.equalsStructurally(dt1, dt2, true)
    } else {
      // SPARK-18058: we shall not care about the nullability of columns
      (dt1: DataType, dt2: DataType) =>
        TypeCoercion.findWiderTypeForTwo(dt1.asNullable, dt2.asNullable).isEmpty
    }
  }
{code}

Return false when data types are compatible, otherwise return true, which is pretty confusing.
Bug: build of spark sql against hadoop-3.4.0-snapshot failing with bouncycastle classpath error
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Mar/22 16:23
Affects Version/s: 3.3.0

builidng spark master with {{-Dhadoop.version=3.4.0-SNAPSHOT}} and a local hadoop build breaks in the sbt compiler plugin


{code}
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.3.0:testCompile (scala-test-compile-first) on project spark-sql_2.12: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.3.0:testCompile failed: A required class was missing while executing net.alchim31.maven:scala-maven-plugin:4.3.0:testCompile: org/bouncycastle/jce/provider/BouncyCastleProvider
[ERROR] -----------------------------------------------------
[ERROR] realm =    plugin>net.alchim31.maven:scala-maven-plugin:4.3.0

{code}

* this is the classpath of the sbt compiler
* hadoop hasn't been doing anything related to bouncy castle.

setting scala-maven-plugin to 3.4.0 makes this go away, i.e. reapplying SPARK-36547

the implication here is that the plugin version is going to have to be configured in different profiles.
Bug: Add `spark-` prefix to namespaces and `-driver` suffix to drivers during IT
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Mar/22 13:11
Affects Version/s: 3.3.0

Currently when there is no configured K8S namespace by the user the Kubernetes IT tests use UUID.toString() without the '-'es:
{code:java}
val namespace = namespaceOption.getOrElse(UUID.randomUUID().toString.replaceAll("-", "")) {code}
 

Proposal: prefix the temporary namespace with "spark-" or "spark-test-".

 

The name of the driver pod is:
{code:java}
driverPodName = "spark-test-app-" + UUID.randomUUID().toString.replaceAll("-", "") {code}
i.e. it does not mention that it is the driver.

For non-test it uses names like `spark-on-k8s-app-f6bfc57f4a938abe-driver`, i.e. the value of "–name" config + hash + "-driver".

In both IT tests and non-test the executor pods always mention "-exec-", so they are clear.

 

Proposal: unify non-test and IT test name and use "spark-test-app-"+ hash + "-driver" for the IT tests.
Bug: Combine compatible scalar subqueries
Status: In Progress | Priority: Minor | Resolution: 
Created: 02/Mar/22 06:15
Affects Version/s: 3.3.0

The idea of this issue is originated from [https://github.com/NVIDIA/spark-rapids/issues/4186]

Currently, Spark SQL executes each incorrelated scalar subquery as an independent spark job. It generates a lot of spark jobs when we run a query with a lot of incorrelated scalar subqueries. Scenarios like this can be optimized in terms of logcial plan. We can combine subquery plans of compatible scalar subqueries into fused subquery plans. And let them shared by multiple scalar subqueries. With combining compatible scalar subqueries, we can cut off the cost of subquery jobs, because common parts of compatible subquery plans (scans/filters) will be reused.

 

Here is an example to demonstrate the basic idea of combining compatible scalar subqueries:
{code:java}
SELECT SUM(i)
FROM t
WHERE l > (SELECT MIN(l2) FROM t)
AND l2 < (SELECT MAX(l) FROM t)
AND i2 <> (SELECT MAX(i2) FROM t)
AND i2 <> (SELECT MIN(i2) FROM t) {code}
Optimized logicial plan of above query looks like:
{code:java}
Aggregate [sum(i)]
+- Project [i]
  +- Filter (((l > scalar-subquery#1) AND (l2 < scalar-subquery#2)) AND (NOT (i2 = scalar-subquery#3) AND NOT (i2 = scalar-subquery#4)))
     :  :- Aggregate [min(l2)]
     :  :  +- Project [l2]
     :  :     +- Relation [l,l2,i,i2]
     :  +- Aggregate [max(l)]
     :     +- Project [l]
     :        +- Relation [l,l2,i,i2]
     :  +- Aggregate [max(i2)]
     :     +- Project [l]
     :        +- Relation [l,l2,i,i2]
     :  +- Aggregate [min(i2)]
     :     +- Project [l]
     :        +- Relation [l,l2,i,i2]
     +- Relation [l,l2,i,i2] {code}
After the combination of compatible scalar subqueries, the logicial plan becomes:
{code:java}
 Aggregate [sum(i)]
 +- Project [i]
   +- Filter (((l > shared-scalar-subquery#1) AND (l2 < shared-scalar-subquery#2)) AND (NOT (i2 = shared-scalar-subquery#3) AND NOT (i2 = shared-scalar-subquery#4)))
      :  :- Aggregate [min(l2),max(l),max(i2),min(i2)]
      :  :  +- Project [l2,l,i2]
      :  :     +- Relation [l,l2,i,i2]
      :  :- Aggregate [min(l2),max(l),max(i2),min(i2)]
      :  :  +- Project [l2,l,i2]
      :        +- Relation [l,l2,i,i2]
      :  :- Aggregate [min(l2),max(l),max(i2),min(i2)]
      :  :  +- Project [l2,l,i2]
      :        +- Relation [l,l2,i,i2]
      :  :- Aggregate [min(l2),max(l),max(i2),min(i2)]
      :  :  +- Project [l2,l,i2]
      :        +- Relation [l,l2,i,i2]
      +- Relation [l,l2,i,i2] {code}
 

There are 4 scalar subqueries within this query. Although they are semantically unequal, they are based on the same relation. Therefore, we can merge all of them into an unified Aggregate to resue the common scan(relation).
Bug: Adding a demo/walkthrough section Running Spark on Kubernetes
Status: Open | Priority: Minor | Resolution: 
Created: 01/Mar/22 21:52
Affects Version/s: 3.2.1

I propose adding a section to [Running Spark on Kubernetes - Spark 3.2.1 Documentation (apache.org)|https://spark.apache.org/docs/latest/running-on-kubernetes.html#configuration] that walks a user through the 'happy path' of:
 # creating and configuring a cluster
 # preparing an example spark job
 # adding the JAR to the container image
 # submitting the job to the cluster using spark-submit
 # getting the results

The current guide covers a lot of this in the abstract, but I have to a lot of searching if I'm trying to walk through setting this up on Kubernetes the first time. I feel this would significantly improve the guide.

The first section can be extensible for local demo cluster (minikube, kind) as well as cloud providers (amazon, google, azure).
Bug: Two links beneath Spark SQL Guide/Data Sources do not work properly
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 23/Mar/22 01:40
Affects Version/s: 3.2.1

I ran into a minor documentation issue, beneath [Spark SQL Guide/Data Sources|https://spark.apache.org/docs/latest/sql-data-sources.html], when click 'Ignore Corrupt File' or 'Ignore Missing Files', it does redirect me to the right page, but does not scroll to the right section. This issue actually has been there since v3.0.0.

I've found the reason, there's a typo in those 2 links. I can make a quick fix for this issue, could someone please assign this to me?
Bug: Change day to month 
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 04/Mar/22 19:35
Affects Version/s: 3.3.0

In spark\python\pyspark\pandas\tests\indexes\test_datetime.py
line 117

{code:java}
   def test_day_name(self):
        for psidx, pidx in self.idx_pairs:
            self.assert_eq(psidx.day_name(), pidx.day_name())

    def test_month_name(self):
        for psidx, pidx in self.idx_pairs:
            self.assert_eq(psidx.day_name(), pidx.day_name())
{code}

Both of this functions are doing the same ting.
Bug: Fix `test_ceil` to test `ceil`
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 07/Mar/22 14:34
Affects Version/s: 3.3.0

In python/pyspark/pandas/tests/test_series_datetime.py line 262 



{code:java}
    def test_floor(self):
        self.check_func(lambda x: x.dt.floor(freq="min"))
        self.check_func(lambda x: x.dt.floor(freq="H"))

    def test_ceil(self):
        self.check_func(lambda x: x.dt.floor(freq="min"))
        self.check_func(lambda x: x.dt.floor(freq="H"))
{code}

Change  x.dt.floor to  x.dt.ceil
Bug: Add Braces with if,else,for,do and while statements
Status: Resolved | Priority: Trivial | Resolution: Won't Fix
Created: 08/Mar/22 01:39
Affects Version/s: 3.2.0

Braces are used with {_}if{_}, {_}else{_}, {_}for{_}, _do_ and _while_ statements, even if the body contains only a single statement. Avoid using the following example: 
{code:java}
if (condition) statements;
{code}