av9ash/GitBugs huggingface dataset:
Bug: Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Jun/20 20:47
Affects Version/s: 3.1.0

Two new options, _modifiiedBefore_ and _modifiedAfter_, is provided expecting a value in 'YYYY-MM-DDTHH:mm:ss' format. _PartioningAwareFileIndex_ considers these options during the process of checking for files, just before considering applied _PathFilters_ such as {{pathGlobFilter.}} In order to filter file results, a new PathFilter class was derived for this purpose. General house-keeping around classes extending PathFilter was performed for neatness. It became apparent support was needed to handle multiple potential path filters. Logic was introduced for this purpose and the associated tests written.

 

When loading files from a data source, there can often times be thousands of file within a respective file path. In many cases I've seen, we want to start loading from a folder path and ideally be able to begin loading files having modification dates past a certain point. This would mean out of thousands of potential files, only the ones with modification dates greater than the specified timestamp would be considered. This saves a ton of time automatically and reduces significant complexity managing this in code.

 

*Example Usages*
_Load all CSV files modified after date:_
{{spark.read.format("csv").option("modifiedAfter","2020-06-15T05:00:00").load()}}

_Load all CSV files modified before date:_
{{spark.read.format("csv").option("modifiedBefore","2020-06-15T05:00:00").load()}}

_Load all CSV files modified between two dates:_
{{spark.read .format("csv") .option("modifiedAfter","2019-01-15T05:00:00") .option("modifiedBefore","2020-06-15T05:00:00") .load()}}
Bug: The name in the with clause when it is same as table name. And when that table name is used in the other places, it is not taking the table, it is considering the with clause.
Status: Resolved | Priority: Minor | Resolution: Incomplete
Created: 26/Jun/20 06:42
Affects Version/s: 2.2.1

The name in the with clause when it is same as table name. And when that table name is used in the other places, it is not taking the table, it is considering the with clause sub table. The example is given below with explanation:

database1.sample structure:

columns: id,name

 

with sample as (

select id, 1 as cnt from database1.sample),

with amp as(

select name,id from database1sample)

select * from sample inner join amp on amp.id=sample.id

;

 

In this example, second alias will fail for missing "name" column, even though the database1.sample table has "name" column. Becuase it is referring column "name" from first alias sample. Because the tablename and alias name are same here and this error came.

The bug is it should not take the alias even i have given databasename along with tablename. It should consider from metadata of the table not from alias.

Here the solution is tablename should not be alias name. The first alias should be sample1 or some thing like that.
Bug: Thriftserver interval returns "4 weeks 2 days" in 2.4 and "30 days" in 3.0
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 29/Jun/20 19:18
Affects Version/s: 3.0.0

In https://github.com/apache/spark/pull/26418, a setting spark.sql.dialect.intervalOutputStyle was implemented, to control interval output style. This PR also removed "toString" from CalendarInterval. This change got reverted in https://github.com/apache/spark/pull/27304, and the CalendarInterval.toString got implemented back in https://github.com/apache/spark/pull/26572.

But it behaves differently now: In 2.4 "4 weeks 2 days" are returned, and 3.0 returns "30 days".
Thriftserver uses HiveResults.toHiveString, which uses CalendarInterval.toString to return interval results as string.  The results are now different in 3.0
Bug: Avoid Pandas import for CategoricalDtype with Arrow conversion
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Jun/20 00:03
Affects Version/s: 3.1.0

The import for CategoricalDtype changed in Pandas from 0.23 to 1.0 and currently pyspark checks 2 places to import. It would be better check the type as a string and avoid any imports.
Bug: pyspark save of serialized model is failing for windows.
Status: Reopened | Priority: Minor | Resolution: 
Created: 22/Jun/20 12:48
Affects Version/s: 2.3.0

{color:#172b4d}Hi, {color}

{color:#172b4d}We are using spark functionality to save the serialized model to disk . On windows platform we are seeing save of the serialized model is failing with the error:  o288.save() failed. {color}

 

 

 

!image-2020-06-22-18-19-32-236.png!
Bug: Refactor the logic to compute SPARK_HOME into a common place.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Jun/20 02:41
Affects Version/s: 3.0.1

Currently several unit tests inline the logic to compute spark home. Secondly the error that is thrown when spark home is not set prints the entire environment making it difficult to find the actual root cause which is "spark home is not set".

1. Refactor the code to compute the spark home in a common place (SQLHelper.scala)
2. Change the error message to make it more readable.
Bug: Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Jun/20 03:58
Affects Version/s: 3.0.0

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123787/testReport/pyspark.mllib.tests.test_streaming_algorithms/StreamingLogisticRegressionWithSGDTests/test_training_and_prediction/

{code}
======================================================================
FAIL: test_training_and_prediction (pyspark.mllib.tests.test_streaming_algorithms.StreamingLogisticRegressionWithSGDTests)
Test that the model improves on toy data with no. of batches
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/mllib/tests/test_streaming_algorithms.py", line 367, in test_training_and_prediction
    self._eventually(condition)
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/mllib/tests/test_streaming_algorithms.py", line 78, in _eventually
    % (timeout, lastValue))
AssertionError: Test failed due to timeout after 30 sec, with last condition returning: Latest errors: 0.67, 0.71, 0.78, 0.7, 0.75, 0.74, 0.73, 0.69, 0.62, 0.71, 0.69, 0.75, 0.72, 0.77, 0.71, 0.74

----------------------------------------------------------------------
Ran 13 tests in 185.051s

FAILED (failures=1, skipped=1)
{code}
Bug: Returns the projected plan directly from NestedColumnAliasing
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Jun/20 08:07
Affects Version/s: 3.0.0

{{NestedColumnAliasing}} and {{GeneratorNestedColumnAliasing}} have a different usage pattern:

{code}
    case p @ NestedColumnAliasing(nestedFieldToAlias, attrToAliases) =>
      NestedColumnAliasing.replaceToAliases(p, nestedFieldToAlias, attrToAliases)
{code}

vs

{code}
    case GeneratorNestedColumnAliasing(p) => p
{code}

It should be better to match it
Bug: Run the tests in registerJavaFunction and registerJavaUDAF only when test classes are complied
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Jun/20 03:08
Affects Version/s: 2.4.6

If you do a plain package with sbt:

{code}
./build/sbt -DskipTests -Phive-thriftserver clean package
./run-tests --python-executable=python3 --testname="pyspark.sql.udf UserDefinedFunction"
{code}

The doctests in registerJavaFunction and registerJavaUDAF fail because it requires some classes from the test compilation.

We should skip them conditionally
Bug: Repartition by key should support partition coalesce for AQE
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Jun/20 19:37
Affects Version/s: 3.0.0

when adaptive query execution is enabled the following expression should support coalescing of partitions:
{code:java}
dataframe.repartition(col("somecolumn")) {code}
currently it does not because it simply calls the repartition implementation where number of partitions is specified:
{code:java}
  def repartition(partitionExprs: Column*): Dataset[T] = {
    repartition(sparkSession.sessionState.conf.numShufflePartitions, partitionExprs: _*)
  }{code}
and repartition with the number of partitions specified does now allow for coalescing of partitions (since this breaks the user's expectation that it will have the number of partitions specified).

for more context see the discussion here:

[https://github.com/apache/spark/pull/27986]

a simple test to confirm that repartition by key does not support coalescing of partitions can be added in AdaptiveQueryExecSuite like this (it currently fails):
{code:java}
  test("SPARK-32056 repartition has less partitions for small data when adaptiveExecutionEnabled") {
    Seq(true, false).foreach { enableAQE =>
      withSQLConf(
        SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> enableAQE.toString,
        SQLConf.SHUFFLE_PARTITIONS.key -> "50",
        SQLConf.COALESCE_PARTITIONS_INITIAL_PARTITION_NUM.key -> "50",
        SQLConf.SHUFFLE_PARTITIONS.key -> "50") {
        val partitionsNum = (1 to 10).toDF.repartition($"value")
          .rdd.collectPartitions().length
        if (enableAQE) {
          assert(partitionsNum < 50)
        } else {
          assert(partitionsNum === 50)
        }
      }
    }
  }
{code}
Bug: Remove references to "blacklist"/"whitelist" language (outside of blacklisting feature)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Jun/20 20:27
Affects Version/s: 3.0.1

As per [discussion on the Spark dev list|https://lists.apache.org/thread.html/rf6b2cdcba4d3875350517a2339619e5d54e12e66626a88553f9fe275%40%3Cdev.spark.apache.org%3E], it will be beneficial to remove references to problematic language that can alienate potential community members. One such reference is "blacklist" and "whitelist". While it seems to me that there is some valid debate as to whether these terms have racist origins, the cultural connotations are inescapable in today's world.

Renaming the entire blacklisting feature would be a large effort with lots of care needed to maintain public-facing APIs and configurations. Though I think this will be a very rewarding effort for which I've filed SPARK-32037, I'd like to start by tackling all of the other references to such terminology in the codebase, of which there are still dozens or hundreds beyond the blacklisting feature.


I'm not sure what the best "Component" is for this so I put Spark Core for now.
Bug: Rename blacklisting feature to avoid language with racist connotation
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Jun/20 20:31
Affects Version/s: 3.0.1

As per [discussion on the Spark dev list|https://lists.apache.org/thread.html/rf6b2cdcba4d3875350517a2339619e5d54e12e66626a88553f9fe275%40%3Cdev.spark.apache.org%3E], it will be beneficial to remove references to problematic language that can alienate potential community members. One such reference is "blacklist". While it seems to me that there is some valid debate as to whether this term has racist origins, the cultural connotations are inescapable in today's world.

I've created a separate task, SPARK-32036, to remove references outside of this feature. Given the large surface area of this feature and the public-facing UI / configs / etc., more care will need to be taken here.

I'd like to start by opening up debate on what the best replacement name would be. Reject-/deny-/ignore-/block-list are common replacements for "blacklist", but I'm not sure that any of them work well for this situation.
Bug: Unnecessary tasks are launched when input is empty with AQE
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Jun/20 07:37
Affects Version/s: 3.0.0

[https://github.com/apache/spark/pull/28226] meant to avoid launching unnecessary tasks for 0-size partitions when AQE is enabled. However, when all partitions are empty, the number of partitions will be `spark.sql.adaptive.coalescePartitions.initialPartitionNum` and (a lot of) unnecessary tasks are launched in this case.
Bug: Huber loss Convergence
Status: Resolved | Priority: Minor | Resolution: Resolved
Created: 23/Jun/20 01:29
Affects Version/s: 3.1.0

|performace test in https://issues.apache.org/jira/browse/SPARK-31783,
 Huber loss seems start to diverge since 70 iters.

  {code:scala}
 for (size <- Seq(1, 4, 16, 64); iter <- Seq(10, 50, 100)) {
    Thread.sleep(10000)
    val hlir = new LinearRegression().setLoss("huber").setSolver("l-bfgs").setMaxIter(iter).setTol(0)
    val start = System.currentTimeMillis
    val model = hlir.setBlockSize(size).fit(df)
    val end = System.currentTimeMillis
    println((model.uid, size, iter, end - start, model.summary.objectiveHistory.last, model.summary.totalIterations, model.coefficients.toString.take(100)))
}{code}
Bug: The metric of MemoryBytesSpill is incorrect when doing Aggregate
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Jun/20 08:34
Affects Version/s: 3.0.0

When doing Aggregate and spill occurs, the Spill(memory) metric is zero while Spill(disk) metric is right. As shown below

!image-2020-06-10-16-35-58-002.png!
Bug: expose mapSideCombine in aggByKey/reduceByKey/foldByKey
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 10/Jun/20 03:53
Affects Version/s: 3.1.0

1. {{aggregateByKey}}, {{reduceByKey}} and  {{foldByKey}} will always perform {{mapSideCombine}};

However, this can be skiped sometime, specially in ML (RobustScaler):
{code:java}
vectors.mapPartitions { iter =>
  if (iter.hasNext) {
    val summaries = Array.fill(numFeatures)(
      new QuantileSummaries(QuantileSummaries.defaultCompressThreshold, relativeError))
    while (iter.hasNext) {
      val vec = iter.next
      vec.foreach { (i, v) => if (!v.isNaN) summaries(i) = summaries(i).insert(v) }
    }
    Iterator.tabulate(numFeatures)(i => (i, summaries(i).compress))
  } else Iterator.empty
}.reduceByKey { case (s1, s2) => s1.merge(s2) } {code}
 

This {{reduceByKey}} in {{RobustScaler}} does not need {{mapSideCombine}} at all, similar places exist in {{KMeans}}, {{GMM}}, etc;
To my knowledge, we do not need {{mapSideCombine}} if the reduction factor isn't high;

 

2. {{treeAggregate}} and {{treeReduce}} are based on {{foldByKey}},  the {{mapSideCombine}} in the first call of {{foldByKey}} can also be avoided.

 

SPARK-772:

{quote}
Map side combine in group by key case does not reduce the amount of data shuffled. Instead, it forces a lot more objects to go into old gen, and leads to worse GC.
{quote}

 

So what about:

1. exposing mapSideCombine in {{aggByKey}}/{{reduceByKey}}/{{foldByKey}}, so that user can disable unnecessary mapSideCombine

2. disabling the {{mapSideCombine}} in the first call of {{foldByKey}} in  {{treeAggregate}} and {{treeReduce}}

3. disabling the unnecessary {{mapSideCombine}} in ML;

Friendly ping [~srowen] [~huaxingao] [~weichenxu123] [~hyukjin.kwon] [~viirya]
Bug: StackOverflowError in FileScanRDD
Status: Resolved | Priority: Minor | Resolution: Cannot Reproduce
Created: 05/Jun/20 10:40
Affects Version/s: 2.4.5

Reading from FileScanRDD may failed with a StackOverflowError in my environment:
- There are a mass of empty files in table partition。
- Set `spark.sql.files.maxPartitionBytes`  with a large value: 1024MB

A quick workaround is set `spark.sql.files.maxPartitionBytes` with a small value, like default 128MB.

A better way is resolve the recursive calls in FileScanRDD.

{code}
java.lang.StackOverflowError
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.getSubject(Subject.java:297)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:648)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2828)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2818)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2684)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
	at org.apache.parquet.hadoop.util.HadoopInputFile.fromPath(HadoopInputFile.java:38)
	at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:640)
	at org.apache.spark.sql.execution.datasources.parquet.SpecificParquetRecordReaderBase.initialize(SpecificParquetRecordReaderBase.java:148)
	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:143)
	at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.$anonfun$buildReaderWithPartitionValues$2(ParquetFileFormat.scala:326)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:116)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:169)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:173)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:173)
{code}
Bug: current_timestamp called in a cache dataframe freezes the time for all future calls
Status: Open | Priority: Minor | Resolution: 
Created: 22/Jun/20 07:03
Affects Version/s: 2.3.0

If I call current_timestamp 3 times while caching the dataframe variable in order to freeze that dataframe's time, the 3rd dataframe time and beyond (4th, 5th, ...) will be frozen to the 2nd dataframe's time. The 1st dataframe and the 2nd will differ in time but will become static on the 3rd usage and beyond (when running on Zeppelin or Jupyter).

Additionally, caching only caused 2 dataframes to cache skipping the 3rd. However,
{code:java}
val df = Seq(java.time.LocalDateTime.now.toString).toDF("datetime").cache
df.count

// this can be run 3 times no issue.
// then later cast to TimestampType{code}
doesn't have this problem and all 3 dataframes cache with correct times displaying.

Running the code in shell and Jupyter or Zeppelin (ZP) also produces different results. In the shell, you only get 1 unique time no matter how many times you run it, current_timestamp. However, in ZP or Jupyter I have always received 2 unique times before it froze.

 
{code:java}
val df1 = spark.range(1).select(current_timestamp as "datetime").cache
df1.count

df1.show(false)

Thread.sleep(9500)

val df2 = spark.range(1).select(current_timestamp as "datetime").cache
df2.count 

df2.show(false)

Thread.sleep(9500)

val df3 = spark.range(1).select(current_timestamp as "datetime").cache 
df3.count 

df3.show(false){code}
Bug: Client memory passed unvalidated to the JVM Xmx
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 03/Jun/20 19:13
Affects Version/s: 2.0.0

When launching in the client mode Spark launcher uses spark.driver.memory config (among other settings in the precedence order). Unlike the cluster mode, the client memory config is not trimmed and not validated to be a valid suffix for the [Xmx option|https://docs.oracle.com/en/java/javase/14/docs/specs/man/java.html#extra-options-for-java].
Bug: Show Spark Driver name on Spark history web page
Status: In Progress | Priority: Minor | Resolution: 
Created: 30/Jun/20 02:25
Affects Version/s: 2.4.5

Our service dynamically creates short-lived YARN clusters in cloud. Spark applications run on these dynamically created clusters. Log data for these applications is stored on a remote file-system. We configure a static instance of SparkHistoryServer to view information on jobs that ran on these clusters.

Since we are using a single History Server for multiple clusters, it will be useful for users to have information on where the jobs were executed. We would like to display this information on the main web-page, instead of having users go through multiple links to retrieve this information.
Bug: Allow reading history log files from multiple directories
Status: In Progress | Priority: Minor | Resolution: 
Created: 25/Jun/20 02:36
Affects Version/s: 2.4.5

Our service dynamically creates short-lived YARN clusters in cloud. Spark applications run on these dynamically created clusters. Log data for these applications is stored on a remote file-system. We want a static instance of SparkHistoryServer to view information on jobs that ran on these clusters. We use glob because we cannot have a static list of directories where the log files reside.
Bug: Inconsistent AWS environment variables in documentation
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Jun/20 10:13
Affects Version/s: 2.4.6

Looking at the actual Scala code, the environment variables used to log into AWS are:
 - AWS_ACCESS_KEY_ID
 - AWS_SECRET_ACCESS_KEY
 - AWS_SESSION_TOKEN

These are the same that AWS uses in their libraries.

However, looking through the Spark documentation and comments, I see that these are not denoted correctly across the board:

docs/cloud-integration.md
 106:1. `spark-submit` reads the `AWS_ACCESS_KEY`, `AWS_SECRET_KEY` *<-- both different*
 107:and `AWS_SESSION_TOKEN` environment variables and sets the associated authentication options

docs/streaming-kinesis-integration.md
 232:- Set up the environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_KEY` with your AWS credentials. *<-- secret key different*

external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
 34: $ export AWS_ACCESS_KEY_ID=<your-access-key>
 35: $ export AWS_SECRET_KEY=<your-secret-key> *<-- different*
 48: Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY *<-- secret key different*

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
 438: val keyId = System.getenv("AWS_ACCESS_KEY_ID")
 439: val accessKey = System.getenv("AWS_SECRET_ACCESS_KEY")
 448: val sessionToken = System.getenv("AWS_SESSION_TOKEN")

external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
 53: * $ export AWS_ACCESS_KEY_ID=<your-access-key>
 54: * $ export AWS_SECRET_KEY=<your-secret-key> *<-- different*
 65: * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY *<-- secret key different*

external/kinesis-asl/src/main/java/org/apache/spark/examples/streaming/JavaKinesisWordCountASL.java
 59: * $ export AWS_ACCESS_KEY_ID=[your-access-key]
 60: * $ export AWS_SECRET_KEY=<your-secret-key> *<-- different*
 71: * Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY *<-- secret key different*
Bug: Add PrometheusServletSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Jun/20 00:41
Affects Version/s: 3.1.0

This Jira aims to be added _PrometheusServletSuite_.

*Note:* This jira was created to propose _Prometheus_ driver metric format change to be consistent with executor metric format. However, currently, _PrometheusServlet_ follows _Spark 3.0 JMX Sink + Prometheus JMX Exporter_ format so this Jira coverage has been converted to just _PrometheusServletSuite_ in the light of discussion on https://github.com/apache/spark/pull/28865.