av9ash/GitBugs huggingface dataset:
Bug: WARN WindowExec: No Partition Defined is being printed 4 times. 
Status: Open | Priority: Major | Resolution: 
Created: 31/Oct/21 21:53
Affects Version/s: 3.3.0

Hi I use this code  
{code:java}
f01 = spark.read.json("/home/test_files/falk/flatted110721/F01.json/*.json")
pf01 = f01.to_pandas_on_spark()
pf01 = pf01.rename(columns=lambda x: re.sub(':P$', '', x))
pf01["OBJECT_CONTRACT:DATE_PUBLICATION_NOTICE"] = ps.to_datetime(pf01["OBJECT_CONTRACT:DATE_PUBLICATION_NOTICE"])
pf01.info(){code}
 

 sometimes it prints 
  
{code:java}
 21/10/31 20:38:04 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
 21/10/31 20:38:04 WARN package: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.
 21/10/31 20:38:08 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
 /opt/spark/python/pyspark/sql/pandas/conversion.py:214: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead.  To get a de-fragmented frame, use `newframe = frame.copy()`
   df[column_name] = series
 /opt/spark/python/pyspark/pandas/utils.py:967: UserWarning: `to_pandas` loads all data into the driver's memory. It should only be used if the resulting pandas Series is expected to be small.
   warnings.warn(message, UserWarning)
 21/10/31 20:38:16 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
 21/10/31 20:38:18 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.{code}
 
 and some other times it "just" prints 
  
{code:java}
 21/10/31 21:24:13 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
 21/10/31 21:24:16 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
 21/10/31 21:24:22 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
 21/10/31 21:24:24 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.{code}
Why does it print df[column_name] = series ?
  
 can we remove /opt/spark/python/pyspark/pandas/utils.py:967: ?
 and warnings.warn(message, UserWarning) ?
 and 3 of WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.?
Bug: Pin PySpark version installed in the Binder environment for tagged commit
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Oct/21 15:26
Affects Version/s: 3.2.0

I noticed that the PySpark 3.1.2 is installed in the live notebook environment even though the notebook is for PySpark 3.2.0.
http://spark.apache.org/docs/3.2.0/api/python/getting_started/index.html

I guess someone accessed to Binder and built the container image with v3.2.0 before we published the pyspark package to PyPi.
https://mybinder.org/

I think it's difficult to rebuild the image manually.
To avoid such accident, I'll propose to pin the version of PySpark in binder/postBuild
Bug: documentation - unclear "Using PySpark Native Features"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/21 18:12
Affects Version/s: 3.2.0

sorry, no idea on the version it affects or what Shepard is? no explanation on this form so guessed whatever!

 

This page of your documentation is UNCLEAR

paragraph "Using PySpark Native Features" QUOTE

"PySpark allows to upload Python files ({{.py}}), zipped Python packages ({{.zip}}), and Egg files ({{.egg}}) to the executors by:
 * Setting the configuration setting {{spark.submit.pyFiles}}

 * Setting {{--py-files}} option in Spark scripts

 * Directly calling [{{pyspark.SparkContext.addPyFile()}}|https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.SparkContext.addPyFile.html#pyspark.SparkContext.addPyFile] in applications

 

QUESTION: is this all of the above or each of the above steps?

suggest adding "OR" between each bullet point?

 

[https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html]
Bug: Application has been removed by master but driver still running
Status: Open | Priority: Major | Resolution: 
Created: 27/Oct/21 07:23
Affects Version/s: 3.0.1

{code:java}
21/08/30 10:27:31 INFO Master: Removing executor app-20210827190502-0030/1 because it is EXITED
21/08/30 10:27:31 INFO Master: Launching executor app-20210827190502-0030/4 on worker worker-20210826183405-10.39.0.69-37147
21/08/30 10:27:31 INFO Master: 10.39.0.68:47160 got disassociated, removing it.
21/08/30 10:27:31 INFO Master: 10.39.0.68:35160 got disassociated, removing it.
21/08/30 10:27:31 INFO Master: Removing app app-20210827190502-0030
21/08/30 10:27:31 WARN Master: Got status update for unknown executor app-20210827190502-0030/4
21/08/30 10:27:31 WARN Master: Got status update for unknown executor app-20210827190502-0030/4
21/08/30 10:27:46 WARN Master: Got status update for unknown executor app-20210827190502-0030/2
21/08/30 10:27:48 WARN Master: Got status update for unknown executor app-20210827190502-0030/0
21/08/30 10:27:50 WARN Master: Got status update for unknown executor app-20210827190502-0030/3{code}
As the logs show, Spark master removed my application. But my driver process is still running. I would like to know what could be the cause of this and how I can avoid it.

 

My Spark version is 3.0.1 and I run Spark standalone on k8s.I don't know how to reproduce it, so it doesn't always come up. Sometimes it appears when an oom exception occurs in the executor
Bug: Support use IN/EXISTS with subquery in Project/Aggregate
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Oct/21 13:28
Affects Version/s: 3.2.0

{code:java}
CREATE TABLE tbl1 (col1 INT, col2 STRING) USING PARQUET;
INSERT OVERWRITE TABLE tbl1 SELECT 0,1;
CREATE TABLE tbl2 (c1 INT, c2 STRING) USING PARQUET; 
INSERT OVERWRITE TABLE tbl2 SELECT 0,2;

case 1:
    select c1 in (select col1 from tbl1) from tbl2 
    Error msg:
        IN/EXISTS predicate sub-queries can only be used in Filter/Join and a few commands: Project []
case 2:
    select count(1), case when c1 in (select col1 from tbl1) then "A" else "B" end as tag from tbl2 group by case when c1 in (select col1 from tbl1) then "A" else "B" end 
    Error msg:
        IN/EXISTS predicate sub-queries can only be used in Filter/Join and a few commands: Aggregate []
{code}
Bug: Add util function to raise advice warning for pandas API on Spark.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/21 04:35
Affects Version/s: 3.3.0

Pandas API on Spark has some features that potentially cause the performance degradation or an unexpected behavior e.g. `sort_index`, `index_col`, `to_pandas`, etc.

 

We should raise the proper advice warning for those functions so that users can adjust their pandas-on-Spark code base more robust.
Bug: TestUtils.isPythonVersionAtLeast38 returns incorrect results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Oct/21 17:55
Affects Version/s: 3.2.0

I was working on {{HiveExternalCatalogVersionsSuite}} recently and noticed that it was never running against the Spark 2.x release lines, only the 3.x ones. The problem was coming from here, specifically the Python 3.8+ version check:
{code}
    versions
      .filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38())
      .filter(v => v.startsWith("3") || !SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9))
{code}

I found that {{TestUtils.isPythonVersionAtLeast38()}} was always returning true, even when my system installation of Python3 was 3.7. Thinking it was an environment issue, I pulled up a debugger to check which version of Python the test JVM was seeing, and it was in fact Python 3.7.

Turns out the issue is with the {{isPythonVersionAtLeast38}} method:
{code}
  def isPythonVersionAtLeast38(): Boolean = {
    val attempt = if (Utils.isWindows) {
      Try(Process(Seq("cmd.exe", "/C", "python3 --version"))
        .run(ProcessLogger(s => s.startsWith("Python 3.8") || s.startsWith("Python 3.9")))
        .exitValue())
    } else {
      Try(Process(Seq("sh", "-c", "python3 --version"))
        .run(ProcessLogger(s => s.startsWith("Python 3.8") || s.startsWith("Python 3.9")))
        .exitValue())
    }
    attempt.isSuccess && attempt.get == 0
  }
{code}
It's trying to evaluate the version of Python using a {{ProcessLogger}}, but the logger accepts a {{String => Unit}} function, i.e., it does not make use of the return value in any way (since it's meant for logging). So the result of the {{startsWith}} checks are thrown away, and {{attempt.isSuccess && attempt.get == 0}} will always be true as long as your system has a {{python3}} binary of any version.
Bug: RDD file loading APIs throw URISyntaxException when there is a colon in the file path
Status: Open | Priority: Major | Resolution: 
Created: 25/Oct/21 14:48
Affects Version/s: 3.2.0

When a colon is present in a path to a file, many of Spark's RDD file loading APIs (textFile, wholeTextFile, possible others), throw a URISyntaxException.  The following Scala code and stack trace example was generated on my laptop running Spark 3.2.0.   I've verified that this issue also affects Python, and SQL and I'm assuming it probably also affects Java.
{code:java}
scala> val df = sc.wholeTextFiles("/Users/brady.tello/test:me/test.txt").take(1) java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: test:me at org.apache.hadoop.fs.Path.initialize(Path.java:259) at org.apache.hadoop.fs.Path.<init>(Path.java:217) at org.apache.hadoop.fs.Path.<init>(Path.java:125) at org.apache.hadoop.fs.Globber.doGlob(Globber.java:229) at org.apache.hadoop.fs.Globber.glob(Globber.java:149) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2034) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:303) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:274) at org.apache.spark.input.WholeTextFileInputFormat.setMinPartitions(WholeTextFileInputFormat.scala:52) at org.apache.spark.rdd.WholeTextFileRDD.getPartitions(WholeTextFileRDD.scala:54) at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:300) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.rdd.RDD.partitions(RDD.scala:296) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49) at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:300) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.rdd.RDD.partitions(RDD.scala:296) at org.apache.spark.rdd.RDD.$anonfun$take$1(RDD.scala:1428) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:414) at org.apache.spark.rdd.RDD.take(RDD.scala:1422) ... 47 elided Caused by: java.net.URISyntaxException: Relative path in absolute URI: test:me at java.base/java.net.URI.checkPath(URI.java:1990) at java.base/java.net.URI.<init>(URI.java:780) at org.apache.hadoop.fs.Path.initialize(Path.java:256) ... 68 more
{code}
Why can't I just not use colons in my paths you ask?  I'm running Spark on top of an S3 environment in which users are only permitted to read and write data to their personal S3 workspace and the path to their personal workspace contains a colon.  Removing that colon would be a major architectural change to the entire authentication architecture for several apps outside of our Spark app and thus we don't really have the flexibility to remove it.  Without a fix to this bug, users simply cannot use the RDD APIs.
Bug: Fix MiMa failure with Scala 2.13
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Oct/21 16:22
Affects Version/s: 3.3.0

SPARK-36151 re-enabled MiMa for Scala 2.13 but it always fails in the scheduled build.
https://github.com/apache/spark/runs/3992588994?check_suite_focus=true#step:9:2303
Bug: Where clause and where operator will report error on varchar column type
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 22/Oct/21 07:28
Affects Version/s: 3.1.1

create table test1(col1 int, col2 varchar(120)) stored as orc;
 insert into test1 values(123, 'abc');
 insert into test1 values(1234, 'abcd');

 

sparkSession.sql(‘select * from test1’)
 is OK，but

sparkSession.sql(‘select * from test1 where col2 = “abc”’)
 or
 sparkSession.sql(‘select * from test1’).where(‘col2 = “abc”’)

report error：

java.lang.UnsuppotedOperationException: DataType: varchar(120)
Bug: In class ShuffleBlockPusher, use config instead of key
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Oct/21 11:30
Affects Version/s: 3.3.0

In class ShuffleBlockPusher
{code:java}
// code placeholder
private[this] val maxBytesInFlight =
  conf.getSizeAsMb("spark.reducer.maxSizeInFlight", "48m") * 1024 * 1024
private[this] val maxReqsInFlight = conf.getInt("spark.reducer.maxReqsInFlight", Int.MaxValue)
{code}
We can use config.REDUCER_MAX_SIZE_IN_FLIGHT and config.REDUCER_MAX_REQS_IN_FLIGHT instead.
Bug: Upgrade the version of RDBMS and corresponding JDBC drivers used by docker-integration-tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/21 04:34
Affects Version/s: 3.3.0

Let's upgrade the version of RDBMS and corresponding JDBC drivers. 
Especially, PostgreSQL 14 was released recently so it's great to ensure that the JDBC source for PostgreSQL works with PostgreSQL 14.
Bug: Switch from Maven to SBT to build Spark on AppVeyor
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Oct/21 23:20
Affects Version/s: 3.3.0

Recently, building Spark on AppVeyor almost always fails due to StackOverflowError at compile time.
We can't identify the reason so far but one workaround would be building with SBT.
Bug: ParquetFileFormat registers task completion listeners lazily, causing Python writer thread to segfault when off-heap vectorized reader is enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/21 18:43
Affects Version/s: 3.0.3

The task completion listener that closes the vectorized reader is registered lazily in ParquetFileFormat#buildReaderWithPartitionValues(). Since task completion listeners are executed in reverse order of registration, it always runs before the Python writer thread can be interrupted.

This contradicts the assumption in https://issues.apache.org/jira/browse/SPARK-37088 / https://github.com/apache/spark/pull/34245 that task completion listeners are registered bottom-up, preventing that fix from working properly.
Bug: DateTimeUtils.stringToTimestamp() incorrectly rejects timezone without colon
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Oct/21 04:26
Affects Version/s: 3.1.0

For the zoneid with format like "+0000" or "+0730", it can be parsed by `ZoneId.of()` but will rejected by Spark's `DateTimeUtils.stringToTimestamp()`. it means we will return null for some valid datetime string, such as: `2021-10-11T03:58:03.000+0700`
Bug: Enhance ParquetSchemaConverter to capture Parquet repetition & definition level
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Oct/21 00:33
Affects Version/s: 3.3.0

In order to support complex type for Parquet vectorized reader, we'll need to capture the repetition & definition level information associated with Catalyst Spark type converted from Parquet {{MessageType}}.
Bug: Add overloads for lpad and rpad for BINARY strings
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/21 17:55
Affects Version/s: 3.3.0

Currently, `lpad` and `rpad` accept BINARY strings as input (both in terms of input string to be padded and padding pattern), and these strings get cast to UTF8 strings. The result of the operation is a UTF8 string which may be invalid as it can contain non-UTF8 characters.

What we would like to do is to overload `lpad` and `rpad` to accept BINARY strings as inputs (both for the string to be padded and the padding pattern) and produce a left or right padded BINARY string as output.
Bug: Introduce a new data source for providing consistent set of rows per microbatch
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Oct/21 20:45
Affects Version/s: 3.3.0

The "rate" data source has been known to be used as a benchmark for streaming query.

While this helps to put the query to the limit (how many rows the query could process per second), the rate data source doesn't provide consistent rows per batch into stream, which leads two environments be hard to compare with.

For example, in many cases, you may want to compare the metrics in the batches between test environments (like running same streaming query with different options). These metrics are strongly affected if the distribution of input rows in batches are changing, especially a micro-batch has been lagged (in any reason) and rate data source produces more input rows to the next batch.

Also, when you test against streaming aggregation, you may want the data source produces the same set of input rows per batch (deterministic), so that you can plan how these input rows will be aggregated and how state rows will be evicted, and craft the test query based on the plan.

The requirements of new data source would follow:

* it should produce a specific number of input rows as requested
* it should also include a timestamp (event time) into each row
** to make the input rows fully deterministic, timestamp should be configured as well (like start timestamp & amount of advance per batch)
Bug: Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/21 01:43
Affects Version/s: 3.3.0

When execute sort operator, we first compare the prefix. However the getPrefix function of byte array is slow. We use first 8 bytes as the prefix, so at most we will call 8 times with `Platform.getByte` which is slower than call once with `Platform.getInt` or `Platform.getLong`.
Bug: Improve byte array sort by unify compareTo function of UTF8String and ByteArray 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/21 04:46
Affects Version/s: 3.3.0

BinaryType use `TypeUtils.compareBinary` to compare two byte array, however it's slow since it compares byte array using unsigned int comparison byte by bye.

We can compare them using `Platform.getLong` with unsigned long comparison if they have more than 8 bytes. And here is some histroy about this `TODO`  [https://github.com/apache/spark/pull/6755/files#r32197461 .|https://github.com/apache/spark/pull/6755/files#r32197461]
Bug: Report accurate shuffle block size if its skewed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Oct/21 02:07
Affects Version/s: 3.3.0

Now map task will report accurate shuffle block size if the block size is greater than "spark.shuffle.accurateBlockThreshold"( 100M by default ). But if there are a large number of map tasks and the shuffle block sizes of these tasks are smaller than "spark.shuffle.accurateBlockThreshold", there may be unrecognized data skew.

For example, there are 10000 map task and 10000 reduce task, and each map task create 50M shuffle blocks for reduce 0, and 10K shuffle blocks for the left reduce tasks, reduce 0 is data skew, but the stat of this plan do not have this information.

    !map_status2.png!

I think we need to judge if a shuffle block is huge and need to be accurate reported while running.
Bug: Add KubernetesCustom[Driver/Executor]FeatureConfigStep developer API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Oct/21 10:43
Affects Version/s: 3.2.0

SPARK-33261 provides us with great convenience, but it only construct a `KubernetesFeatureConfigStep` with a empty construction method.

It would be better to use the construction method with `KubernetesConf` (or more detail: `KubernetesDriverConf` and `KubernetesExecutorConf`)
Bug: Improving schema filtering flexibility
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Oct/21 11:48
Affects Version/s: 3.3.0

Our spark usage, requires us to build an external schema and pass it on while creating a DataSet.

While working through this, I found an optimization would improve greatly Spark's flexibility to query external schema management.

Scope: ability to retrieve a field's name and schema in one single call, requesting to return a tupple by index. 

Means extending the StructType class to support an additional method

This is what the function would look like:

/**
 * Returns the index and field structure by name.
 * If it doesn't find it, returns None.
 * Avoids two client calls/loops to obtain consolidated field info.
*
*/
def getIndexAndFieldByName(name: String): Option[(Int, StructField)] = \{   val field = nameToField.get(name)   if(field.isDefined) {     Some((fieldIndex(name), field.get))   }
else \{     None   }
}

This is particularly useful from an efficiency perspective, when we're parsing a Json structure and we want to check for every field what is the name and field type already defined in the schema

I will create a corresponding branch for PR review, assuming that there are no concerns with the above proposal.
Bug: InferConstraints rule should create IsNotNull constraints on the nested field instead of the root nested type 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Oct/21 23:45
Affects Version/s: 3.3.0

[InferFiltersFromConstraints|https://github.com/apache/spark/blob/05c0fa573881b49d8ead9a5e16071190e5841e1b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L1206] optimization rule generates {{IsNotNull}} constraints corresponding to null intolerant predicates. The {{IsNotNull}} constraints are generated on the attribute inside the corresponding predicate. 
 e.g. A predicate {{a > 0}} on an integer column {{a}} will result in a constraint {{IsNotNull(a)}}. On the other hand a predicate on a nested int column {{structCol.b}} where {{structCol}} is a struct column results in a constraint {{IsNotNull(structCol)}}.

This generation of constraints on the root level nested type is extremely conservative as it could lead to materialization of the the entire struct. The constraint should instead be generated on the nested field being referenced by the predicate. In the above example, the constraint should be {{IsNotNull(structCol.b)}} instead of {{IsNotNull(structCol)}}

 

The new constraints also create opportunities for nested pruning. Currently {{IsNotNull(structCol)}} constraint would preclude pruning of {{structCol}}. However the constraint {{IsNotNull(structCol.b)}} could create opportunities to prune {{structCol}}.
Bug:  Add a 'kill' executor link in the Web UI.
Status: Closed | Priority: Major | Resolution: Won't Do
Created: 17/Oct/21 04:23
Affects Version/s: 3.3.0

The executor which is running in a bad node(eg. The system is overloaded or disks are busy) or has big GC overheads may affect the efficiency of job execution, although there are speculative mechanisms to resolve this problem, but sometimes the speculated task may also run in a bad executor.
 We should have a 'kill' link for each executor, similar to what we have for each stage, so it's easier for users to kill executors in the UI.