av9ash/GitBugs huggingface dataset:
Bug: percentile_approx() returns incorrect results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Sep/20 07:22
Affects Version/s: 2.3.4

Read input data from the attached CSV file:
{code:scala}
      val df = spark.read.option("header", "true")
        .option("inferSchema", "true")
        .csv("/Users/maximgekk/tmp/percentile_approx-input.csv")
        .repartition(1)
      df.createOrReplaceTempView(table)
{code}
Calculate the 0.77 percentile with accuracy 1e-05:
{code:Scala}
      spark.sql(
        s"""SELECT
           |  percentile_approx(tr_rat_resampling_score, 0.77, 100000)
           |FROM $table
           """.stripMargin).show
{code}
{code}
+------------------------------------------------------------------------+
|percentile_approx(tr_rat_resampling_score, CAST(0.77 AS DOUBLE), 100000)|
+------------------------------------------------------------------------+
|                                                                    1000|
+------------------------------------------------------------------------+
{code}
 The same for smaller accuracy 0.001:
{code}
+----------------------------------------------------------------------+
|percentile_approx(tr_rat_resampling_score, CAST(0.77 AS DOUBLE), 1000)|
+----------------------------------------------------------------------+
|                                                                    18|
+----------------------------------------------------------------------+
{code} 
and better accuracy 1e-06:
{code}
+-------------------------------------------------------------------------+
|percentile_approx(tr_rat_resampling_score, CAST(0.77 AS DOUBLE), 1000000)|
+-------------------------------------------------------------------------+
|                                                                       17|
+-------------------------------------------------------------------------+
{code}

For the accuracy 1e-05, the result must be around 17-18 but not 1000.

Here is percentile calculation in Google Sheets for the same input:
https://docs.google.com/spreadsheets/d/1Y1i4Td6s9jZQ-bD4IRTESLXP3UxKpqJSXGtmx0Q5TA0/edit?usp=sharing
Bug: Install mypy on the Jenkins CI workers
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 04/Sep/20 07:10
Affects Version/s: 3.1.0

We want to check the types of the PySpark code. This requires mypy to be installed on the CI. Can you do this [~shaneknapp]? 

Related PR: [https://github.com/apache/spark/pull/29180]

You can install this using pip: [https://pypi.org/project/mypy/] Should be similar to flake8 and sphinx. The latest version is ok! Thanks!
Bug: Metaclasses are broken for a few classes in Python 3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/20 05:57
Affects Version/s: 2.4.0

As of Python 3 {{__metaclass__}} is no longer supported https://www.python.org/dev/peps/pep-3115/.

However, we have multiple classes which where never migrated to Python 3 compatible syntax:

- A number of ML {{Params}}} with {{__metaclass__ = ABCMeta}}
- Some of the SQL {{types}} with {{__metaclass__ = DataTypeSingleton}}


As a result some functionalities are broken in Python 3. For example 


{code:python}
>>> from pyspark.sql.types import BooleanType                                                                                                                                                                      
>>> BooleanType() is BooleanType()                                                                                                                                                                                 
False
{code}

or

{code:python}
>>> import inspect                                                                                                                                                                                                 
>>> from pyspark.ml import Estimator                                                                                                                                                                               
>>> inspect.isabstract(Estimator)                                                                                                                                                                                  
False
{code}

where in both cases we expect to see {{True}}.
Bug: Make Spark scripts working with the spaces in paths on Windows
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Sep/20 06:44
Affects Version/s: 3.0.0

Currently, Spark on Windows does not work with the error messages below:

{code}
>>> SparkSession.builder.getOrCreate()
Presence of build for multiple Scala versions detected (C:\...\assembly\target\scala-2.13 and C:\...\assembly\target\scala-2.12).
Remove one of them or, set SPARK_SCALA_VERSION=2.13 in spark-env.cmd.
Visit https://spark.apache.org/docs/latest/configuration.html#environment-variables for more details about setting environment variables in spark-env.cmd.
Either clean one of them or, set SPARK_SCALA_VERSION in spark-env.cmd.
{code}

We should handle whitespaces in the cmd scripts at Spark.
Bug: Post-port removal of non-API stubs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Sep/20 22:10
Affects Version/s: 3.1.0

To simplify initial port we merge all existing stubs.

However, some of these cover non-API components and are usually dynamically annotated (generated with stubgen).

This includes modules like {{serializers}}, {{utils}}, {{shell}}, {{worker}}, etc.

These can be safely removed as:

- MyPy can infer types from the source, where stub is not present.
- No longer provide value, when corresponding modules are present in the same directory structure.
- Annotations are here primarily to help end users, not Spark developers and many of the annotations cannot be meaningfully refined.

It should also reduce overhead of maintaining annotations (especially when places where we don't guarantee signature stability).
Bug: CSV/JSON data sources should avoid globbing paths when inferring schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/20 07:15
Affects Version/s: 2.4.6

The problem is that when the user doesn't specify the schema when reading a CSV table, The CSV file format and data source needs to infer schema, and it does so by creating a base DataSource relation, and there's a mismatch: *FileFormat.inferSchema* expects actual file paths without glob patterns, but *DataSource.paths* expects file paths in glob patterns.
 An example is demonstrated below:
{code:java}
^
|         DataSource.resolveRelation    tries to glob again (incorrectly) on glob pattern """[abc].csv"""
|         DataSource.apply                      ^
|       CSVDataSource.inferSchema               |
|     CSVFileFormat.inferSchema                 |
|   ...                                         |
|   DataSource.resolveRelation          globbed into """[abc].csv""", should be treated as verbatim path, not as glob pattern
|   DataSource.apply                            ^
| DataFrameReader.load                          |
|                                       input """\[abc\].csv"""
{code}
The same problem exists in the JSON data source as well. Ditto for MLlib's LibSVM data source.
Bug: The error is confusing when resource .amount not provided 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/20 15:22
Affects Version/s: 3.0.0

If the user forgets to specify the .amount when specifying a resource, the error that comes out is confusing, we should improve.

 

$ $SPARK_HOME/bin/spark-shell  --master spark://host9:7077 --conf spark.executor.resource.gpu=1

 
{code:java}
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.propertiesSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).20/09/08 08:19:35 ERROR SparkContext: Error initializing SparkContext.java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1967) at org.apache.spark.resource.ResourceUtils$.$anonfun$listResourceIds$1(ResourceUtils.scala:151) at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) at scala.collection.TraversableLike.map(TraversableLike.scala:238) at scala.collection.TraversableLike.map$(TraversableLike.scala:231) at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198) at org.apache.spark.resource.ResourceUtils$.listResourceIds(ResourceUtils.scala:150) at org.apache.spark.resource.ResourceUtils$.parseAllResourceRequests(ResourceUtils.scala:158) at org.apache.spark.SparkContext$.checkResourcesPerTask$1(SparkContext.scala:2773) at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2884) at org.apache.spark.SparkContext.<init>(SparkContext.scala:528) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2555) at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$1(SparkSession.scala:930) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921) at org.apache.spark.repl.Main$.createSparkSession(Main.scala:106) at $line3.$read$$iw$$iw.<init>(<console>:15) at $line3.$read$$iw.<init>(<console>:42) at $line3.$read.<init>(<console>:44){code}
'
Bug: Standalone Master UI resources in use wrong
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/20 14:53
Affects Version/s: 3.0.0

I was using the standalone deployment with workers with GPUs and the master ui was wrong for:
 * *Resources in use:* 0 / 4 gpu

In this case I had 2 workers, each with 4 gpus, so this total should have been 8.  It seems like its just looking at a single worker.
Bug: Add a test case to ensure changes to spark.sql.optimizer.maxIterations take effect at runtime
Status: Closed | Priority: Major | Resolution: Fixed
Created: 30/Sep/20 23:45
Affects Version/s: 3.1.0

**Add a test case to ensure changes to `spark.sql.optimizer.maxIterations` take effect at runtime.

Currently, there is only one related test case: [https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfSuite.scala#L156]

However, this test case only checks the value of the conf can be changed at runtime. It does not check the updated value is actually used by the Optimizer.
Bug: RDD  different partitions cause didderent results 
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 07/Sep/20 06:03
Affects Version/s: 2.2.0

{code}
class Exec3 {
  private val exec: SparkConf = new SparkConf().setMaster("local[1]").setAppName("exec3")
  private val context = new SparkContext(exec)
  context.setCheckpointDir("checkPoint")
 
  /**
   * get total number by key 
   * in this project desired results are ("apple",25) ("huwei"，20)
   * but in fact i get ("apple",150) ("huawei"，20)
   *   when i change it to local[3] the result is correct
   *  i want to know   which cause it and how to slove it 
   */
  @Test
  def testError(): Unit ={
    val rdd = context.parallelize(Seq(("apple", 10), ("apple", 15), ("huawei", 20)))
    rdd.aggregateByKey(1.0)(
      seqOp = (zero, price) => price * zero,
      combOp = (curr, agg) => curr + agg).collect().foreach(println(_))
    context.stop()
  }
}
{code}
Bug: Accidental Data Deletion on calling saveAsTable
Status: Reopened | Priority: Major | Resolution: 
Created: 02/Sep/20 19:18
Affects Version/s: 2.4.4

{code:java}
df.write.option("path", "/already/existing/path").mode(SaveMode.Append).format("json").saveAsTable(db.table)
{code}
Above code deleted the data present in path "/already/existing/path". This happened because table was already not there in hive metastore however, path given had data. And if table is not present in Hive Metastore, SaveMode gets modified internally to SaveMode.Overwrite irrespective of what user has provided, which leads to data deletion. This change was introduced as part of https://issues.apache.org/jira/browse/SPARK-19583. 

Now, suppose if user is not using external hive metastore (hive metastore is associated with a cluster) and if cluster goes down or due to some reason user has to migrate to a new cluster. Once user tries to save data using above code in new cluster, it will first delete the data. It could be a production data and user is completely unaware of it as they have provided SaveMode.Append or ErrorIfExists. This will be an accidental data deletion.

 

Repro Steps:

 
 # Save data through a hive table as mentioned in above code
 # create another cluster and save data in new table in new cluster by giving same path

 

Proposed Fix:

Instead of modifying SaveMode to Overwrite, we should modify it to ErrorIfExists in class CreateDataSourceTableAsSelectCommand.

Change (line 154)

 
{code:java}
val result = saveDataIntoTable(
 sparkSession, table, tableLocation, child, SaveMode.Overwrite, tableExists = false)
 
{code}
to

 
{code:java}
val result = saveDataIntoTable(
 sparkSession, table, tableLocation, child, SaveMode.ErrorIfExists, tableExists = false){code}
This should not break CTAS. Even in case of CTAS, user may not want to delete data if already exists as it could be accidental.
Bug: Spark/Hive3 interaction potentially causes deadlock
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Sep/20 23:19
Affects Version/s: 3.0.0

This is an issue for applications that share a Spark Session across multiple threads.

sessionCatalog.loadPartition (after checking that the table exists) grabs locks in this order:
 - HiveExternalCatalog
 - HiveSessionCatalog (in Shim_v3_0)

Other operations (e.g., sessionCatalog.tableExists), grab locks in this order:
 - HiveSessionCatalog
 - HiveExternalCatalog

[This|https://github.com/apache/spark/blob/ad6b887541bf90cc3ea830a1a3322b71ccdd80ee/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala#L1332] appears to be the culprit. Maybe db name should be defaulted _before_ the call to HiveClient so that Shim_v3_0 doesn't have to call back into SessionCatalog. Or possibly this is not needed at all, since loadPartition in Shim_v2_1 doesn't worry about the default db name, but that might be because of differences between Hive client libraries.

Reproduction case:
 - You need to have a running Hive 3.x HMS instance and the appropriate hive-site.xml for your Spark instance
 - Adjust your spark.sql.hive.metastore.version accordingly
 - It might take more than one try to hit the deadlock

Launch Spark:
{noformat}
bin/spark-shell --conf "spark.sql.hive.metastore.jars=${HIVE_HOME}/lib/*" --conf spark.sql.hive.metastore.version=3.1
{noformat}
Then use the following code:
{noformat}
import scala.collection.mutable.ArrayBuffer
import scala.util.Random

val tableCount = 4
for (i <- 0 until tableCount) {
  val tableName = s"partitioned${i+1}"
  sql(s"drop table if exists $tableName")
  sql(s"create table $tableName (a bigint) partitioned by (b bigint) stored as orc")
}

val threads = new ArrayBuffer[Thread]
for (i <- 0 until tableCount) {
  threads.append(new Thread( new Runnable {
    override def run: Unit = {
      val tableName = s"partitioned${i + 1}"
      val rand = Random
      val df = spark.range(0, 20000).toDF("a")
      val location = s"/tmp/${rand.nextLong.abs}"
      df.write.mode("overwrite").orc(location)
      sql(
        s"""
        LOAD DATA LOCAL INPATH '$location' INTO TABLE $tableName partition (b=$i)""")
    }
  }, s"worker$i"))
  threads(i).start()
}

for (i <- 0 until tableCount) {
  println(s"Joining with thread $i")
  threads(i).join()
}
println("All done")
{noformat}
The job often gets stuck after one or two "Joining..." lines.

{{kill -3}} shows something like this:
{noformat}
Found one Java-level deadlock:
=============================
"worker3":
  waiting to lock monitor 0x00007fdc3cde6798 (object 0x0000000784d98ac8, a org.apache.spark.sql.hive.HiveSessionCatalog),
  which is held by "worker0"
"worker0":
  waiting to lock monitor 0x00007fdc441d1b88 (object 0x00000007861d1208, a org.apache.spark.sql.hive.HiveExternalCatalog),
  which is held by "worker3"
{noformat}
Bug: Add SparkR API for invoking vector_to_array
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Sep/20 19:51
Affects Version/s: 3.0.0

SPARK-30154 introduced Scala UDF with Python API, that allows us to convert from ML / MLlib vectors to array types.

It would be very useful (arguably more useful than in Python, where we have more user-friendly UDF and Vector API) to have R wrapper.
Bug: Run tests script for Python fails in certain environments
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/20 13:30
Affects Version/s: 3.1.0

When running PySpark test in the local environment with "python/run-tests" command, the following error could occur.

 {code}

Traceback (most recent call last):
 File "<string>", line 1, in <module>
...

raise RuntimeError('''
RuntimeError:
 An attempt has been made to start a new process before the
 current process has finished its bootstrapping phase.

This probably means that you are not using fork to start your
 child processes and you have forgotten to use the proper idiom
 in the main module:

if __name__ == '__main__':
 freeze_support()
 ...

The "freeze_support()" line can be omitted if the program
 is not going to be frozen to produce an executable.
Traceback (most recent call last):
...
 raise EOFError
EOFError

 {code}
Bug: Remove ExpressionSet from the 2.13 branch
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Sep/20 09:46
Affects Version/s: 3.1.0

ExpressionSet does not extent Scala Set anymore, and therefore, can be removed from the 2.13 branch. This is a followup on https://issues.apache.org/jira/browse/SPARK-32755.
Bug: ApplicationInfo#removedExecutors can cause OOM
Status: Open | Priority: Major | Resolution: 
Created: 04/Sep/20 04:26
Affects Version/s: 2.4.0

!image-2020-09-03-23-27-11-809.png|width=840,height=439!

In my case, the Standalone Spark master process had a max heap of 1g. 738mb were consumed by these ExecutorDesc objects, the vast majority of which were the 18.5M removedExecutors. This caused the master to OOM and leave the application driver process dangling.

The reason for this is that the worker node ran out of disk space, so for whatever reason decided to go in a fast and endless loop trying to launch new executors and they in turn crashed too. It got up to the 18M before the master just couldn't handle the history anymore.
Bug: Limit in streaming should not be optimized away by PropagateEmptyRelation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Sep/20 01:31
Affects Version/s: 3.1.0

Right now, the limit operator in a streaming query may get optimized away when the relation is empty. This can be problematic for stateful streaming, as this empty batch will not write any state store files, and the next batch will fail when trying to read these state store files and throw a file not found error.

We should not let PropagateEmptyRelation optimize away the Limit operator for streaming queries.

This ticket is intended to apply a small and safe fix for PropagateEmptyRelation. A fundamental fix that can prevent this from happening again in the future and in other optimizer rules is more desirable, but that's a much larger task.
Bug: Spark ignores limit(1) and starts tasks for all partition
Status: Open | Priority: Major | Resolution: 
Created: 01/Sep/20 03:50
Affects Version/s: 2.4.0

If we run the following code
{code:scala}
  val sparkConf = new SparkConf()
    .setAppName("test-app")
    .setMaster("local[1]")
  val sparkSession = SparkSession.builder().config(sparkConf).getOrCreate()

  import sparkSession.implicits._

  val df = (1 to 100000)
    .toDF("c1")
    .repartition(1000)

  implicit val encoder: ExpressionEncoder[Row] = RowEncoder(df.schema)

  df.limit(1)
    .map(identity)
    .collect()

  df.map(identity)
    .limit(1)
    .collect()

  Thread.sleep(100000)
{code}
we will see that in the first case spark started 1002 tasks despite the fact there is limit(1) -

!image-2020-09-01-10-51-09-417.png!

Expected behavior - both scenarios (limit before and after map) will produce the same results - one or two tasks to get one value from the DataFrame.
Bug: Support for INET data type
Status: Resolved | Priority: Major | Resolution: Later
Created: 01/Sep/20 04:13
Affects Version/s: 2.4.0

PostgreSQL has support for `INET` data type 

[https://www.postgresql.org/docs/9.1/datatype-net-types.html]

We have a few customers that are interested in similar, native support for IP addresses, just like in PostgreSQL.

The issue with storing IP addresses as strings, is that most of the matches (like if an IP address belong to a subnet) in most cases can't take leverage of parquet bloom filters.
Bug: pyspark.mllib.evaluation.MulticlassMetrics needs to swap the results of precision( ) and recall( )
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 16/Sep/20 22:58
Affects Version/s: 3.0.1

[https://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/evaluation/MulticlassMetrics.html]

*The values returned by the precision() and recall() methods of this API should be swapped.*

Following is the example results I got when I run this API. It prints out precision  
metrics = MulticlassMetrics(predictionAndLabels)
print (metrics.confusionMatrix().toArray())
print ("precision: ",metrics.precision(1))
print ("recall: ",metrics.recall(1))
[[36631. 2845.]

[ 3839. 1610.]]

precision: 0.3613916947250281

recall: 0.2954670581758121

 
predictions.select('prediction').agg(\{'prediction':'sum'}).show()
|sum(prediction)| 5449.0|

As you can see, my model predicted 5449 cases with label=1, and 1610 out of the 5449 cases are true positive, so precision should be  1610/5449=0.2954670581758121, but this API assigned the precision value to recall() method, which should be swapped.
Bug: Leverage pip and setup.py configurations to pass Hadoop and Hive options in pip installation
Status: Open | Priority: Major | Resolution: 
Created: 10/Sep/20 03:52
Affects Version/s: 3.1.0

Currently, pip itself allows a custom option to set via {{--install-option}}. However, when you pass this option to {{pip}},  it passes that option to all dependency installation. Please also see https://github.com/pypa/pip/issues/1883

It is very hacky or impossible to use this option to pass the option to a specific package. They are discussing to have a more general option.

Once they have a way to do it, we should remove the environment variables from {{setup.py}} and switch it to proper options.
Bug: SparkSession.builder.getOrCreate should not show deprecation warning of SQLContext
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/20 03:14
Affects Version/s: 3.0.1

In PySpark shell:

{code}
import warnings
from pyspark.sql import SparkSession, SQLContext
warnings.simplefilter('always', DeprecationWarning)
spark.stop()
SparkSession.builder.getOrCreate()
{code}

shows a deprecation warning from {{SQLContext}}

{code}
/.../spark/python/pyspark/sql/context.py:72: DeprecationWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  DeprecationWarning)
{code}
Bug: UnsafeExternalSorter.SpillableIterator cannot spill after reading all records
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Sep/20 12:06
Affects Version/s: 2.4.7

No memory is freed after calling {{UnsafeExternalSorter.SpillableIterator.spill()}} when all records have been read, even though it is still holding onto some memory. This may starve other {{MemoryConsumer}}s of memory.