av9ash/GitBugs huggingface dataset:
Bug: spark.storage.blockManagerSlaveTimeoutMs default value does not follow spark.network.timeout value when the latter was changed
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Nov/20 11:12
Affects Version/s: 3.0.0

According to the documentation "spark.network.timeout" is the default timeout for "spark.storage.blockManagerSlaveTimeoutMs" which implies that when the user sets "spark.network.timeout"  the effective value of "spark.storage.blockManagerSlaveTimeoutMs" should also be changed if it was not specifically changed.

However this is not the case since the default value of "spark.storage.blockManagerSlaveTimeoutMs" is always the default value of "spark.network.timeout" (120s)

 

"spark.storage.blockManagerSlaveTimeoutMs" is defined in the package object of "org.apache.spark.internal.config" as follows:
{code:java}
private[spark] val STORAGE_BLOCKMANAGER_SLAVE_TIMEOUT =
  ConfigBuilder("spark.storage.blockManagerSlaveTimeoutMs")
    .version("0.7.0")
    .timeConf(TimeUnit.MILLISECONDS)
    .createWithDefaultString(Network.NETWORK_TIMEOUT.defaultValueString)
{code}
So it seems like the its default value is indeed "fixed" to "spark.network.timeout" default value.
Bug: Incorrect menu item display and link in PySpark Usage Guide for Pandas with Apache Arrow
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Nov/20 06:33
Affects Version/s: 3.0.0

Error setting menu item and link, change "Apache Arrow in Spark" to "Apache Arrow in PySpark"

  !image-2020-11-23-18-47-01-591.png!

 

after:

!image-2020-11-27-09-43-58-141.png!
Bug: Column pruning with monotonically_increasing_id
Status: Open | Priority: Minor | Resolution: 
Created: 25/Nov/20 13:02
Affects Version/s: 2.4.0

{code:java}
df = ss.read.parquet("/path/to/parquet/dataset") df.select("partnerid").withColumn("index", sf.monotonically_increasing_id()).explain(True){code}

 {{We should expect to only read partnerid from parquet dataset but we actually read the whole dataset:}}


{code:java}
... == Physical Plan == Project [partnerid#6794, monotonically_increasing_id() AS index#24939L] +- FileScan parquet [impression_id#6550,arbitrage_id#6551,display_timestamp#6552L,requesttimestamputc#6553,affiliateid#6554,amp_adrequest_type#6555,app_id#6556,app_name#6557,appnexus_viewability#6558,apxpagevertical#6559,arbitrage_time#6560,banner_type#6561,display_type_int#6562,has_multiple_display_types#6563,bannerid#6564,bid_app_id_hash#6565,bid_url_domain_hash#6566,bidding_details#6567,bid_level_core#6568,bidrandomization_user_factor#6569,bidrandomization_user_mu#6570,bidrandomization_user_sigma#6571,big_lastrequesttimestampsession#6572,big_nbrequestaffiliatesession#6573,... 566 more fields] ...{code}
Bug: Add "unused import" check to Maven compilation process
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Nov/20 14:00
Affects Version/s: 3.1.0

Similar to SPARK-33441, need add "unused import" check  to maven pom.

The blocker is how to achieve the same effect as SBT compiler check, It seems that adding "-P:silencer:globalFilters=.*deprecated.*" configuration to "scala-maven-plugin" is not supported at present
Bug: Expose inverse hyperbolic trig functions in PySpark and SparkR
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Nov/20 16:50
Affects Version/s: 3.1.0

{{acosh}}, {{asinh}} and {{atanh}} were exposed in Scala {{sql.functions}} in Spark 3.1 (SPARK-33061).

For consistency, we should expose these in Python and R as well.
Bug: Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Nov/20 12:20
Affects Version/s: 3.1.0

There are two places to call the "SpecificParquetRecordReaderBase.initialize(

InputSplit inputSplit, TaskAttemptContext taskAttemptContext

)" method, one is in ParquetFileFormat and the other one is in ParquetPartitionReaderFactory, 

the "inputSplit.rowGroupOffsets" passed in both places are null, it seems that the branch of "rowgroupoffsets! = null" is useless.
Bug: Support null for conversion from and to Arrow type
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Nov/20 12:06
Affects Version/s: 3.0.1

I got below error when using from_arrow_type() in pyspark.sql.pandas.types

{{Unsupported type in conversion from Arrow: null}}

I noticed NullType exists under pyspark.sql.types so it seems possible to convert from pyarrow null to pyspark null type and vice versa.
Bug: PySpark equivalent of SparkContext.hadoopConfiguration
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 12/Nov/20 19:07
Affects Version/s: 3.1.0

PySpark should offer an API to {{hadoopConfiguration}} to [match Scala's|http://spark.apache.org/docs/latest/api/scala/org/apache/spark/SparkContext.html#hadoopConfiguration:org.apache.hadoop.conf.Configuration].

Setting Hadoop configs within a job is handy for any configurations that are not appropriate as cluster defaults, or that will not be known until run time. The various {{fs.s3a.*}} configs are a good example of this.

Currently, what people are doing is setting things like this [via SparkContext._jsc.hadoopConfiguration()|https://stackoverflow.com/a/32661336/877069].
Bug: Missing submenus for Performance Tuning in Spark SQL Guide 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Nov/20 15:05
Affects Version/s: 3.0.0

Sub-menus for \{Coalesce Hints for SQL Queries} and {Adaptive Query Execution) are missing 

!image-2020-11-30-00-04-07-969.png!
Bug: Simplifying equality of the instance of Option and Some(value)
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 12/Nov/20 11:32
Affects Version/s: 3.1.0

There are some code like "InstanceOfOption == Some(value)",

for example "TaskSetManager#handleSuccessfulTask" method:
{code:java}
def handleSuccessfulTask(tid: Long, result: DirectTaskResult[_]): Unit = {
  ...
  if (successful(index) && killedByOtherAttempt.contains(tid)) {
    ...
    val resultSizeAcc = result.accumUpdates.find(a =>
      a.name == Some(InternalAccumulator.RESULT_SIZE)) 
    ...
}
{code}
We need create a new Option instance "Some(InternalAccumulator.RESULT_SIZE)"  to compare equivalently with a.name.

In this scenario, can use the "Option#contains" method to avoid creating new option instance
Bug: Let ML ALS recommend for BOTH subsets - users and items
Status: Open | Priority: Minor | Resolution: 
Created: 19/Nov/20 10:11
Affects Version/s: 3.0.1

Currently ALS in Spark ML supports next methods for getting recommendations:
 * {{recommendForAllUsers(numItems: Int): DataFrame}}
 * {{recommendForAllItems(numUsers: Int): DataFrame}}
 * {{recommendForUserSubset(dataset: Dataset[_], numItems: Int): DataFrame}}
 * {{recommendForItemSubset(dataset: Dataset[_], numUsers: Int): DataFrame}}

 

*Feature request:* to add a method that recommends subset of items for subset of users, i.e. both users and items are selected from provided subsets. 

*Why it is important:* in real-time recommender systems you usually make predict for current users (that's why we need subset of users). And you can just recommend all items that you have, but only those who satisfy some business filters (that's why we need subset of items). 

*For example:* consider real-time news recommender system. Predict is done for small subset of users (say, for example, visitors for last minute), but it is not allowed to recommend old news or news not related to user country or etc, so at each predict we have some "white" list of items.

So that's why it will be extremely useful to control *BOTH* for which users make recommendations *AND* which items include in these recommendations. 

*Related issues:* -SPARK-20679- , but there is just subsets either on users *OR* items. 

*What we do now:* just make additional filtering after {{recommendForUserSubset}} call, but this method has significant cost - we must receive all items recommendations, i.e. *{{numItems = # all available items}}* and then filter and only then select top-k among them.

*Why it is bad:* usually subset of items allowed to recommend right now is much smaller than the amount of all seen items in an origin data (in my real dataset it is 220k vs 500). 

*Design:* I am sorry - I am not familiar with Spark internals so I offer solution based just on my human logic :) 
{code:scala}
def recommendForUserItemSubsets(userDataset: Dataset[_], 
                                itemDataset: Dataset[_], 
				numItems: Int): DataFrame = {
    val userFactorSubset = getSourceFactorSubset(dataset, userFactors, $(userCol))
    val itemFactorSubset = getSourceFactorSubset(dataset, itemFactors, $(itemCol))
    recommendForAll(userFactorSubset, itemFactorSubset, $(userCol), $(itemCol), numItems, $(blockSize))
}
{code}
 

I will be glad to receive some feedback, is it reasonable request or not and maybe more efficient workarounds. 

 

Thanks!
Bug: Accessibility: spark history server page, the download button has insufficient 2.2:1 contrast .
Status: Open | Priority: Minor | Resolution: 
Created: 25/Nov/20 08:35
Affects Version/s: 2.4.5

In spark history server page, the download button has insufficient 2.2:1 contrast .Visual impaired user will find difficulty to see the control or element if that control or element have insufficient color contrast as user won't be able to perform functionality over such control.
Bug: Use scala.jdk.CollectionConverters replace scala.collection.JavaConverters
Status: Open | Priority: Minor | Resolution: 
Created: 04/Nov/20 17:14
Affects Version/s: 3.1.0

`scala.collection.JavaConverters` is deprecated in Scala 2.13, there are many compilation warnings about this, should use `scala.jdk.CollectionConverters` replace it. 

But `scala.jdk.CollectionConverters` only available in Scala 2.13.
Bug: Enable `-Wunused:imports` in Scala 2.13 SBT
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 20/Nov/20 10:49
Affects Version/s: 3.1.0

As the image, Scala 2.13 think `scala.language.higherKinds` is a {{unused import, }}but Scala 2.12 compile failed without this import, so comments {{-Wunused:imports}} in Scala 2.13 and left a TODO, 

we should enabled `-Wunused:imports` when Scala 2.12 is no longer supported.
Bug: Batch fix compilation warnings about "method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is deprecated (since 2.13.0)"
Status: Open | Priority: Minor | Resolution: 
Created: 05/Nov/20 09:09
Affects Version/s: 3.1.0

compilation warnings as follow：
{code:java}
[WARNING] [Warn] /spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1711: method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is deprecated (since 2.13.0): Implicit conversions from Array to immutable.IndexedSeq are implemented by copying; Use the more efficient non-copying ArraySeq.unsafeWrapArray or an explicit toIndexedSeq call
{code}
this Jira is a placeholder now,  wait until Scala 2.12 is no longer supported
Bug:  Batch fix compilation warnings about "Widening conversion from XXX to XXX is deprecated"
Status: Open | Priority: Minor | Resolution: 
Created: 04/Nov/20 14:55
Affects Version/s: 3.1.0

There is a batch of compilation warnings in Scala 2.13 as follows:
{code:java}
[WARNING] [Warn] /spark/core/src/main/scala/org/apache/spark/input/FixedLengthBinaryInputFormat.scala:77: Widening conversion from Long to Double is deprecated because it loses precision. Write `.toDouble` instead.
{code}
Bug: Fix Compilation warnings of "multiarg infix syntax looks like a tuple and will be deprecated" in Scala 2.13
Status: Open | Priority: Minor | Resolution: 
Created: 04/Nov/20 14:50
Affects Version/s: 3.1.0

There is a batch of compilation warnings in Scala 2.13 as follow:
{code:java}
[WARNING] [Warn] /spark/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:656: multiarg infix syntax looks like a tuple and will be deprecated
{code}
Bug: Imputer support mode(most_frequent) strategy
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Nov/20 09:16
Affects Version/s: 3.1.0

[sklearn.impute.SimpleImputer|https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html#sklearn.impute.SimpleImputer] supports *most_frequent(mode)*, which replace missing using the most frequent value along each column.

It should be easy to implement it in MLlib.
Bug: Incorrect output from example script pi.py
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 07/Nov/20 05:54
Affects Version/s: 2.4.6

 
I have Apache Spark v2.4.6 installed on my mini cluster of 1 driver and 2 worker nodes. To test the installation, I ran the $SPARK_HOME/examples/src/main/python/pi.py script included with Spark-2.4.6. Three runs produced the following output:
 
m4-nn:~:spark-submit  --master spark://[10.0.0.20:7077|http://10.0.0.20:7077/]  /usr/local/spark/examples/src/main/python/pi.py
Pi is roughly 3.149880
m4-nn:~:spark-submit  --master spark://[10.0.0.20:7077|http://10.0.0.20:7077/]  /usr/local/spark/examples/src/main/python/pi.py
Pi is roughly 3.137760
m4-nn:~:spark-submit  --master spark://[10.0.0.20:7077|http://10.0.0.20:7077/]  /usr/local/spark/examples/src/main/python/pi.py
Pi is roughly 3.155640
 
I noted that the computed value of Pi varies with each run.
Next, I ran the same script 3 more times with a higher number of partitions (16). The following output was noted.

m4-nn:~:spark-submit  --master spark://[10.0.0.20:7077|http://10.0.0.20:7077/]  /usr/local/spark/examples/src/main/python/pi.py 16
Pi is roughly 3.141100
m4-nn:~:spark-submit  --master spark://[10.0.0.20:7077|http://10.0.0.20:7077/]  /usr/local/spark/examples/src/main/python/pi.py 16
Pi is roughly 3.137720
m4-nn:~:spark-submit  --master spark://[10.0.0.20:7077|http://10.0.0.20:7077/]  /usr/local/spark/examples/src/main/python/pi.py 16
Pi is roughly 3.145660
 
Again, I noted that the computed value of Pi varies with each run. 
 
IMO, there are 2 issues with this example script:
1. The output (value of pi) is non-deterministic because the script uses random.random(). 
2. Specifying the number of partitions (accepted as a command-line argument) has no observable positive impact on the accuracy or precision. 
 
It may be argued that the intent of these examples scripts is simply to demonstrate how to use Spark as well as offer a means to quickly verify an installation. However, we can achieve that objective without compromising on the accuracy or determinism of the computed value. Unless the user examines the script and understands that use of random.random() (to generate random points within the top right quadrant of the circle) as the reason behind the non-determinism, it seems confusing at first that the value varies per run and also that it is inaccurate. Someone may (incorrectly) infer that as a limitation of the framework!
 
To mitigate this, I wrote an alternate version to compute pi using a partial sum of terms from an infinite series. This script is both deterministic and can produce more accurate output if the user configures it to use more terms. To me, that behavior feels intuitive and logical. I will be happy to share it if it is appropriate.
 
Best regards,
Milind
Bug: enableHiveSupport is invalid after sparkContext that without hive support created 
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 27/Nov/20 12:17
Affects Version/s: 3.1.0

reproduce as follow code:

        SparkConf sparkConf = new SparkConf().setAppName("hello");

        sparkConf.set("spark.master", "local");

        JavaSparkContext jssc = new JavaSparkContext(sparkConf);

        spark = SparkSession.builder()

                .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")

                .config("hive.exec.dynamici.partition", true).config("hive.exec.dynamic.partition.mode", "nonstrict")

                .config("hive.metastore.uris", "thrift://hivemetastore:9083")

                .enableHiveSupport()

                .master("local")

                .getOrCreate();

       spark.sql("select * from hudi_db.hudi_test_order").show();

 

 it will produce follow Exception    

AssertionError: assertion failed: No plan for HiveTableRelation [`hudi_db`.`hudi_test_order` … (at current master branch)  org.apache.spark.sql.AnalysisException: Table or view not found: `hudi_db`.`hudi_test_order`;  (at spark v2.4.4)

  

 the reason is SparkContext#getOrCreate(SparkConf) will return activeContext that include previous spark config if it has

but the input SparkConf is the newest which include previous spark config and options.

  enableHiveSupport will set options (“spark.sql.catalogImplementation", "hive”) when spark session created it will miss this conf

SharedState load conf from sparkContext and will miss hive catalog
Bug: The comment for SQLContext.tables() doesn't mention the `database` column
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Nov/20 14:50
Affects Version/s: 2.4.8

The comment says: "The returned DataFrame has two columns, tableName and isTemporary":
https://github.com/apache/spark/blob/b26ae98407c6c017a4061c0c420f48685ddd6163/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala#L664

but actually the dataframe has 3 columns:
{code:scala}
scala> spark.range(10).createOrReplaceTempView("view1")
scala> val tables = spark.sqlContext.tables()
tables: org.apache.spark.sql.DataFrame = [database: string, tableName: string ... 1 more field]

scala> tables.printSchema
root
 |-- database: string (nullable = false)
 |-- tableName: string (nullable = false)
 |-- isTemporary: boolean (nullable = false)


scala> tables.show
+--------+---------+-----------+
|database|tableName|isTemporary|
+--------+---------+-----------+
| default|       t1|      false|
| default|       t2|      false|
| default|      ymd|      false|
|        |    view1|       true|
+--------+---------+-----------+
{code}
Bug: Support drop column in spark also like in postgresql
Status: Open | Priority: Minor | Resolution: 
Created: 12/Nov/20 07:07
Affects Version/s: 2.4.5

Support alter table with drop column in spark also like in postgresql.

 

[https://www.postgresql.org/docs/12/sql-altertable.html]

alter table tablename DROP COLUMN/columns [ IF EXISTS ]

columnname

 
{code:java}
spark-sql> drop database if exists hivemetastoretest cascade;
Time taken: 1.067 seconds
spark-sql> create database hivemetastoretest;
Time taken: 0.326 seconds
spark-sql> use hivemetastoretest;
Time taken: 0.053 seconds
spark-sql> create table jobit4 using parquet as select 2.5;
Time taken: 5.058 seconds
spark-sql> alter table jobit4 add columns(name string);
Time taken: 1.194 seconds
spark-sql> alter table jobit4 drop columns(name);
Error in query:
mismatched input 'columns' expecting \{'PARTITION', 'IF'}(line 1, pos 25)

== SQL ==
 alter table jobit4 drop columns(name)
-------------------------^^^

spark-sql> alter table jobit4 drop columns name;
Error in query:
mismatched input 'columns' expecting \{'PARTITION', 'IF'}(line 1, pos 25)

== SQL ==
 alter table jobit4 drop columns name
-------------------------^^^

spark-sql> [
{code}
Bug: Remove configuration spark.sql.legacy.allowCastNumericToTimestamp
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Nov/20 04:10
Affects Version/s: 3.1.0

In the current master branch, there is a new configuration `spark.sql.legacy.allowCastNumericToTimestamp` which controls whether to cast Numeric types to Timestamp or not. The default value is true.

After https://github.com/apache/spark/pull/30260, the type conversion between Timestamp type and Numeric type is disallowed in ANSI mode. So, we don't need to a separate configuration `spark.sql.legacy.allowCastNumericToTimestamp` for disallowing the conversion.
We should remove the configuration.
Bug: jcl-over-slf4j conflicts with commons-logging.jar
Status: In Progress | Priority: Minor | Resolution: 
Created: 20/Nov/20 03:17
Affects Version/s: 2.4.7

spark had introduces jcl-over-slf4j as the bridge between commons-logging and slf4j. And refer to:

[https://jira.qos.ch/browse/SLF4J-250?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel]

[http://www.slf4j.org/legacy.html]

because jcl-over-slf4j.jar contains duplicates classes with commons-logging.jar, so it's better to remove the dependency of commons-logging.

And we also find one deadlock issue cause by jcl-over-slf4j.jar and commons-logging.jar coexistence. this issue happen during the cinit of LogFactory and SLF4JlogFactory
Bug: Make apiKey of docsearch configurable
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Nov/20 10:40
Affects Version/s: 3.1.0

After https://github.com/apache/spark/pull/30292, our Spark documentation site supports searching. 
However, the default API key always points to the latest release doc. We have to set different API keys for different releases. Otherwise, the search results are always based on the latest documentation(https://spark.apache.org/docs/latest/) even when visiting the
documentation of previous releases.

As per discussion in https://github.com/apache/spark/pull/30292#issuecomment-725613417, we should make the API key configurable and avoid hardcoding in the HTML template.
Bug: DoubleParam can't parse a JSON number without decimal places
Status: In Progress | Priority: Minor | Resolution: 
Created: 11/Nov/20 12:52
Affects Version/s: 2.4.6

{quote}{color:#0033b3}new {color}DoubleParam({color:#067d17}""{color}, {color:#067d17}""{color}, {color:#067d17}""{color}).jsonDecode({color:#067d17}"1"{color})
{quote}
 

throws an following exception (see below). It would be great if it could make a double from a JInt (trivial)

 

{{ java.lang.IllegalArgumentException: Cannot decode JInt(1) to Double.}}
{{at org.apache.spark.ml.param.DoubleParam$.jValueDecode(params.scala:380)}}
{{at org.apache.spark.ml.param.DoubleParam.jsonDecode(params.scala:349)}}
{{at org.apache.spark.ml.param.DoubleParam.jsonDecode(params.scala:330)}}
Bug: Spark Thrift Server, keep Job Id when using incremental collect
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Nov/20 21:49
Affects Version/s: 3.1.0

When enabling *spark.sql.thriftServer.incrementalCollect* Job Ids get lost and tracing queries in Spark Thrift Server ends up being too complicated.

By fixing the Job Id, queries and Spark jobs are unequivocally related one to each other.
Bug: Kafka flatMapGroupsWithState not always ordered.
Status: Open | Priority: Minor | Resolution: 
Created: 19/Nov/20 05:21
Affects Version/s: 3.0.1

When using a Kafka source and `groupByKey` followed by `flatMapGroupsWithState`, it appears that if a single group contains 500k+ records in a batch, the Iterator supplied to the matching function is not ordered by Kafka offset.

I know that semantically the groups themselves in `groupByKey` are not guaranteed to be ordered, but this seems extremely desirable in the case of Structured Streaming. It consumes a lot of memory to manually sort this Iterator inside of the mapping function and if ordering could be preserved more efficiently by reducing Kafka consumer parallelism, that would be preferable.

If this is considered normal behavior, I believe that the it should be more clear in the docs that ordering is not guaranteed within a batch, and I would appreciate some insight into how to avoid this behavior (I have been fiddling with `maxOffsetsPerTrigger` but I do not know exactly what size batch triggers the issue, and setting it too low could prevent the job from being able to keep up with the topic).