av9ash/GitBugs huggingface dataset:
Bug: Update MiMa plugin
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Aug/20 19:11
Affects Version/s: 3.1.0

As a part of upgrade to SBT 1.x it was found that MiMa (in form of sbt-mima plugin) needs to be upgraded too.

After changes in build file to apply plugin changes between versions 0.3.0 and 0.7.0 binary incompatiblity errors are found during `mimaReportBinaryIssues` task run.

In summary there are of two types:

unclear, if should be checked or excluded
{code:java}
[error] (examples / mimaReportBinaryIssues) mimaPreviousArtifacts not set, not analyzing binary compatibility 
[error] (assembly / mimaReportBinaryIssues) mimaPreviousArtifacts not set, not analyzing binary compatibility 
[error] (tools / mimaReportBinaryIssues) mimaPreviousArtifacts not set, not analyzing binary compatibility 
[error] (streaming-kafka-0-10-assembly / mimaReportBinaryIssues) mimaPreviousArtifacts not set, not analyzing binary compatibility
{code}
and found compatibility errors
{code:java}
[error] (mllib-local / mimaReportBinaryIssues) Failed binary compatibility check against org.apache.spark:spark-mllib-local_2.12:2.4.0! Found 2 potential problems (filtered 2) 
[error] (graphx / mimaReportBinaryIssues) Failed binary compatibility check against org.apache.spark:spark-graphx_2.12:2.4.0! Found 3 potential problems (filtered 3) 
[error] (streaming-kafka-0-10 / mimaReportBinaryIssues) Failed binary compatibility check against org.apache.spark:spark-streaming-kafka-0-10_2.12:2.4.0! Found 6 potential problems (filtered 1) 
[error] (sql / mimaReportBinaryIssues) Failed binary compatibility check against org.apache.spark:spark-sql_2.12:2.4.0! Found 24 potential problems (filtered 1451) 
[error] (streaming / mimaReportBinaryIssues) Failed binary compatibility check against org.apache.spark:spark-streaming_2.12:2.4.0! Found 124 potential problems (filtered 4) 
[error] (mllib / mimaReportBinaryIssues) Failed binary compatibility check against org.apache.spark:spark-mllib_2.12:2.4.0! Found 636 potential problems (filtered 579) 
[error] (core / mimaReportBinaryIssues) Failed binary compatibility check against org.apache.spark:spark-core_2.12:2.4.0! Found 1030 potential problems (filtered 1047)
{code}
I could not take a decision on my own in those cases, so help is needed.
Bug: Direct Date and timestamp format data insertion fails
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 25/Aug/20 12:46
Affects Version/s: 3.0.0

Direct Date and timestamp format data tried to be inserted but the data insertion fails as shown below.

spark-sql> create table test(no timestamp) stored as parquet;
Time taken: 0.561 seconds


spark-sql> insert into test select '1979-04-27 00:00:00';
Error in query: Cannot write incompatible data to table '`default`.`test`':
- Cannot safely cast 'no': string to timestamp;
Bug: Add percentage of missingness to df.summary()
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 25/Aug/20 14:42
Affects Version/s: 3.0.0

 
h2. In df.summary(), we are returned counts of non-nulls for each column. It would be really helpful to have a percentage of non-nulls, since percentage of missingness is often an indicator for data scientists to discard the columns.
Bug: Support for treating the line as valid record if it starts with \u0000 or null character, or starts with any character mentioned as comment
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Aug/20 04:42
Affects Version/s: 2.2.3

In most of the data ware housing scenarios files does not have comment records and every line needs to be treated as a valid record even though it starts with default comment character as \u0000 or null character.Though user can set any comment character other than \u0000, but there is a chance the actual record can start with those characters.

Currently for the below piece of code and the given testdata where first row starts with null \u0000
character it will throw the below error.

*eg: val df = spark.read.option("delimiter",",").csv("file:/E:/Data/Testdata.dat");
      df.show(false);*

*+TestData+*
 
 !screenshot-1.png! 

Internal state when error was thrown: line=1, column=0, record=0, charIndex=7
	at com.univocity.parsers.common.AbstractParser.handleException(AbstractParser.java:339)
	at com.univocity.parsers.common.AbstractParser.parseLine(AbstractParser.java:552)
	at org.apache.spark.sql.execution.datasources.csv.TextInputCSVDataSource$.inferFromDataset(CSVDataSource.scala:160)
	at org.apache.spark.sql.execution.datasources.csv.TextInputCSVDataSource$.infer(CSVDataSource.scala:148)
	at org.apache.spark.sql.execution.datasources.csv.CSVDataSource.inferSchema(CSVDataSource.scala:62)
	at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat.inferSchema(CSVFileFormat.scala:57)

*Note:*

Though its the limitation of the univocity parser and the workaround is to provide any other comment character by mentioning .option("comment","#"), but if my actual data starts with this character then the particular row will be discarded.

Currently I pushed the code in univocity parser to handle this scenario as part of the below PR
https://github.com/uniVocity/univocity-parsers/pull/412

please accept the jira so that we can enable this feature in spark-csv by adding a parameter in spark csvoptions.
Bug: Add suggestion for parallel directory listing in tuning doc
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Aug/20 23:16
Affects Version/s: 3.0.0

Sometimes directory listing could become a bottleneck when user jobs have large number of input directories. This is especially true when against object store like S3. 

There are a few parameters to tune this. This proposes to add some info in the tuning guide so that the knowledge can be better shared.
Bug: Replace deprecated API calls from SpecificParquetRecordReaderBase
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Aug/20 19:43
Affects Version/s: 3.0.0

Currently in {{SpecificParquetRecordReaderBase}} we use deprecated APIs in a few places from Parquet, such as {{readFooter}}, {{ParquetInputSplit}}, deprecated ctor for {{ParquetFileReader}}, {{filterRowGroups}}, etc. These are going to be removed in some of the future Parquet versions so we should move to the new APIs for better maintainability.
Bug: converting dataframe to dataset for the json schema
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 14/Aug/20 15:14
Affects Version/s: 2.4.3

have a schema 

 

{

Details :[{

phone : "98977999"

contacts: [{

name:"manjay"

---------- has missing street block in json

]}

]}

 

}

 

Case class , based on schema

case class Details (

 phone : String,

contacts : Array[Adress]

)

 

case class Adress(

name : String

street : String

 

)

 

 

this throws : No such struct field street - Analysis exception.

 

dataframe.as[Details]

 

Is this a bug ?? or there is a resolution for this.
Bug: Cannot load a Pipeline Model on a stopped Spark Context
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 04/Aug/20 22:02
Affects Version/s: 2.4.6

I am running Spark in a Kubernetes cluster than is running Spark NLP using the Pyspark ML Pipeline Model class to load the model and then transform on the spark dataframe. We run this within a docker container that starts up a spark context, mounts volumes, spins up executors, etc and then does it transformations, udfs, etc and then closes down the spark context. The first time I load the model when my service has just been started, everything is fine. If I run my application for a second time without resetting my service, even though the context is entirely stopped from the previous run and a new one is started up, the Pipeline Model has some attribute in one of its base classes that thinks the context its running on is closed, so then I get a : cannot call a function on a stopped spark context when I try and load the model in my service again. I have to shut down my service each time if I want consecutive runs through my spark pipeline, which is not ideal, so I was wondering if this was a common issue amongst fellow pyspark users that use Pipeline Model, or is there a common work around to resetting all spark contexts or whether the pipeline model caches a spark context of some sort. Any help is very useful. 

 

 
cls.pipeline = PipelineModel.read().load(NLP_MODEL)
 
is how I load the model. And our spark context is very similar to a typical kubernetes/spark setup. Nothing special there
Bug: mapreduce.fileoutputcommitter.algorithm.version default depends on runtime environment
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Aug/20 18:44
Affects Version/s: 2.4.0

When someone reads the documentation in its current state, the assumption is that the default value of spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version is 1 and that's not entirely accurate. 

Spark doesn't explicitly set this configuration and instead is inherited from Hadoop's FileOutputCommitter class. The default value is 1 until Hadoop 3.0 where this changed to 2.

I'm proposing that we clarify that this value's default will depend on the Hadoop version in a user's runtime environment, where:

1 for < Hadoop 3.0
2 for >= Hadoop 3.0

There are also plans to revert this default again to v1 so might also be useful to reference this JIRA:
https://issues.apache.org/jira/browse/MAPREDUCE-7282
Bug: Support writing Hive non-ORC/Parquet bucketed table 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Aug/20 06:12
Affects Version/s: 3.1.0

Hive non-ORC/Parquet write code path is original Hive table write path (InsertIntoHiveTable). This JIRA is to support write hivehash bucketed table (for Hive 1.x.y and 2.x.y), and hive murmur3hash bucketed table (for Hive 3.x.y), for these non-ORC/Parquet-serde Hive bucketed table.
Bug: Write Hive ORC/Parquet bucketed table with hivehash (for Hive 1,2)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Aug/20 06:03
Affects Version/s: 3.1.0

Hive ORC/Parquet write code path is same as data source v1 code path (FileFormatWriter). This JIRA is to add the support to write Hive ORC/Parquet bucketed table with hivehash. The change is to custom `bucketIdExpression` to use hivehash when the table is Hive bucketed table, and the Hive version is 1.x.y or 2.x.y.

 

This will allow us write Hive/Presto-compatible bucketed table for Hive 1 and 2.
Bug: Introduce sort-based fallback mechanism for shuffled hash join 
Status: In Progress | Priority: Minor | Resolution: 
Created: 17/Aug/20 06:22
Affects Version/s: 3.1.0

A major pain point for spark users to stay away from using shuffled hash join is out of memory issue. Shuffled hash join tends to have OOM issue because it allocates in-memory hashed relation (`UnsafeHashedRelation` or `LongHashedRelation`) for build side, and there's no recovery (e.g. fallback/spill) once the size of hashed relation grows and cannot fit in memory. On the other hand, shuffled hash join is more CPU and IO efficient than sort merge join when joining one large table and a small table (but small table is too large to be broadcasted), as SHJ does not sort the large table, but SMJ needs to do that.

To improve the reliability of shuffled hash join, a fallback mechanism can be introduced to avoid shuffled hash join OOM issue completely. Similarly we already have a fallback to sort-based aggregation for hash aggregate. The idea is:

(1).Build hashed relation as current, but monitor the hashed relation size when inserting each build side row. If size of hashed relation being always smaller than a configurable threshold, go to (2.1), else go to (2.2).

(2.1).Current shuffled hash join logic: reading stream side rows and probing hashed relation.

(2.2).Fall back to sort merge join: Sort stream side rows, and sort build side rows (iterate rows already in hashed relation (e.g. through `BytesToBytesMap.destructiveIterator`), then iterate rest of un-read build side rows). Then doing sort merge join for stream + build side rows.

 

Note:

(1).the fallback is dynamic and happened per task, which means task 0 can incur the fallback e.g. if it has a big build side, but task 1,2 don't need to incur the fallback depending on the size of hashed relation.

(2).there's no major code change for SHJ and SMJ. Major change is around HashedRelation to introduce some new methods, e.g. `HashedRelation.destructiveValues()` to return an Iterator of build side rows in hashed relation and cleaning up hashed relation along the way.

(3).we have run this feature by default in our internal fork more than 2 years, and we benefit a lot from it with users can choose to use SHJ, and we don't need to worry about SHJ reliability (see https://issues.apache.org/jira/browse/SPARK-21505 for the original proposal from our side, I tweak here to make it less intrusive and more acceptable, e.g. not introducing a separate join operator, but doing the fallback automatically inside SHJ operator itself).
Bug: Do not fall back to default parallelism if the minimum number of coalesced partitions is not set in AQE
Status: Resolved | Priority: Minor | Resolution: Won't Do
Created: 25/Aug/20 12:47
Affects Version/s: 3.0.0

Currently in AQE when coalescing shuffling partitions,
{quote}We fall back to Spark default parallelism if the minimum number of coalesced partitions is not set, so to avoid perf regressions compared to no coalescing.
{quote}
From our experience, this has resulted in a lot of uncertainty of the number of tasks after coalescing especially with dynamic allocation, and also lead to many small output files. It's complex and hard to reason about.

Hence, I'm proposing not falling back to the default parallelism but coalescing towards the target size when the minimum number of coalesced partitions is not set.
Bug: Add code-gen for SortAggregateExec
Status: Open | Priority: Minor | Resolution: 
Created: 31/Aug/20 05:49
Affects Version/s: 3.1.0

We have codegen for hash aggregate (`HashAggregateExec`) for a long time, but missing codegen for sort aggregate (`SortAggregate`). Sort aggregate is still useful in terms of performance if (1). the data after aggregate still too big to fit in memory (both hash aggregate and object hash aggregate needs to spill), (2).user can disable hash aggregate and object hash aggregate by config to prefer sort aggregate if the aggregate is after e.g. sort merge join and do not need sort at all.

Create this Jira to add codegen support for sort aggregate.
Bug: Add code-gen for ObjectHashAggregateExec
Status: Open | Priority: Minor | Resolution: 
Created: 31/Aug/20 05:54
Affects Version/s: 3.1.0

Object hash aggregate is useful when aggregate function is not supported by hash aggregate, and hash aggregate has codegen long time ago. Do not see a strong reason why we don't have object hash aggregate. Create this Jira to add codegen for object hash aggregate for better CPU execution performance.
Bug: Not able to see driver logs in spark history server in standalone mode
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Aug/20 06:19
Affects Version/s: 2.3.0

Driver logs are not coming in history server in spark standalone mode. Checked in the spark events logs it is not there. Is this by design or can I fix it by creating a patch?. Not able to see any proper documentation regarding this.

 

!image-2020-08-12-11-50-01-899.png!
Bug: Add a test case for hive serde/default-serde mode's null value '\\N'
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Aug/20 14:51
Affects Version/s: 3.2.0

Hive serde default NULL value is '\N'
{code:java}
String nullString = tbl.getProperty(
    serdeConstants.SERIALIZATION_NULL_FORMAT, "\\N");
nullSequence = new Text(nullString);
{code}
Bug: Let sql/catalyst module tests pass for Scala 2.13
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Aug/20 07:42
Affects Version/s: 3.0.0

sql/catalyst module has following compile errors with scala-2.13 profile:
{code:java}
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1284: type mismatch;
 found   : scala.collection.mutable.ArrayBuffer[(org.apache.spark.sql.catalyst.expressions.Attribute, org.apache.spark.sql.catalyst.expressions.Attribute)]
 required: Seq[(org.apache.spark.sql.catalyst.expressions.Attribute, org.apache.spark.sql.catalyst.expressions.Attribute)]
[INFO] [Info] : scala.collection.mutable.ArrayBuffer[(org.apache.spark.sql.catalyst.expressions.Attribute, org.apache.spark.sql.catalyst.expressions.Attribute)] <: Seq[(org.apache.spark.sql.catalyst.expressions.Attribute, org.apache.spark.sql.catalyst.expressions.Attribute)]?
[INFO] [Info] : false
[ERROR] [Error] /Users/baidu/SourceCode/git/spark-mine/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1289: type mismatch;
 found   : scala.collection.mutable.ArrayBuffer[(org.apache.spark.sql.catalyst.expressions.Attribute, org.apache.spark.sql.catalyst.expressions.Attribute)]
 required: Seq[(org.apache.spark.sql.catalyst.expressions.Attribute, ?)]
[INFO] [Info] : scala.collection.mutable.ArrayBuffer[(org.apache.spark.sql.catalyst.expressions.Attribute, org.apache.spark.sql.catalyst.expressions.Attribute)] <: Seq[(org.apache.spark.sql.catalyst.expressions.Attribute, ?)]?
[INFO] [Info] : false
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1297: type mismatch;
 found   : scala.collection.mutable.ArrayBuffer[(org.apache.spark.sql.catalyst.expressions.Attribute, org.apache.spark.sql.catalyst.expressions.Attribute)]
 required: Seq[(org.apache.spark.sql.catalyst.expressions.Attribute, org.apache.spark.sql.catalyst.expressions.Attribute)]
[INFO] [Info] : scala.collection.mutable.ArrayBuffer[(org.apache.spark.sql.catalyst.expressions.Attribute, org.apache.spark.sql.catalyst.expressions.Attribute)] <: Seq[(org.apache.spark.sql.catalyst.expressions.Attribute, org.apache.spark.sql.catalyst.expressions.Attribute)]?
[INFO] [Info] : false
[ERROR] [Error] /Users/baidu/SourceCode/git/spark-mine/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:952: type mismatch;
 found   : scala.collection.mutable.ArrayBuffer[org.apache.spark.sql.catalyst.plans.logical.LogicalPlan]
 required: Seq[org.apache.spark.sql.catalyst.plans.logical.LogicalPlan]
{code}
Similar as https://issues.apache.org/jira/browse/SPARK-29292 ,  call .toSeq on these to ensue they still works on 2.12.
Bug: Override alter table in JDBC dialects
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 03/Aug/20 22:17
Affects Version/s: 3.1.0

SPARK-32402 adds new method alterTable to JdbcDialect with the default implementation for add column, rename column, delete column, update column type, update column nullability. However, different database servers may have different syntax for these update table commands. This ticket aims to override the default alter table syntax with the correct ones in all JDBC dialects.
Bug: Spark executors on K8S do not request extra memory for off-heap allocations
Status: Closed | Priority: Minor | Resolution: Duplicate
Created: 19/Aug/20 12:52
Affects Version/s: 3.0.0

Off-heap memory allocations are configured using `spark.memory.offHeap.enabled=true` and `conf spark.memory.offHeap.size=<size>`. Spark on YARN adds the off-heap memory size to the executor container resources. Spark on Kubernetes does not request the allocation of the off-heap memory. Currently, this can be worked around by using spark.executor.memoryOverhead to reserve memory for off-heap allocations. This proposes make Spark on Kubernetes behave as in the case of YARN, that is adding spark.memory.offHeap.size to the memory request for executor containers.
Bug: Logging plan changes for execution
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Aug/20 00:57
Affects Version/s: 3.1.0

Since we only log plan changes for analyzer/optimizer now, this ticket targets adding code to log plan changes in the preparation phase in QueryExecution for execution.

{code}
scala> spark.sql("SET spark.sql.optimizer.planChangeLog.level=WARN")
scala> spark.range(10).groupBy("id").count().queryExecution.executedPlan
...
20/08/26 09:32:36 WARN PlanChangeLogger: 
=== Applying Rule org.apache.spark.sql.execution.CollapseCodegenStages ===
!HashAggregate(keys=[id#19L], functions=[count(1)], output=[id#19L, count#23L])              *(1) HashAggregate(keys=[id#19L], functions=[count(1)], output=[id#19L, count#23L])
!+- HashAggregate(keys=[id#19L], functions=[partial_count(1)], output=[id#19L, count#27L])   +- *(1) HashAggregate(keys=[id#19L], functions=[partial_count(1)], output=[id#19L, count#27L])
!   +- Range (0, 10, step=1, splits=4)                                                          +- *(1) Range (0, 10, step=1, splits=4)
         
20/08/26 09:32:36 WARN PlanChangeLogger: 
=== Result of Batch Preparations ===
!HashAggregate(keys=[id#19L], functions=[count(1)], output=[id#19L, count#23L])              *(1) HashAggregate(keys=[id#19L], functions=[count(1)], output=[id#19L, count#23L])
!+- HashAggregate(keys=[id#19L], functions=[partial_count(1)], output=[id#19L, count#27L])   +- *(1) HashAggregate(keys=[id#19L], functions=[partial_count(1)], output=[id#19L, count#27L])
!   +- Range (0, 10, step=1, splits=4)  
{code}
Bug: LiteralGenerator for float and double does not generate special values
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Aug/20 16:37
Affects Version/s: 3.1.0

Values like Double.NaN, Double.PositiveInfinity, Double.NegativeInfinity are never returned.
The main usage of LiteralGenerator is in the checkConsistencyBetweenInterpretedAndCodegen method.
This would have detected SPARK-32640
Bug: Remove the words "experimental" in the k8s document
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Aug/20 02:27
Affects Version/s: 3.0.1

To make users understood more correctly about the current development status of the k8s scheduler, this ticket targets at updating the k8s document in the primary branch/branch-3.0;

BEFORE:
{code:java}
 The Kubernetes scheduler is currently experimental. In future versions, there may be behavioral changes around 
configuration, container images and entrypoints.{code}
AFTER:
{code:java}
{code}
This comes from a thread in the spark-dev mailing list: [http://apache-spark-developers-list.1001551.n3.nabble.com/spark-on-k8s-is-still-experimental-td29942.html]
Bug: Poor performance when casting invalid decimal string to decimal type
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Aug/20 12:34
Affects Version/s: 3.1.0

How to reproduce this issue:

{code:java}
spark.read.parquet("/path/to/part-00000.parquet").selectExpr("cast(bd as decimal(18, 0)) as x").write.mode("overwrite").save("/tmp/spark/decimal")
{code}