av9ash/GitBugs huggingface dataset:
Bug: Reading wrong dates from dictionary encoded columns in Parquet files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/May/20 12:03
Affects Version/s: 3.0.0

Write dates with dictionary encoding enabled to parquet files:
{code:scala}
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.1.0-SNAPSHOT
      /_/

Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_242)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.conf.set("spark.sql.legacy.parquet.rebaseDateTimeInWrite.enabled", true)

scala> :paste
// Entering paste mode (ctrl-D to finish)

          Seq.tabulate(8)(_ => "1001-01-01").toDF("dateS")
            .select($"dateS".cast("date").as("date"))
            .repartition(1)
            .write
            .option("parquet.enable.dictionary", true)
            .mode("overwrite")
            .parquet("/Users/maximgekk/tmp/parquet-date-dict")

// Exiting paste mode, now interpreting.
{code}

Read them back:
{code:scala}
scala> spark.read.parquet("/Users/maximgekk/tmp/parquet-date-dict").show(false)
+----------+
|date      |
+----------+
|1001-01-07|
|1001-01-07|
|1001-01-07|
|1001-01-07|
|1001-01-07|
|1001-01-07|
|1001-01-07|
|1001-01-07|
+----------+
{code}

*Expected values must be 1000-01-01.*

I checked that the date column is encoded by dictionary via:
{code}
➜  parquet-date-dict java -jar ~/Downloads/parquet-tools-1.12.0.jar dump ./part-00000-84a77214-0c8c-45e9-ac41-5ca863b9dd94-c000.snappy.parquet
row group 0
--------------------------------------------------------------------------------
date:  INT32 SNAPPY DO:0 FPO:4 SZ:74/70/0.95 VC:8 ENC:BIT_PACKED,RLE,P [more]...
    date TV=8 RL=0 DL=1 DS: 1 DE:PLAIN_DICTIONARY
    ----------------------------------------------------------------------------
    page 0:                  DLE:RLE RLE:BIT_PACKED VLE:PLAIN_DICTIONARY [more]... VC:8
INT32 date
--------------------------------------------------------------------------------
*** row group 1 of 1, values 1 to 8 ***
value 1: R:0 D:1 V:1001-01-07
value 2: R:0 D:1 V:1001-01-07
value 3: R:0 D:1 V:1001-01-07
value 4: R:0 D:1 V:1001-01-07
value 5: R:0 D:1 V:1001-01-07
value 6: R:0 D:1 V:1001-01-07
value 7: R:0 D:1 V:1001-01-07
value 8: R:0 D:1 V:1001-01-07
{code}
Bug: Pass-through of Kerberos credentials from Spark SQL to a jdbc source
Status: Open | Priority: Major | Resolution: 
Created: 25/May/20 10:43
Affects Version/s: 2.4.5

I am connecting to Spark SQL through the Thrift JDBC/ODBC server using kerberos. From Spark SQL I have connected to a JDBC source using basic authentication but I am interested in doing a pass-through of kerberos credentials to this JDBC source. 

Would it be possible to do something like that? If not possible, could you consider adding this functionality?

Anyway I would like to start testing this pass-through and try to develop an approach by myself. How could this functionality be added? Could you give me any indication to start this development?
Bug: DataFrame.inputFiles() not Available
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/May/20 20:29
Affects Version/s: 2.4.5

I have been trying to list inputFiles that compose my DataSet by using *PySpark* 

spark_session.read
 .format(sourceFileFormat)
 .load(S3A_FILESYSTEM_PREFIX + bucket + File.separator + sourceFolderPrefix)
 *.inputFiles();*

but I get an exception saying inputFiles attribute not present. But I was able to get this functionality with Spark Java. 

*So is this something missing in PySpark?*
Bug: Spark Datasource Tables Creating Incorrect Hive Metadata
Status: Reopened | Priority: Major | Resolution: 
Created: 23/May/20 01:29
Affects Version/s: 2.4.5

I found that if I create a CSV or JSON table using Spark SQL, it writes the wrong Hive table metadata, breaking compatibility with other query engines like Hive and Presto. Here is a very simple example:

{code:sql}
CREATE TABLE test_csv (id String, name String)
USING csv
  LOCATION  's3://[...]'
;
{code}

If you describe the table using Presto, you will see:

{code:sql}
CREATE EXTERNAL TABLE `test_csv`(
  `col` array<string> COMMENT 'from deserializer')
ROW FORMAT SERDE 
  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
WITH SERDEPROPERTIES ( 
  'path'='s3://[...]') 
STORED AS INPUTFORMAT 
  'org.apache.hadoop.mapred.SequenceFileInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat'
LOCATION
  's3://[...]/test_csv-__PLACEHOLDER__'
TBLPROPERTIES (
  'spark.sql.create.version'='2.4.4', 
  'spark.sql.sources.provider'='csv', 
  'spark.sql.sources.schema.numParts'='1', 
  'spark.sql.sources.schema.part.0'='{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}', 
  'transient_lastDdlTime'='1590196086')
  ;
{code}

 The table location is set to a placeholder value - the schema is always set to _col array<string>_. The serde/inputformat is wrong - it says _SequenceFileInputFormat_ and _LazySimpleSerDe_ even though the requested format is CSV.

But all the right metadata is written to the custom table properties with prefix _spark.sql_. However, Hive and Presto does not understand these table properties and this breaks them. I could reproduce this with JSON too, but not with Parquet. 

I root-caused this issue to CSV and JSON tables not handled [here|https://github.com/apache/spark/blob/721cba540292d8d76102b18922dabe2a7d918dc5/sql/core/src/main/scala/org/apache/spark/sql/internal/HiveSerDe.scala#L31-L66] in HiveSerde.scala. As a result, these default values are written.

Is there a reason why CSV and JSON are not handled? I could send a patch to fix this, but the caveat is that the CSV and JSON Hive serdes should be in the Spark classpath, otherwise the table creation will fail.
Bug: Stream Data with API to ServiceNow
Status: Resolved | Priority: Major | Resolution: Workaround
Created: 22/May/20 09:53
Affects Version/s: 2.4.5

1) Create class

2) Instantiate class 

3) Setup stream

4) Write stream. Here do I get a pickeling error, I really don't know how to get it work without error.

 

 

class CMDB:
 #Public Properties
 @property
 def streamDF(self):
 return spark.readStream.table(self.__source_table)
 
 #Constructor
 def __init__(self, destination_table, source_table):
 self.__destination_table = destination_table
 self.__source_table = source_table

#Private Methodes 
 def __processRow(self, row):
 #API connection info
 url = 'https://foo.service-now.com/api/now/table/' + self.__destination_table + '?sysparm_display_value=true'
 user = 'username'
 password = 'psw'
 
 headers = \{"Content-Type":"application/json","Accept":"application/json"}
 response = requests.post(url, auth=(user, password), headers=headers, data = json.dumps(row.asDict()))
 
 return response

#Public Methodes
 def uploadStreamDF(self, df):
 return df.writeStream.foreach(self.__processRow).trigger(once=True).start()

 

################################################################################

 

cmdb = CMDB('destination_table_name','source_table_name')

streamDF = (cmdb.streamDF
 .withColumn('object_id',col('source_column_id'))
 .withColumn('name',col('source_column_name'))
 ).select('object_id','name')
#set stream works, able to display data

cmdb.uploadStreamDF(streamDF)
#cmdb.uploadStreamDF(streamDF) fails with error: PicklingError: Could not serialize object: Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063. See exception below:

'''
Exception Traceback (most recent call last)
/databricks/spark/python/pyspark/serializers.py in dumps(self, obj)
 704 try:
--> 705 return cloudpickle.dumps(obj, 2)
 706 except pickle.PickleError:

/databricks/spark/python/pyspark/cloudpickle.py in dumps(obj, protocol)
 862 cp = CloudPickler(file,protocol)
--> 863 cp.dump(obj)
 864 return file.getvalue()

/databricks/spark/python/pyspark/cloudpickle.py in dump(self, obj)
 259 try:
--> 260 return Pickler.dump(self, obj)
 261 except RuntimeError as e:

/databricks/python/lib/python3.7/pickle.py in dump(self, obj)
 436 self.framer.start_framing()
--> 437 self.save(obj)
 438 self.write(STOP)
'''
Bug: Fix recoverPartitions test in DDLSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/May/20 07:32
Affects Version/s: 2.3.4

The recoverPartitions test [[link|https://github.com/apache/spark/blob/v3.0.0-rc2/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala#L1205]]] overrides RDD_PARALLEL_LISTING_THRESHOLD treating it as SQLConf. But RDD_PARALLEL_LISTING_THRESHOLD is not a SQLConf, so it doesn't get overriden in SharedSparkSession and so the existing test doesn't end up testing the required behaviour.
Bug: Incorrect timestamp parsing from JSON
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 19/May/20 09:46
Affects Version/s: 2.4.5

Parsing a json string into a TimestampType can give incorrect results.
{code:python}
schema = StructType([StructField("timestamp", TimestampType())])
df = spark.createDataFrame([('{"timestamp":"2020-01-01T20:00:00.900125Z"}', )], ["body"])
df.select(from_json("body", schema)).collect(){code}
Output:
{code:python}
datetime.datetime(2020, 1, 1, 20, 15, 0, 125000){code}
This seems to happen when the timestamp has sub-millisecond precision and a Z suffix. For example, if the fraction is .900125, the output fraction is .125 and 900 seconds is added to the timestamp.

Workaround: Adding the timestampFormat option fixes the problem, even if the format string is not exactly correct.
{code:python}
df.select(from_json("body", schema, {"timestampFormat": "yyyy-MM-dd HH:mm:ss"})).collect()
{code}
Bug: Json schema reading is not consistent between int and string types
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 20/May/20 16:27
Affects Version/s: 2.4.4

When reading json file using a schema, int value is converted to string if field is string but string field is not converted to int value if field is int.

Sample Code:

read_schema = StructType([StructField({color:#008080}"a"{color}, IntegerType()),
 StructField({color:#008080}"b"{color}, StringType())])
df = {color:#94558d}self{color}.spark_session.read.schema(read_schema).json({color:#008080}"input/json/temp_test"{color})
df.show()

 

json temp_test

{"a": 1,"b": "b1"}
{"a": 2,"b": "b2"}
{"a": 3,"b": 3}
{"a": "4","b": 4}

 

actual:

| a| b|
+----+----+
| 1| b1|
| 2| b2|
| 3| 3|
|null|null|
+----+----+

 

expected:

Third line will be nulled as the fourth line as b is int while in schema it's string.
Bug: Format Java date-time types in Row.jsonValue directly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/May/20 05:36
Affects Version/s: 3.0.0

Row.jsonValue has to convert incoming Java date/timestamps types to days/microseconds because  DateFormatter/TimestampFormatter APIs didn't accept java.sql.Timestamp/java.util.Date and java.time.Instant/java.time.LocalDate before https://issues.apache.org/jira/browse/SPARK-31762. Internally, the formatters perform conversions to Java types again. This badly impacts on the performance. The ticket aims to add new APIs to DateFormatter and TimestampFormatter that should accept Java types.
Bug: Add a helper function to test all parquet readers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/May/20 11:41
Affects Version/s: 3.1.0

Add the method withAllParquetReaders {} which runs the block of code for all supported parquet readers. And re-use it in test suites. This should de-dup code, and allow OSS Spark based projects that have their own parquet readers to re-use existing tests.
Bug: Eliminate UpCast if child's dataType is DecimalType
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/20 14:03
Affects Version/s: 3.1.0

 
{code:java}
sql("select cast(11111111111111111111111111111111111111 as decimal(38, 0)) as d")
  .write.mode("overwrite")
  .parquet(f.getAbsolutePath)

spark.read.parquet(f.getAbsolutePath).as[BigDecimal]
{code}
{code:java}
[info]   org.apache.spark.sql.AnalysisException: Cannot up cast `d` from decimal(38,0) to decimal(38,18).
[info] The type path of the target object is:
[info] - root class: "scala.math.BigDecimal"
[info] You can either add an explicit cast to the input data or choose a higher precision type of the field in the target object;
[info]   at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveUpCast$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveUpCast$$fail(Analyzer.scala:3060)
[info]   at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveUpCast$$anonfun$apply$33$$anonfun$applyOrElse$174.applyOrElse(Analyzer.scala:3087)
[info]   at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveUpCast$$anonfun$apply$33$$anonfun$applyOrElse$174.applyOrElse(Analyzer.scala:3071)
[info]   at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$1(TreeNode.scala:309)
[info]   at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:72)
[info]   at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:309)
[info]   at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$3(TreeNode.scala:314)
{code}
Bug: Improve error message for incompatible data types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/May/20 07:30
Affects Version/s: 3.0.0


{code:java}
spark-sql> create table SPARK_31834(a int) using parquet;
spark-sql> insert into SPARK_31834 select '1';
Error in query: Cannot write incompatible data to table '`default`.`spark_31834`':
- Cannot safely cast 'a': StringType to IntegerType;
spark-sql>
{code}
Bug: BigDecimal setScale is not working in Spark UDF
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 13/May/20 03:23
Affects Version/s: 2.3.4

I was trying to convert json column to map. I tried udf for converting json to map. but it is not working as expected.
  
{code:java}
val df1 = Seq(("{\"k\":10.004}")).toDF("json")
def udfJsonStrToMapDecimal = udf((jsonStr: String)=> { var jsonMap:Map[String,Any] = parse(jsonStr).values.asInstanceOf[Map[String, Any]]
     jsonMap.map{case(k,v) => (k,BigDecimal.decimal(v.asInstanceOf[Double]).setScale(6))}.toMap
})
val f = df1.withColumn("map",udfJsonStrToMapDecimal($"json"))
scala> f.printSchema
root
 |-- json: string (nullable = true)
 |-- map: map (nullable = true)
 |    |-- key: string
 |    |-- value: decimal(38,18) (valueContainsNull = true)
{code}
 

*instead of decimal(38,6) it converting the value as decimal(38,18)*
Bug: Old POSIXlt, POSIXct and Date become corrupt due to calendar difference
Status: Open | Priority: Major | Resolution: 
Created: 13/May/20 11:54
Affects Version/s: 2.4.5

Old POSIXlt, POSIXct and Date become corrupt in SparkR. For example, see below:

{code}
# Non-existent timestamp in hybrid Julian and Gregorian Calendar
showDF(createDataFrame(as.data.frame(list(list(POSIXct=as.POSIXct("1582-10-10 00:01:00"), POSIXlt=as.POSIXlt("1582-10-10 00:01:00"))))))
{code}

{code}
+-------------------+-------------------+
|            POSIXct|            POSIXlt|
+-------------------+-------------------+
|1582-09-30 00:33:08|1582-09-30 00:33:08|
+-------------------+-------------------+
{code}

See https://docs.google.com/document/d/1an3Mzv6s0naO4mDwGFHJ48gLT--6EliA1GG3kbgBymo/edit?usp=sharing

Note that the results seem wrong from the very first implementation. The cause seems because R side uses Proleptic Gregorian calendar but JVM side is using hybrid Juilian and Gregoiran calendar.
Bug: Document resource module in PySpark doc and rename/move classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/20 09:03
Affects Version/s: 3.0.0

SPARK-29641 and SPARK-28234 added new pyspark.resrouce module.
It should be documented as it's an API.

Also, the current structure is as follows:

{code}
pyspark
├── resource
│   ├── executorrequests.py
│   │   ├── class ExecutorResourceRequest 
│   │   └── class ExecutorResourceRequests
│   ├── taskrequests.py
│   │   ├── class TaskResourceRequest 
│   │   └── class TaskResourceRequests
│   ├── resourceprofilebuilder.py
│   │   └── class ResourceProfileBuilder
│   ├── resourceprofile.py
│   │   └── class ResourceProfile
└── resourceinformation
    └── class ResourceInformation
{code}

Might better put into fewer and simpler modules.
Bug: Remove Hive dependency in AppVeyor build temporarily
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/20 01:25
Affects Version/s: 3.0.0

Seems like we're skipping Hive related tests in SparkR (https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark/builds/32853533), see below:

{code}
test_sparkSQL.R:307: skip: create DataFrame from RDD
Reason: Hive is not build with SparkSQL, skipped
test_sparkSQL.R:1341: skip: test HiveContext
Reason: Hive is not build with SparkSQL, skipped
test_sparkSQL.R:2813: skip: read/write ORC files
Reason: Hive is not build with SparkSQL, skipped
test_sparkSQL.R:2834: skip: read/write ORC files - compression option
Reason: Hive is not build with SparkSQL, skipped
test_sparkSQL.R:3727: skip: enableHiveSupport on SparkSession
Reason: Hive is not build with SparkSQL, skipped
{code}

Although we build with Hive profile (https://github.com/apache/spark/blob/master/appveyor.yml#L51).

It looks better to remove the profile for now which could also speed up the build
Bug: Deprecate 'spark.sql.optimizer.metadataOnly'
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/May/20 03:41
Affects Version/s: 3.0.0

This optimization can cause a potential correctness issue, see also SPARK-26709.
Also, it seems difficult to extend the optimization. Basically you should whitelist all available functions.

Seems we should rather deprecate and remove this.
Bug: Allow to set owner reference for the driver pod (cluster mode)
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 18/May/20 10:57
Affects Version/s: 2.4.5

Currently there is no way to pass ownerReferences to the driver pod in cluster mode. This makes it difficult for the upstream process to clean up pods after they completed. 

 

Something like this would be useful:

spark.kubernetes.driver.ownerReferences.[Name]
Bug: Fail to insert data to a table with remote location which causes by hive encryption check
Status: In Progress | Priority: Major | Resolution: 
Created: 11/May/20 09:15
Affects Version/s: 2.4.6

Before this fix https://issues.apache.org/jira/browse/HIVE-14380 in Hive 2.2.0, when moving files from staging dir to the final table dir, Hive will do encryption check for the srcPaths and destPaths


{code:java}
// Some comments here
     if (!isSrcLocal) {
        // For NOT local src file, rename the file
        if (hdfsEncryptionShim != null && (hdfsEncryptionShim.isPathEncrypted(srcf) || hdfsEncryptionShim.isPathEncrypted(destf))
            && !hdfsEncryptionShim.arePathsOnSameEncryptionZone(srcf, destf))
        {
          LOG.info("Copying source " + srcf + " to " + destf + " because HDFS encryption zones are different.");
          success = FileUtils.copy(srcf.getFileSystem(conf), srcf, destf.getFileSystem(conf), destf,
              true,    // delete source
              replace, // overwrite destination
              conf);
        } else {
{code}

The hdfsEncryptionShim instance holds a global FileSystem instance belong to the default fileSystem. It causes failures when checking a path that belongs to a remote file system.

For example, I 
{code:sql}
key	int	NULL

# Detailed Table Information
Database	bdms_hzyaoqin_test_2
Table	abc
Owner	bdms_hzyaoqin
Created Time	Mon May 11 15:14:15 CST 2020
Last Access	Thu Jan 01 08:00:00 CST 1970
Created By	Spark 2.4.3
Type	MANAGED
Provider	hive
Table Properties	[transient_lastDdlTime=1589181255]
Location	hdfs://cluster2/user/warehouse/bdms_hzyaoqin_test.db/abc
Serde Library	org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat	org.apache.hadoop.mapred.TextInputFormat
OutputFormat	org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Storage Properties	[serialization.format=1]
Partition Provider	Catalog
Time taken: 0.224 seconds, Fetched 18 row(s)
{code}

The table abc belongs to the remote hdfs 'hdfs://cluster2', and when we run command below via a spark sql job with default fs is ' 'hdfs://cluster1'
{code:sql}
insert into bdms_hzyaoqin_test_2.abc values(1);
{code}


{code:java}

Error in query: java.lang.IllegalArgumentException: Wrong FS: hdfs://cluster2/user/warehouse/bdms_hzyaoqin_test.db/abc/.hive-staging_hive_2020-05-11_17-10-27_123_6306294638950056285-1/-ext-10000/part-00000-badf2a31-ab36-4b60-82a1-0848774e4af5-c000, expected: hdfs://cluster1
{code}
Bug: Proper base path for location when it is a relative path
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/May/20 08:45
Affects Version/s: 2.4.5

Currently, the user home directory is used as the base path for the database and table locations when their location is specified with a relative path, e.g.
{code:sql}
> set spark.sql.warehouse.dir;
spark.sql.warehouse.dir	file:/Users/kentyao/Downloads/spark/spark-3.1.0-SNAPSHOT-bin-20200512/spark-warehouse/
spark-sql> create database loctest location 'loctestdbdir';

spark-sql> desc database loctest;
Database Name	loctest
Comment
Location	file:/Users/kentyao/Downloads/spark/spark-3.1.0-SNAPSHOT-bin-20200512/loctestdbdir
Owner	kentyao

spark-sql> create table loctest(id int) location 'loctestdbdir';
spark-sql> desc formatted loctest;
id	int	NULL

# Detailed Table Information
Database	default
Table	loctest
Owner	kentyao
Created Time	Thu May 14 16:29:05 CST 2020
Last Access	UNKNOWN
Created By	Spark 3.1.0-SNAPSHOT
Type	EXTERNAL
Provider	parquet
Location	file:/Users/kentyao/Downloads/spark/spark-3.1.0-SNAPSHOT-bin-20200512/loctestdbdir
Serde Library	org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
InputFormat	org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
OutputFormat	org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
{code}


The user home is not always warehouse-related, unchangeable in runtime, and shared both by database and table as the parent directory. Meanwhile, we use the table path as the parent directory for relative partition locations.

the config `spark.sql.warehouse.dir` represents the default location for managed databases and tables. For databases, the case above seems not to follow its semantics. For tables it is right but here I suggest enriching its meaning that is also for external tables with relative paths for locations.