av9ash/GitBugs huggingface dataset:
Bug: from_json changes times of timestmaps by several minutes without error 
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 09/Feb/20 18:04
Affects Version/s: 2.4.4

When a json text column includes a timestamp and the timestamp has a format like {{2020-01-25T06:39:45.887429Z}}, the function {{from_json(Column,StructType)}} is able to infer a timestamp but that timestamp is changed by several minutes. 
Spark does not throw any kind of error but continues to run with the invalidated timestamp. 

The following scala snipped is able to reproduce the issue.
 
{code:scala}
import org.apache.spark.sql._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._

val df = Seq("""{"time":"2020-01-25T06:39:45.887429Z"}""").toDF("json")

val struct = new StructType().add("time", TimestampType, nullable = true)

val timeDF = df
  .withColumn("time (string)", get_json_object(col("json"), "$.time"))
  .withColumn("time casted directly (CORRECT)", col("time (string)").cast(TimestampType))
  .withColumn("time casted via struct (INVALID)", from_json(col("json"), struct))

display(timeDF)
{code}


Output: 
||json||time (string)||time casted directly (CORRECT)||time casted via struct (INVALID)
|{"time":"2020-01-25T06:39:45.887429Z"}|2020-01-25T06:39:45.887429Z|2020-01-25T06:39:45.887+0000|{"time":"2020-01-25T06:54:32.429+0000"}
Bug: Wrong results of `converTz` for different session and system time zones
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 04/Feb/20 20:34
Affects Version/s: 3.0.0

Currently, DateTimeUtils.convertTz() assumes that timestamp strings are casted to TimestampType using the JVM system timezone but in fact the session time zone defined by the SQL config *spark.sql.session.timeZone* is used in the casting. This leads to wrong results of from_utc_timestamp and to_utc_timestamp when session time zone is different from JVM time zones. The issues can be reproduces by the code:
{code:java}
  test("to_utc_timestamp in various system and session time zones") {
    val localTs = "2020-02-04T22:42:10"
    val defaultTz = TimeZone.getDefault
    try {
      DateTimeTestUtils.outstandingTimezonesIds.foreach { systemTz =>
        TimeZone.setDefault(DateTimeUtils.getTimeZone(systemTz))
        DateTimeTestUtils.outstandingTimezonesIds.foreach { sessionTz =>
          withSQLConf(
            SQLConf.DATETIME_JAVA8API_ENABLED.key -> "true",
            SQLConf.SESSION_LOCAL_TIMEZONE.key -> sessionTz) {

            DateTimeTestUtils.outstandingTimezonesIds.foreach { toTz =>
              val instant = LocalDateTime
                .parse(localTs)
                .atZone(DateTimeUtils.getZoneId(toTz))
                .toInstant
              val df = Seq(localTs).toDF("localTs")
              val res = df.select(to_utc_timestamp(col("localTs"), toTz)).first().apply(0)
              if (instant != res) {
                println(s"system = $systemTz session = $sessionTz to = $toTz")
              }
            }
          }
        }
      }
    } catch {
      case NonFatal(_) => TimeZone.setDefault(defaultTz)
    }
  }
{code}
{code:java}
system = UTC session = PST to = UTC
system = UTC session = PST to = PST
system = UTC session = PST to = CET
system = UTC session = PST to = Africa/Dakar
system = UTC session = PST to = America/Los_Angeles
system = UTC session = PST to = Antarctica/Vostok
system = UTC session = PST to = Asia/Hong_Kong
system = UTC session = PST to = Europe/Amsterdam
...
{code}
Bug: first_value/last_value window function throws ParseException
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 03/Feb/20 08:01
Affects Version/s: 3.0.0

first_value/last_value throws ParseException

 
{code:java}
SELECT first_value(unique1) over w,
last_value(unique1) over w, unique1, four
FROM tenk1 WHERE unique1 < 10
+WINDOW w AS (order by four range between current row and unbounded following)
 
org.apache.spark.sql.catalyst.parser.ParseException
 
no viable alternative at input 'first_value'(line 1, pos 7)
 
== SQL ==
SELECT first_value(unique1) over w,
-------^^^
last_value(unique1) over w, unique1, four
FROM tenk1 WHERE unique1 < 10
WINDOW w AS (order by four range between current row and unbounded following)
{code}
 

Maybe we need fix this issue.
Bug: stage level scheduling: Add ability to set dynamic allocation configs
Status: Open | Priority: Major | Resolution: 
Created: 06/Feb/20 15:50
Affects Version/s: 3.1.0

the initial Jira to modify dynamic allocation for stage level scheduling applies the configs - minimum, initial, max number of executors to each profile individually. This means that you can't set those different per resource profile.

Ideally those would be configurable per ResourceProfile so look at adding support for that.
Bug: stage level scheduling: Better cleanup of Resource profiles
Status: Open | Priority: Major | Resolution: 
Created: 06/Feb/20 15:48
Affects Version/s: 3.1.0

In the initial stage level scheduling jiras we aren't doing a great job of cleaning up datastructures when the ResourceProfile is done being used. This is mostly because its hard to tell when its done being used.

We should find a way to clean up better.

Also in the dynamic allocation manager, if the resource profile is not being used anymore we should be more active about getting rid of the executors that are up. Especially if there is a minimum number that is set, we can kill those off.

 

I think this can be done as followup to the main feature
Bug: fix DataFrameAggregateSuite when enabling AQE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/20 03:47
Affects Version/s: 3.0.0

This is a follow up for [https://github.com/apache/spark/pull/26813#discussion_r373044512].

We need to fix test DataFrameAggregateSuite with AQE on.
Bug: Support subexpression elimination in whole stage codegen
Status: Open | Priority: Major | Resolution: 
Created: 01/Feb/20 09:34
Affects Version/s: 3.1.0

Please see https://github.com/apache/spark/blob/a3a42b30d04009282e770c289b043ca5941e32e5/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala#L2011-L2067 for more details.
Bug: Change `SkewedPartitionReaderExec` into `UnaryExecNode` and replace the direct link with dependent stages with a reuse link
Status: Open | Priority: Major | Resolution: 
Created: 03/Feb/20 16:55
Affects Version/s: 3.0.0

Otherwise it breaks two assumptions in AQE:
 # The only aqe leaf nodes are `AdaptiveSparkPlanExec`, `QueryStageExec`.
 # The plan is strictly a tree.
Bug: Bumps checkstyle from 8.25 to 8.29.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/20 02:44
Affects Version/s: 3.0.0

I found checkstyle have a new release [https://checkstyle.org/releasenotes.html#Release_8.29]

Bumps checkstyle from 8.25 to 8.29.
Bug: Remove manual backport of Murmur3 MurmurHash3.productHash fix from Scala 2.13
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Feb/20 08:38
Affects Version/s: 3.1.0

SPARK-30847 introduced a manual backport to work around a Scala issue in hash implementation. Once we drop Scala 2.12, we can remove the fix.
Bug: Refine doc-building workflow
Status: Reopened | Priority: Minor | Resolution: 
Created: 19/Feb/20 19:15
Affects Version/s: 3.1.0

There are a few rough edges in the workflow for building docs that could be refined:
 * sudo pip installing stuff
 * no pinned versions of any doc dependencies
 * using some deprecated options
 * race condition with jekyll serve
Bug: The default checkpointing interval is not as claimed in the comment.
Status: Resolved | Priority: Minor | Resolution: Incomplete
Created: 10/Feb/20 15:57
Affects Version/s: 2.4.5

[https://github.com/apache/spark/blob/71737861531180bbda9aec8d241b1428fe91cab2/streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala#L199-L203MajorMajor]

-The checkpoint duration is set to be the window duration, maybe the idea in the old comment wanting to set to the higher of 10s or window-size is no longer relevant.-

-I propose we either adapt the comment to just say to just say that we set the checkpoint duration to the window size and clean up how that value is set, or we change the code to do as the comment remarks.-

 

So, the original statement I made was wrong. This code is still broken though. Consider the case where window duration is 3, the result would be a checkpoint size of 12s. That doesn't correspond to the rule implied by the comment and is thus unexpected behaviour.

This code does however result in the checkpoint size being a multiple of the slide duration, which is safe as far as I know.
Bug: Storage Memory in Spark Web UI means
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 06/Feb/20 14:10
Affects Version/s: 2.4.0

Does the denominator of 'Storage Memory' in Spark Web UI include execution memory?
In my environment, set 'spark.executor.memory' to 20g and the denominator of 'Storage Memory' is 11.3g. I think it is too big just include storage memory.
Bug: Add Generic Algorithm optimizer feature to spark-ml
Status: Resolved | Priority: Minor | Resolution: Incomplete
Created: 11/Feb/20 09:37
Affects Version/s: 2.4.5

Hi. 

It seems that spark does not have Generic Algoritm Optimizer.

I think that this algorithm fit well in distributed system like spark.

It is aimed to solve problems like Traveling Salesman Problem,graph partitioning, Optimizing Network topology ... etc

 

Is there some reason that Spark does not include this feature?

 

Can i work on this?
Bug: How to write using JDBC driver to SQL Server / Azure DWH to column of BINARY type?
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 26/Feb/20 10:04
Affects Version/s: 2.4.4

My initial goal is to save UUId values to SQL Server/Azure DWH to column of BINARY(16) type.

For example, I have demo table:
{code:java}
CREATE TABLE [Events] ([EventId] [binary](16) NOT NULL){code}
 

I want to write data to it using Spark like this:
{code:java}
import java.util.UUID

val uuid = UUID.randomUUID()

val uuidBytes = Array.ofDim[Byte](16)
ByteBuffer.wrap(uuidBytes)
    .order(ByteOrder.BIG_ENDIAN)
    .putLong(uuid.getMostSignificantBits())
    .putLong(uuid.getLeastSignificantBits()

val schema = StructType(
    List(
        StructField("EventId", BinaryType, false)
    )
 )

val data = Seq((uuidBytes)).toDF("EventId").rdd;
val df = spark.createDataFrame(data, schema);

df.write
    .format("jdbc")
    .option("url", "<DATABASE_CONNECTION_URL>")
    .option("dbTable", "Events")
    .mode(org.apache.spark.sql.SaveMode.Append)
    .save()
{code}
 

This code returns an error: 
{noformat}
java.sql.BatchUpdateException: Conversion from variable or parameter type VARBINARY to target column type BINARY is not supported.{noformat}
 

My question is how to cope with this situation and insert UUId value to BINARY(16) column?

 

My investigation:

Spark uses conception of JdbcDialects and has a mapping for each Catalyst type to database type and vice versa. For example here is [MsSqlServerDialect|[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/jdbc/MsSqlServerDialect.scala]] which is used when we work against SQL Server or Azure DWH. In the method `getJDBCType` you can see the mapping:
{code:java}
case BinaryType => Some(JdbcType("VARBINARY(MAX)", java.sql.Types.VARBINARY)){code}
and this is the root of my problem as I think.

 

So, I decide to implement my own JdbcDialect to override this behavior:
{code:java}
class SqlServerDialect extends JdbcDialect {
    override def canHandle(url: String) : Boolean = url.startsWith("jdbc:sqlserver")

    override def getJDBCType(dt: DataType): Option[JdbcType] = dt match {
        case BinaryType => Option(JdbcType("BINARY(16)", java.sql.Types.BINARY))
        case _ => None
    }
 }

val dialect = new SqlServerDialect
JdbcDialects.registerDialect(dialect)
{code}
With this modification I still catch exactly the same error. It looks like that Spark do not use mapping from my custom dialect. But I checked that the dialect is registered. So it is strange situation.
Bug: TimeZone in writing pure date type in CSV output
Status: Resolved | Priority: Minor | Resolution: Incomplete
Created: 03/Feb/20 07:31
Affects Version/s: 2.4.3

If I read string date from CSV file, then cast to date type and write into CSV file again on the west of Greenwich to csv file again it writes date of one day ago. This way making this operation in loop we can get unwillingly past date.

If the spark-shell work on the east of Greenwich all is OK.

When writing to parquet also is OK.

Example of code:
{code:java}
//
val test_5_load = "hdfs://192.168.44.161:8020/db/wbiernacki/test_5_load.csv"
val test_5_save = "hdfs://192.168.44.161:8020/db/wbiernacki/test_5_save.csv"
val test_5 = spark.read.format("csv")
  .option("header","true")
  .load( test_5_load )
  .withColumn("begin",to_date(col("begin" ),"yyyy-MM-dd"))
  .withColumn("end" ,to_date(col("end" ),"yyyy-MM-dd"))
test_5.show()
test_5
  .write.mode("overwrite")
  .format("csv")
  .option("header","true")
  .save( test_5_save )

{code}
 Please perform this few times.. The test_5_load.csv file looks like:
{code:java}
// 
+--------+----------+----------+----+
| patient|     begin|       end| new|
+--------+----------+----------+----+
|waldemar|2015-09-22|2015-09-23|old1|
+--------+----------+----------+----+{code}
Bug: Extract o.a.s.sql.catalyst.util.ParseMode to external project
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 14/Feb/20 20:01
Affects Version/s: 2.4.5

In multiple projects downstream of Spark, we use an enum
{code:java}
enum ValidationStringency { STRICT, LENIENT, SILENT }
{code}
This is similar to but not the same as [ParseMode|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ParseMode.scala] in the SQL Catalyst utility package
{code:java}
enum ParseMode { FAILFAST, DROPMALFORMED, PERMISSIVE }
{code}
I wonder if there might be interest in extracting this enum to a project external to Spark, along with some related utility code, and refactoring Spark to use that external reference?

It would also be more convenient to use a Java enum instead of Scala one, if that does not cause trouble for Spark.
Bug: method setQueryTimeout not support
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 14/Feb/20 12:43
Affects Version/s: 2.4.0

I found 
{code:java}
statement.setQueryTimeout(options.queryTimeout)
{code}
in org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD.$resolveTable 

but the default version for hive-jdbc in spark2.x is 1.2.1.spark2:
{code:java}
@Override
public void setQueryTimeout(int seconds) throws SQLException {
  throw new SQLException("Method not supported");
}
{code}
thus when I use hive-jdbc in spark, it will not work:
{code:java}
diagnostics: User class threw exception: java.sql.SQLException: Method not supporteddiagnostics: User class threw exception: java.sql.SQLException: Method not supported at org.apache.hive.jdbc.HiveStatement.setQueryTimeout(HiveStatement.java:739) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:60) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:210) at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:35) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167) 
{code}
 I fixed this problem by repackaging hive-jdbc-1.2.1.spark2 :
{code:java}
  @Override
  public void setQueryTimeout(int seconds) throws SQLException {
//    throw new SQLException("Method not supported");
  }
{code}
Do we have a better way?
Bug: support envFrom configMapRef
Status: Resolved | Priority: Minor | Resolution: Incomplete
Created: 06/Feb/20 04:15
Affects Version/s: 2.4.4

Add a configuration parameter to add environment variables to executor and driver from ConfigMap. Something like:
{code:java}
spark.kubernetes.executor.envFromConfigMapRef=myConfigMap1,myConfigMap2
{code}
which should result in spec:
{code:java}
envFrom:
- configMapRef:
    name: myConfigMap1
- configMapRef:
    name: myConfigMap2
{code}
Bug: spark-submit - no need to resolve dependencies in kubernetes cluster mode
Status: Resolved | Priority: Minor | Resolution: Incomplete
Created: 28/Feb/20 02:22
Affects Version/s: 2.4.5

[https://github.com/apache/spark/blob/branch-2.4/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L301]

 

when use spark-operator, observed that spark operator is trying to download all the dependencies, but it is really not necessary as the driver will do it again.

 should the check be ```

if (!isMesosCluster && !isStandAloneCluster && !isKubernetesCluster) {
 }

```
Bug: Dataframe .limit() performance improvements
Status: Resolved | Priority: Minor | Resolution: Incomplete
Created: 11/Feb/20 17:18
Affects Version/s: 2.4.0

It seems that
{code:java}
.limit(){code}
is much less efficient than it could be/one would expect when reading a large dataset from parquet:
{code:java}
val sample = spark.read.parquet("/Some/Large/Data.parquet").limit(1000)
// Do something with sample ...{code}
This might take hours, depending on the size of the data.

By comparison,
{code:java}
spark.read.parquet("/Some/Large/Data.parquet").show(1000){code}
is essentially instant.
Bug: Javadoc of Java classes doesn't show up in Scala doc
Status: Resolved | Priority: Minor | Resolution: Incomplete
Created: 10/Feb/20 19:12
Affects Version/s: 2.4.5

There are some APIs written in Java. However, Javadoc of Java classes doesn't show up in Scala doc. For example, http://spark.apache.org/docs/2.4.5/api/scala/index.html#org.apache.spark.sql.SaveMode

Scala users need to look at Javadoc for these Java classes right now.

This looks like a unidoc bug, but it's worth to investigate whether we can fix it since we are adding more and more Java classes.
Bug: Make all legacy SQL configs as internal configs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Feb/20 09:38
Affects Version/s: 3.0.0

Assumed that legacy SQL configs shouldn't be set by users in common cases. The purpose of the configs is to allow switching to old behavior in corner cases. So, the configs can be marked as internals. The ticket aims to inspect existing SQL configs in SQLConf and add internal() call to config entry builders.
Bug: Update roxygen2 to 7.0.1
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Feb/20 10:50
Affects Version/s: 3.1.0

Currently Spark uses {{roxygen2}} 5.0.1. It is already pretty old (2015-11-11) so it could be a good idea to use current R updates to update it as well.

At crude inspection:

* SPARK-22430 has been resolved a while ago.
* SPARK-30737][SPARK-27262,  https://github.com/apache/spark/pull/27437 and https://github.com/apache/spark/commit/b95ccb1d8b726b11435789cdb5882df6643430ed resolved persisting warnings
* Documentation builds and CRAN checks pass
* Generated HTML docs are identical to 5.0.1

Since {{roxygen2}} shares some potentially unstable dependencies with {{devtools}} (primarily {{rlang}}) it might be a good idea to keep these in sync (as a bonus we wouldn't have to worry about {{DESCRIPTION}} being overwritten by local tests).
Bug: FileStreamSource: Avoid reading compact metadata log twice if the query stops from compact batch and restarts
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Feb/20 14:04
Affects Version/s: 3.1.0

When restarting the query, there is a case which the query starts from compaction batch, and the batch has source metadata file to read. One case is that the previous query succeeded to read from inputs, but not finalized the batch for various reasons.

This case FileStreamSource will read the compact metadata file twice, one for retrieving all files to build seen file map, another one for retrieving entries in the batch. If the query processes huge number of inputs so far, compact metadata file becomes considerably bigger, so reading once more adds unnecessary latency on processing startup batch.

This issue tracks the effort to address this case.
Bug: Thrift server returns wrong timestamps/dates strings before 1582
Status: Resolved | Priority: Minor | Resolution: Cannot Reproduce
Created: 13/Feb/20 09:15
Affects Version/s: 3.0.0

Set the environment variable:
{code:bash}
export TZ="America/Los_Angeles"
./bin/spark-sql -S
{code}
{code:sql}
spark-sql> set spark.sql.session.timeZone=America/Los_Angeles;
spark.sql.session.timeZone	America/Los_Angeles
spark-sql> SELECT DATE_TRUNC('MILLENNIUM', DATE '1970-03-20');
1001-01-01 00:07:02
{code}

The expected result must be *1001-01-01 00:00:00*.
Bug: Improve the decommissioning K8s integration tests
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Feb/20 20:32
Affects Version/s: 3.0.0

See [https://github.com/apache/spark/pull/26440#discussion_r373155825] & [https://github.com/apache/spark/pull/26440#discussion_r373153511]
Bug: Use JRE instead of JDK in K8S integration test
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Feb/20 23:17
Affects Version/s: 3.0.0

This will save some resources and make it sure we only needs JRE at runtime and testing.

- https://lists.apache.org/thread.html/3145150b711d7806a86bcd3ab43e18bcd0e4892ab5f11600689ba087%40%3Cdev.spark.apache.org%3E
Bug: Improve insertInto behaviour
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 14/Feb/20 08:19
Affects Version/s: 3.1.0

Actually when you call *_insertInto_* to add a dataFrame into an existing table the only safety check is that the number of columns match, but the order doesn't matter, and the message in case that the number of columns doesn't match is not very helpful, specially when you have  a lot of columns:
{code:java}
 org.apache.spark.sql.AnalysisException: `default`.`table` requires that the data to be inserted have the same number of columns as the target table: target table has 2 column(s) but the inserted data has 1 column(s), including 0 partition column(s) having constant value(s).; {code}
I think a standard column check would be very helpful, just like in almost other cases with Spark:

 
{code:java}
"cannot resolve 'p2' given input columns: [id, p1];"  
{code}
Bug: %PYTHONPATH% not set in python/docs/make2.bat, resulting in failed/wrong documentation builds
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Feb/20 21:06
Affects Version/s: 2.4.5

When building the PySpark documentation on Windows, by changing directory to {{python/docs}} and running {{make.bat}} (which runs {{make2.bat}}), the majority of the documentation may not be built if {{pyspark}} is not in the default {{%PYTHONPATH%}}. Sphinx then reports that {{pyspark}} (and possibly dependencies) cannot be imported.

If {{pyspark}} is in the default {{%PYTHONPATH%}}, I suppose it is that version of {{pyspark}} – as opposed to the version found above the {{python/docs}} directory – that is considered when building the documentation, which may result in documentation that does not correspond to the development version one is trying to build.

{{python/docs/Makefile}} avoids this issue by setting
 ??export PYTHONPATH=$(realpath ..):$(realpath ../lib/py4j-0.10.8.1-src.zip)??
 on line 10, but {{make2.bat}} does no such thing. The fix consist of adding
 ??set PYTHONPATH=..;..\lib\py4j-0.10.8.1-src.zip??
 to {{make2.bat}}.

See [GitHub PR #27569|https://github.com/apache/spark/pull/27569].