av9ash/GitBugs huggingface dataset:
Bug: to_timestamp() doesn't work with certain timezones
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 24/Jan/20 10:18
Affects Version/s: 2.3.0

It seams that to_timestamp() doesn't work with timezones of the type <Country>/<City>, e.g. America/Los_Angeles.

The code

{code:scala}
val df = Seq(
    ("2019-01-24 11:30:00.123", "America/Los_Angeles"), 
    ("2020-01-01 01:30:00.123", "PST")
).toDF("ts_str", "tz_name")

val ts_parsed = to_timestamp(
    concat_ws(" ", $"ts_str", $"tz_name"), "yyyy-MM-dd HH:mm:ss.SSS z"
).as("timestamp")

df.select(ts_parsed).show(false)
{code}

prints


{code}
+-------------------+
|timestamp          |
+-------------------+
|null               |
|2020-01-01 10:30:00|
+-------------------+
{code}

So, the datetime string with timezone PST is properly parsed, whereas the one with America/Los_Angeles is converted to null. According to [this|https://github.com/apache/spark/pull/24195#issuecomment-578055146] response on GitHub, this code works when run on the recent master version. 

See also the discussion in [this|https://github.com/apache/spark/pull/24195#issue] issue for more context.
Bug: Whether to block streaming batch commit, merge all blocking batchs as one batch commit
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 20/Jan/20 03:03
Affects Version/s: 2.4.4

When the current job is not completed block streaming batch commit, until completed。The next job will merge all batch which  during the blocking.

for example:

 the input seq [1, 2, 3, 4, 5, 6]。 

batch duration: 1s。

The 3th batch will take a long time。Normally the other batches will be completed quickly.

We expect:

      1. the 4th batch will not be commited during the 3th batch computing, and 4th batch will be merge in the next batch. So that

the size of jobSets  is always less than 1。

      2. the num completedBatches less than the size of seq。

      3. the data is not lost
Bug: Unable to load properties fine in SparkStandalone HDFS mode
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 21/Jan/20 11:42
Affects Version/s: 2.4.3

We run the spark application in Yarn HDFS/NFS/WebHDFS and standalone HDFS/NFS mode.

when the application is submitted in standalone HDFS mode the configuration jar(properties file) is not read when the application is started, and this make the logger to fall back to default log file and log level.

So when application is submitted to standalone HDFS, configuration files are not read. 

STD OUT LOGS from Standalone HDFS  - properties file is not found 

{code}
log4j: Trying to find [osa-log.properties] using context classloader sun.misc.Launcher$AppClassLoader@4cdf35a9.
log4j: Trying to find [osa-log.properties] using sun.misc.Launcher$AppClassLoader@4cdf35a9 class loader.
log4j: Trying to find [osa-log.properties] using ClassLoader.getSystemResource().
log4j: Could not find resource: [osa-log.properties].
log4j: Reading configuration from URL jar:[file:/osa/spark-2.4.3-bin-hadoop2.7/jars/spark-core_2.11-2.4.3.jar!/org/apache/spark/log4j-defaults.properties|file:///osa/spark-2.4.3-bin-hadoop2.7/jars/spark-core_2.11-2.4.3.jar!/org/apache/spark/log4j-defaults.properties]
{code}


STD out from Standalone NFS  - properties files are found and able to load it

{code}
log4j: Trying to find [osa-log.properties] using context classloader sun.misc.Launcher$AppClassLoader@4cdf35a9.
log4j: Using URL [jar:file:/scratch/xxxx/gitlocal/soa-osa/out/configdir/ux6m3UQZ/app/sx_DatePipeline_A44C5337_B0D6_4A67_9D60_6BE629DABADA_x0jR7lg5_public/__config__.jar!/osa-log.properties] for automatic log4j configuration.
log4j: Preferred configurator class: OSALogPropertyConfigurator
{code}


STD out from YARN HDFS  — properties files are found and able to load it

{code}
log4j: Trying to find [osa-log.properties] using context classloader sun.misc.Launcher$AppClassLoader@2626b418.
log4j: Using URL [file:/tmp/hadoop-yarn/nm-local-dir/usercache/osa/filecache/16/__osa_config__8272345390991627491.jar/osa-log.properties|file:///tmp/hadoop-yarn/nm-local-dir/usercache/osa/filecache/16/__osa_config__8272345390991627491.jar/osa-log.properties] for automatic log4j configuration.
log4j: Preferred configurator class: OSALogPropertyConfigurator
log4j: configuration [file:/tmp/hadoop-yarn/nm-local-dir/usercache/osa/filecache/16/__osa_config__8272345390991627491.jar/osa-log.properties|file:///tmp/hadoop-yarn/nm-local-dir/usercache/osa/filecache/16/__osa_config__8272345390991627491.jar/osa-log.properties]
{code}
Bug: HistoryServer application link is incorrect when one application having multiple attempts
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 17/Jan/20 07:17
Affects Version/s: 2.4.4

Code is [here|https://github.com/apache/spark/blob/0a1e01c30e2c00d079c7a135a11a5664928c5c1c/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html#L81].

When having multiple attempts, the applicaiton link is incorrect. No matter which attempt is clicked, the link still points to the largest attemptId.
Bug: Spark 2.4.4 metrics endpoint throwing error
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 30/Jan/20 16:30
Affects Version/s: 2.4.4

I am using Spark-standalone in HA mode with zookeeper.

Once the driver is up and running, whenever I try to access the metrics api using the following URL
http://master_address/proxy/app-20200130041234-0123/api/v1/applications
I get the following exception.

It seems that the request never even reaches the spark code. It would be helpful if somebody can help me.

{code:java}
HTTP ERROR 500
Problem accessing /api/v1/applications. Reason:

    Server Error
Caused by:
java.lang.NullPointerException: while trying to invoke the method org.glassfish.jersey.servlet.WebComponent.service(java.net.URI, java.net.URI, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) of a null object loaded from field org.glassfish.jersey.servlet.ServletContainer.webComponent of an object loaded from local variable 'this'
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
	at org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
	at org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584)
	at org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
	at org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
	at org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
	at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
	at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.spark_project.jetty.server.Server.handle(Server.java:539)
	at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333)
	at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
	at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)
	at org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
	at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
	at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
	at java.lang.Thread.run(Thread.java:808)
{code}
Bug: Mechanism to check that all queries of spark structured-streaming are started in case of multiple sink actions.
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 31/Jan/20 11:26
Affects Version/s: 2.3.2

Get the Spark StructuredStreaming job status (start/stop) having multiple sink actions

We are trying to get the status of StructuredStreaming job, below is the requirement

We wanted to push data to a kafkatopic with offset value set to latest, we are using spark-listeners to get the job status, however we observed that listener is invoked because one of the spark query started but complete spark-job isn't actually started as other queries are still initializing, this results in data-loss because we pushed the data to kafka topic and kafka server set the offset inventory value to the latest, as complete spark job is not started yet but listener gets invoked, once spark job is started it didn't consume data from kafka as offset on kafka server has been already set to latest.
Bug: Update WebUI Bootstrap to 4.4.1
Status: Open | Priority: Major | Resolution: 
Created: 27/Jan/20 22:15
Affects Version/s: 3.1.0

Spark is using an older version of Bootstrap (v. 2.3.2) for the Web UI pages.  Bootstrap 2.x was moved to EOL in Aug 2013 and Bootstrap 3.x was moved to EOL in July 2019 ([https://github.com/twbs/release)].  Older versions of Bootstrap are also getting flagged in security scans for various CVEs:
 * [https://snyk.io/vuln/SNYK-JS-BOOTSTRAP-72889]
 * [https://snyk.io/vuln/SNYK-JS-BOOTSTRAP-173700]
 * [https://snyk.io/vuln/npm:bootstrap:20180529]
 * [https://snyk.io/vuln/npm:bootstrap:20160627]

I haven't validated each CVE, but it would probably be good practice to resolve any potential issues and get on a supported release.

The bad news is that there have been quite a few changes between Bootstrap 2 and Bootstrap 4.  I've tried updating the library, refactoring/tweaking the CSS and JS to maintain a similar appearance and functionality, and testing the documentation.  As with the ticket created for the outdated Bootstrap version in the docs (SPARK-30654), this is a fairly large change so I'm sure additional testing and fixes will be needed.
Bug: Update Docs Bootstrap to 4.4.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jan/20 20:00
Affects Version/s: 3.1.0

We are using an older version of Bootstrap (v. 2.1.0) for the online documentation site.  Bootstrap 2.x was moved to EOL in Aug 2013 and Bootstrap 3.x was moved to EOL in July 2019 ([https://github.com/twbs/release)].  Older versions of Bootstrap are also getting flagged in security scans for various CVEs:
 * [https://snyk.io/vuln/SNYK-JS-BOOTSTRAP-72889]
 * [https://snyk.io/vuln/SNYK-JS-BOOTSTRAP-173700]
 * [https://snyk.io/vuln/npm:bootstrap:20180529]
 * [https://snyk.io/vuln/npm:bootstrap:20160627]

I haven't validated each CVE, but it would probably be good practice to resolve any potential issues and get on a supported release.

The bad news is that there have been quite a few changes between Bootstrap 2 and Bootstrap 4.  I've tried updating the library, refactoring/tweaking the CSS and JS to maintain a similar appearance and functionality, and testing the documentation.  This is a fairly large change so I'm sure additional testing and fixes will be needed.
Bug: TableCalalog reserved properties shoudn't be changed with options or tblpropeties
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jan/20 02:36
Affects Version/s: 3.0.0

Instead of using `OPTIONS (k='v')` or TBLPROPERTIES  (k='v'), if k is a reserved TableCatalog property, we should use its specific syntax to add/modify/delete it.
e.g. provider is a reserved property, we should use the USING clause to specify it, and should not allow ALTER TABLE ... UNSET TBLPROPERTIES('provider') to delete it.
Bug: Make interval type more future-proof
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/20 08:12
Affects Version/s: 3.0.0

We've decided to not follow the SQL standard to define the interval type in 3.0. We should try our best to hide intervals from data sources/external catalogs as much as possible, to not leak internals to external systems.

In Spark 2.4, intervals are exposed in the following places:
1. The `CalendarIntervalType` is public
2. `Colum.cast` accepts `CalendarIntervalType` and can cast string to interval.
3. `DataFrame.collect` can return `CalendarInterval` objects.
4. UDF can tale `CalendarInterval` as input.
5. data sources can return IntervalRow directly which may contain `CalendarInterval`.

In Spark 3.0, we don't want to break Spark 2.4 applications, but we should not expose intervals wider than 2.4. In general, we should avoid leaking intervals to DS v2 and catalog plugins. We should also revert some PostgresSQL specific interval features.
Bug: Structured Streaming _spark_metadata fills up Spark Driver memory when having lots of objects
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jan/20 14:57
Affects Version/s: 2.4.3

Hi,

With the current implementation of the Spark Structured Streaming it does not seem to be possible to have a constantly running stream, writing millions of files, without increasing the spark driver's memory to dozens of GB's.

In our scenario we are using Spark structured streaming to consume messages from a Kafka cluster, transform them, and write them as compressed Parquet files in an S3 Objectstore Service.
Each 30 seconds a new batch of the spark-streaming is writing hundreds of objects, which respectively results within time to millions of objects in S3.
As all written objects are recorded in the _spark_metadata, the size of the compact files there grows to GB's that eventually fill up the Spark Driver's memory and lead to OOM errors.

We need the functionality to configure the spark structured streaming to run without loading all the historically accumulated metadata in its memory. 
Regularly resetting the _spark_metadata and the checkpoint folders is not an option in our use-case, as we are using the information from the _spark_metadata to have a register of the objects for faster querying and search of the written objects.
Bug: Introduce TTL config option for SQL Metadata Cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jan/20 06:18
Affects Version/s: 3.1.0

From [documentation|https://spark.apache.org/docs/2.4.4/sql-data-sources-parquet.html#metadata-refreshing]:
{quote}Spark SQL caches Parquet metadata for better performance. When Hive metastore Parquet table conversion is enabled, metadata of those converted tables are also cached. If these tables are updated by Hive or other external tools, you need to refresh them manually to ensure consistent metadata.
{quote}
Currently Spark [caches file listing for tables|https://spark.apache.org/docs/2.4.4/sql-data-sources-parquet.html#metadata-refreshing] and requires issuing "{{REFRESH TABLE"}} any time the file listing has changed outside of Spark. Unfortunately, simply submitting "{{REFRESH TABLE"}} commands could be very cumbersome. Assuming frequently added files, hundreds of tables and dozens of users querying the data (and expecting up-to-date results), manually refreshing metadata for each table is not a solution.

This is a pretty common use-case for streaming ingestion of data, which can be done outside of Spark (with tools like Kafka Connect, etc.).

A similar feature exists in Presto: {{hive.file-status-cache-expire-time}} can be found [here|https://prestosql.io/docs/current/connector/hive.html#hive-configuration-properties].

I propose to introduce a new option in Spark (something like "spark.sql.hive.filesourcePartitionFileCacheTTL") that controls the TTL of this metadata cache. It can be disabled by default (-1), so it doesn't change the existing behaviour.
Bug: Deprecate UserDefinedAggregateFunction
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/20 17:34
Affects Version/s: 3.0.0

Anticipating the merging of SPARK-27296, the legacy methodology for implementing custom user defined aggregators over untyped DataFrame based on UserDefinedAggregateFunction will be made obsolete. This class should be annotated as deprecated once the new capability is officially merged.
Bug: Support simple all gather in barrier task context
Status: Resolved | Priority: Major | Resolution: Done
Created: 28/Jan/20 19:30
Affects Version/s: 3.0.0

Currently we offer task.barrier() to coordinate tasks in barrier mode. Tasks can see all IP addresses from BarrierTaskContext. It would be simpler to integrate with distributed frameworks like TensorFlow DistributionStrategy if we provide all gather that can let tasks share additional information with others, e.g., an available port.

Note that with all gather, tasks are share their IP addresses as well.

{code}
port = ... # get an available port
ports = context.all_gather(port) # get all available ports, ordered by task ID
...  # set up distributed training service
{code}
Bug: Allow custom resource scheduling to work with YARN versions that don't support custom resource scheduling
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jan/20 00:27
Affects Version/s: 3.0.0

Many people/companies will not be moving to Hadoop 3.1 or greater, where it supports custom resource scheduling for things like GPUs soon and have requested support for it in older hadoop 2.x versions. This also means that they may not have isolation enabled which is what the default behavior relies on.

right now the option is to write a custom discovery script to handle on their own. This is ok but has some limitation because the script runs as a separate process.  It also just a shell script.

I think we can make this a lot more flexible by making the entire resource discovery class pluggable. The default one would stay as is and call the discovery script, but if an advanced user wanted to replace the entire thing they could implement a pluggable class which they could write custom code on how to discovery resource addresses.

This will also help users if they are running hadoop 3.1.x or greater but don't have the resources configured or aren't running in an isolated environment.
Bug: Spark external shuffle allow disable of separate event loop group
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jan/20 16:12
Affects Version/s: 3.0.0

In SPARK-24355 changes were made to add a separate event loop group for processing ChunkFetchRequests , this  allow fors the other threads to handle regular connection requests when the configuration value is set. This however seems to have added some latency (see pr 22173 comments at the end).

To help with this we could make sure the secondary event loop group isn't used when the configuration of spark.shuffle.server.chunkFetchHandlerThreadsPercent isn't explicitly set. This should result in getting the same behavior as before.
Bug: Use a dedicated boss event group loop in the netty pipeline for external shuffle service
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jan/20 19:54
Affects Version/s: 3.0.0

We have been seeing a large number of SASL authentication (RPC requests) timing out with the external shuffle service.
 The issue and all the analysis we did is described here:
 [https://github.com/netty/netty/issues/9890]

I added a {{LoggingHandler}} to netty pipeline and realized that even the channel registration is delayed by 30 seconds. 
 In the Spark External Shuffle service, the boss event group and the worker event group are same which is causing this delay.
{code:java}
    EventLoopGroup bossGroup =
      NettyUtils.createEventLoop(ioMode, conf.serverThreads(), conf.getModuleName() + "-server");
    EventLoopGroup workerGroup = bossGroup;

    bootstrap = new ServerBootstrap()
      .group(bossGroup, workerGroup)
      .channel(NettyUtils.getServerChannelClass(ioMode))
      .option(ChannelOption.ALLOCATOR, allocator)
      .childOption(ChannelOption.ALLOCATOR, allocator);
{code}
When the load at the shuffle service increases, since the worker threads are busy with existing channels, registering new channels gets delayed.

The fix is simple. I created a dedicated boss thread event loop group with 1 thread.
{code:java}
    EventLoopGroup bossGroup = NettyUtils.createEventLoop(ioMode, 1,
      conf.getModuleName() + "-boss");
    EventLoopGroup workerGroup =  NettyUtils.createEventLoop(ioMode, conf.serverThreads(),
    conf.getModuleName() + "-server");

    bootstrap = new ServerBootstrap()
      .group(bossGroup, workerGroup)
      .channel(NettyUtils.getServerChannelClass(ioMode))
      .option(ChannelOption.ALLOCATOR, allocator)
{code}
This fixed the issue.
 We just need 1 thread in the boss group because there is only a single server bootstrap.
Bug: add ENV_PYSPARK_MAJOR_PYTHON_VERSION support for JavaMainAppResource
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Jan/20 01:40
Affects Version/s: 3.1.0

In apache Livy, the program is first started with JavaMainAppResource, and then start the Python worker. At this time, the program needs to be able to pass Python environment variables.

In spark on yarn, we support it through spark.yarn.isPython. In k8s, we can support this in a better way.
Bug: Unable to create local temp dir on spark on k8s mode, with defaults.
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 21/Jan/20 08:45
Affects Version/s: 3.0.0

Unless we configure the property,  {code} spark.local.dir /tmp {code} following error occurs:

{noformat}
*20/01/21 08:33:17 INFO SparkEnv: Registering BlockManagerMasterHeartbeat*

*20/01/21 08:33:17 ERROR DiskBlockManager: Failed to create local dir in /var/data/spark-284c6844-8969-4288-9a6b-b72679c5b8e4. Ignoring this directory.*

*java.io.IOException: Failed to create a temp directory (under /var/data/spark-284c6844-8969-4288-9a6b-b72679c5b8e4) after 10 attempts!*

*at org.apache.spark.util.Utils$.createDirectory(Utils.scala:304)*

*at org.apache.spark.storage.DiskBlockManager.$anonfun$createLocalDirs$1(DiskBlockManager.scala:164)*

*at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)*

*at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)*
{noformat}

I have not yet fully understood the root cause, will post my findings once it is clear.
Bug: Pyspark on kubernetes does not support --py-files from remote storage in cluster mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/20 03:48
Affects Version/s: 2.4.4

The following spark-submit on yarn works fine downloading the file from remote storage and putting it into PYTHONPATH,

spark-submit --master yarn --deploy-mode cluster --py-files s3://bucket/packages.zip s3://bucket/etl.py

 

While, the same fails on k8s with import errors for packages in the zip file. The following is set to PYTHONPATH on k8s, which has a link to s3 file which can't be supported by PYTHONPATH
{noformat}
PYTHONPATH='/opt/spark/python/lib/pyspark.zip:/opt/spark/python/lib/py4j-*.zip:s3://bucket/packages.zip'{noformat}