av9ash/GitBugs huggingface dataset:
Bug: Drop R < 3.5 support
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jun/20 12:49
Affects Version/s: 3.0.0

Spark 3.0.0 is built by R 3.6.3 which does not support R < 3.5:

{code}
Error in readRDS(pfile) : cannot read workspace version 3 written by R 3.6.3; need R 3.5.0 or newer version.
{code}

In fact, with SPARK-31918, we will have to drop R < 3.5 entirely to support R 4.0.0.

This JIRA targets to drop R < 3.5 in SparkR.
Bug: Spark 3 UI task launch time show in error time zone
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jun/20 05:56
Affects Version/s: 3.0.0

For example,

In this link: history/app-20200623133209-0015/stages/ , stage submit time is correct (CST)

 

But in this link: history/app-20200623133209-0015/stages/stage/?id=0&attempt=0 , task launch time is incorrect(UTC)

 

The same problem exists in port 4040 Web UI.
Bug: Why spark does not preserve the original timestamp format while writing dataset to file or hdfs
Status: Open | Priority: Major | Resolution: 
Created: 17/Jun/20 18:29
Affects Version/s: 2.3.0

Want to write spark dataset having few timestamp columns into hdfs.
 * While reading, by default spark infers data as timestamp, if format is similar to "*yyyy-MM-dd HH:mm:ss*".
 * But while writing to file, saves in format as "*yyyy-MM-dd'T'HH:mm:ss.SSSXXX*"
 * For e.g. source data *2020-06-01 12:10:03* is written as *2020-06-01T12:10:03.000+05:30*.
 * Expected is to preserve the oroginal timestamp format before writing.

Why spark does not preserve the original timestamp format while writing dataset to file or hdfs?

Using simple java code like:

{color:#4c9aff}Dataset<Row> ds = spark.read().format("csv").option("path",the_path).option("inferSchema","true").load(); {color}
{color:#4c9aff}ds.write().format("csv").save("path_to_save");{color}

I know the workaround:
 * Use "*timestampFormat*" option before save.
 * But may have performance overhead and also its global for all columns.
 * So lets say have 2 columns having formats "*yyyy-MM-dd HH:mm:ss*" and "*yyyy-MM-dd HH*". Both can be inferred as timestamp by default, but outputs in a single specified "timestampFormat".
 * Another way is to use date_format(col, format). But that also may have performance overhead and includes operations to apply, whereas I expect spark to preserve the original format

Tried with spark2.3 and spark2.4
Bug: Can many executors share one gpu for spark3.0?
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 18/Jun/20 12:01
Affects Version/s: 3.0.0

hi, I want to run mang executors（for example, 2） in a server with only one GPU card，I tested spark3.0 in yarn cluster mode with the following config：

spark-shell  --conf spark.executor.resource.gpu.amount=0.5 

 

Then ，I find the following errors：

 

20/06/18 16:24:46 [main] {color:#FF0000}ERROR{color} SparkContext: Error initializing SparkContext.
java.lang.NumberFormatException:{color:#FF0000} For input string: "0.5"{color}
 at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
 at java.lang.Integer.parseInt(Integer.java:580)
 at java.lang.Integer.parseInt(Integer.java:615)
 at scala.collection.immutable.StringLike.toInt(StringLike.scala:304)
 at scala.collection.immutable.StringLike.toInt$(StringLike.scala:304)
 at scala.collection.immutable.StringOps.toInt(StringOps.scala:33)
 at org.apache.spark.resource.ResourceUtils$.parseResourceRequest(ResourceUtils.scala:142)
 at org.apache.spark.resource.ResourceUtils$.$anonfun$parseAllResourceRequests$1(ResourceUtils.scala:159)
 at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
 at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:75)
 at scala.collection.TraversableLike.map(TraversableLike.scala:238)
 at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
 at scala.collection.AbstractTraversable.map(Traversable.scala:108)
 at org.apache.spark.resource.ResourceUtils$.parseAllResourceRequests(ResourceUtils.scala:159)
 at org.apache.spark.SparkContext$.checkResourcesPerTask$1(SparkContext.scala:2773)
 at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2921)
 at org.apache.spark.SparkContext.<init>(SparkContext.scala:528)
 at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2555)
 at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$1(SparkSession.scala:931)
 at scala.Option.getOrElse(Option.scala:189)

 

。。。。

 

My question：for spark3.0 Can one gpu card support many executors ?

Thank you!
Bug: df.explain("mode") should work in PySpark side as well for consistency
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jun/20 01:01
Affects Version/s: 3.0.0

Scala:

{code}
scala> spark.range(10).explain("cost")
== Optimized Logical Plan ==
Range (0, 10, step=1, splits=Some(12)), Statistics(sizeInBytes=80.0 B)

== Physical Plan ==
*(1) Range (0, 10, step=1, splits=12)
{code}

PySpark:

{code}
>>> spark.range(10).explain("cost")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../spark/python/pyspark/sql/dataframe.py", line 333, in explain
    raise TypeError(err_msg)
TypeError: extended (optional) should be provided as bool, got <class 'str'>
{code}
Bug: Create Kerberos authentication provider API in JDBC connector
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jun/20 11:58
Affects Version/s: 3.1.0

Adding embedded provider to all the possible databases would generate high maintenance cost on Spark side.
Instead an API can be introduced which would allow to implement further providers independently.

One important requirement what I suggest is: JDBC connection providers must be loaded independently just like delegation token providers.
Bug: Migrate to NumPy documentation style
Status: Resolved | Priority: Major | Resolution: Done
Created: 24/Jun/20 08:43
Affects Version/s: 3.1.0

https://github.com/numpy/numpydoc

For example,

Before: https://github.com/apache/spark/blob/f0e6d0ec13d9cdadf341d1b976623345bcdb1028/python/pyspark/sql/dataframe.py#L276-L318
 After: https://github.com/databricks/koalas/blob/6711e9c0f50c79dd57eeedb530da6c4ea3298de2/databricks/koalas/frame.py#L1122-L1176

We can incrementally start to switch.

NOTE that this JIRA targets only to switch the style. It does not target to add additional information or fixes together.
Bug: Generate JSON files with rebasing switch points using smaller steps
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jun/20 18:17
Affects Version/s: 3.1.0

Currently, the JSON files 'gregorian-julian-rebase-micros.json' and 'julian-gregorian-rebase-micros.json' are generated w/ step of 1 week. The files can miss short spikes of diffs. The ticket aims to change the max step from 1 week to 30 minutes, and speed up execution.
Bug: Remove warnings about pin-thread modes and guide collectWithJobGroups for now
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jun/20 01:21
Affects Version/s: 3.0.0

Currently, it guides users to use pin-thread mode to use multiple threads. There's a thread leak issue in this mode (SPARK-32010).

We shouldn't promote this mode in Spark 3.0 for now.
Bug: Resolve the grouping column properly per the case sensitivity in grouped and cogrouped pandas UDFs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jun/20 17:11
Affects Version/s: 3.0.0


{code}
from pyspark.sql.functions import *

df = spark.createDataFrame([[1, 1]], ["column", "Score"])

@pandas_udf("column integer, Score float", PandasUDFType.GROUPED_MAP)
def my_pandas_udf(pdf):
    return pdf.assign(Score=0.5)

df.groupby('COLUMN').apply(my_pandas_udf).show()
{code}

{code}
pyspark.sql.utils.AnalysisException: Reference 'COLUMN' is ambiguous, could be: COLUMN, COLUMN.;
{code}

{code}
df1 = spark.createDataFrame([(1, 1)], ("column", "value"))
df2 = spark.createDataFrame([(1, 1)], ("column", "value"))

df1.groupby("COLUMN").cogroup(
    df2.groupby("COLUMN")
).applyInPandas(lambda r, l: r + l, df1.schema).show()
{code}

{code}
pyspark.sql.utils.AnalysisException: cannot resolve '`COLUMN`' given input columns: [COLUMN, COLUMN, value, value];;
'FlatMapCoGroupsInPandas ['COLUMN], ['COLUMN], <lambda>(column#9L, value#10L, column#13L, value#14L), [column#22L, value#23L]
:- Project [COLUMN#9L, column#9L, value#10L]
:  +- LogicalRDD [column#9L, value#10L], false
+- Project [COLUMN#13L, column#13L, value#14L]
   +- LogicalRDD [column#13L, value#14L], false
{code}
Bug: Nested Schema Pruning not Working in Window Functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jun/20 23:47
Affects Version/s: 3.1.0

Using tables and data structures in `SchemaPruningSuite.scala`

 
{code:java}
// code placeholder
case class FullName(first: String, middle: String, last: String)
case class Company(name: String, address: String)
case class Employer(id: Int, company: Company)
case class Contact(
  id: Int,
  name: FullName,
  address: String,
  pets: Int,
  friends: Array[FullName] = Array.empty,
  relatives: Map[String, FullName] = Map.empty,
  employer: Employer = null,
  relations: Map[FullName, String] = Map.empty)
case class Department(
  depId: Int,
  depName: String,
  contactId: Int,
  employer: Employer)
{code}
 

The query to run:
{code:java}
// code placeholder
select a.name.first from (select row_number() over (partition by address order by id desc) as __rank, contacts.* from contacts) a where a.name.first = 'A' AND a.__rank = 1
{code}
 

The current physical plan:
{code:java}
// code placeholder
== Physical Plan ==
*(3) Project [name#46.first AS first#74]
+- *(3) Filter (((isnotnull(name#46) AND isnotnull(__rank#71)) AND (name#46.first = A)) AND (__rank#71 = 1))
   +- Window [row_number() windowspecdefinition(address#47, id#45 DESC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS __rank#71], [address#47], [id#45 DESC NULLS LAST]
      +- *(2) Sort [address#47 ASC NULLS FIRST, id#45 DESC NULLS LAST], false, 0
         +- Exchange hashpartitioning(address#47, 5), true, [id=#52]
            +- *(1) Project [id#45, name#46, address#47]
               +- FileScan parquet [id#45,name#46,address#47,p#53] Batched: false, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/private/var/folders/_c/4r2j33dd14n9ldfc2xqyzs400000gn/T/spark-85d173af-42..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:int,name:struct<first:string,middle:string,last:string>,address:string>
{code}
 

The desired physical plan:

 
{code:java}
// code placeholder
== Physical Plan ==
*(3) Project [_gen_alias_77#77 AS first#74]
+- *(3) Filter (((isnotnull(_gen_alias_77#77) AND isnotnull(__rank#71)) AND (_gen_alias_77#77 = A)) AND (__rank#71 = 1))
   +- Window [row_number() windowspecdefinition(address#47, id#45 DESC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS __rank#71], [address#47], [id#45 DESC NULLS LAST]
      +- *(2) Sort [address#47 ASC NULLS FIRST, id#45 DESC NULLS LAST], false, 0
         +- Exchange hashpartitioning(address#47, 5), true, [id=#52]
            +- *(1) Project [id#45, name#46.first AS _gen_alias_77#77, address#47]
               +- FileScan parquet [id#45,name#46,address#47,p#53] Batched: false, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/private/var/folders/_c/4r2j33dd14n9ldfc2xqyzs400000gn/T/spark-c64e0b29-d9..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:int,name:struct<first:string>,address:string>
{code}
Bug: Spark 3.0 ERROR TThreadPoolServer
Status: Open | Priority: Major | Resolution: 
Created: 23/Jun/20 09:10
Affects Version/s: 3.0.0

I'm using the latest spark 3.0.0 with simple settings,  after loading files with beeline (bin/beeline -u jdbc:hive2://127.0.0.1:10000/ -f load.view), the message has appeared in Thrift's log.

Also, when query is executed (select * from load; ) at the end of execution in the Thrift's log there is also the same message.

Even when you connect with beeline (bin/beeline -u jdbc:hive2://127.0.0.1:10000/) and do "!quit" the same message is generated.



Queries with beeline are working fine, just there is the same message.

Error message:

ERROR TThreadPoolServer: Thrift error occurred during processing of message.
org.apache.thrift.transport.TTransportException
 at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
 at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
 at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:374)
 at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:451)
 at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:433)
 at org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:43)
 at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
 at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:425)
 at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321)
 at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225)
 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
 at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:53)
 at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

 

 

I've tried to run Spark master and slave version 3.0.0 and Thrift from Spark 2.4.5, I got the same message.

Config is simple and used for testing before production:

spark.driver.extraClassPath=/var/opt/engine/spark/mysql-connector-java-8.0.20.jar
spark.executor.extraClassPath=/var/opt/engine/spark/mysql-connector-java-8.0.20.jar

spark.driver.memory=30g
spark.driver.cores=12
spark.executor.cores=12
spark.executor.instances=1
spark.executor.memory=30g
spark.network.timeout=1800
spark.rpc.askTimeout=6000
spark.rpc.lookupTimeout=3000
spark.sql.hive.thriftServer.singleSession=true
spark.sql.warehouse.dir=/var/opt/engine/spark
spark.hadoop.hive.metastore.warehouse.dir=/var/opt/engine/spark
hive.metastore.warehouse.dir=/var/opt/engine/spark
user.dir=/var/opt/engine/spark
derby.system.home=/var/opt/engine/spark

 

SPARK_WORKER_MEMORY=32g
SPARK_WORKER_CORES=12
SPARK_WORKER_INSTANCES=1
SPARK_LOCAL_IP=10.10.10.10
Bug: Using S3A staging committer, pending uploads are committed more than once and listed incorrectly in _SUCCESS data
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jun/20 08:25
Affects Version/s: 2.4.4

First of all thanks for the great work on the S3 committers. I was able set up the directory staging committer in my environment following docs at [https://github.com/apache/spark/blob/master/docs/cloud-integration.md#committing-work-into-cloud-storage-safely-and-fast] and tested one of my Spark applications using it. The Spark version is 2.4.4 with Hadoop 3.2.1 and the cloud committer bindings. The application writes multiple DataFrames to ORC/Parquet in S3, submitting them as write jobs to Spark in parallel.

I think I'm seeing a bug where the staging committer will complete pending uploads more than once. The main symptom how I discovered this is that the _SUCCESS data files under each table will contain overlapping file names that belong to separate tables. From my reading of the code, that's because the filenames in _SUCCESS reflect which multipart uploads were completed in the commit for that particular table.

An example:

Concurrently, fire off DataFrame.write.orc("s3a://bucket/a") and DataFrame.write.orc("s3a://bucket/b"). Suppose each table has one partition so writes one partition file.

When the two writes are done,
 * /a/_SUCCESS contains two filenames: /a/part-0000 and /b/part-0000.
 * /b/_SUCCESS contains the same two filenames.

Setting S3A logs to debug, I see the commitJob operation belonging to table a includes completing the uploads of /a/part-0000 and /b/part-0000. Then again, commitJob for table b includes the same completions. I haven't had a problem yet, but I wonder if having these extra requests would become an issue at higher scale, where dozens of commits with hundreds of files may be happening concurrently in the application.

I believe this may be caused from the way the pendingSet files are stored in the staging directory. They are stored under one directory named by the jobID, in the Hadoop code. However, for all write jobs executed by the Spark application, the jobID passed to Hadoop is the same - the application ID. Maybe the staging commit algorithm was built on the assumption that each instance of the algorithm would use a unique random jobID.

[~stevel@apache.org] , [~rdblue] Having seen your names on most of this work (thank you), I would be interested to know your thoughts on this. Also it's my first time opening a bug here, so let me know if there's anything else I can do to help report the issue.
Bug: Add ability to disable Sort,Spill in Partial aggregation 
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Jun/20 00:16
Affects Version/s: 3.0.0

In case of HashAggregation, a partial aggregation(update) is done followed by final aggregation(merge) 

During partial aggregation we sort and spill to disk everytime, when the fast Map(when enabled) and  UnsafeFixedWidthAggregationMap gets exhausted

*When the cardinality of grouping column is close to the total number of records being processed*, the sorting of data spilling to disk is not required, since it is kind of no-op and we can directly use in Final aggregation.

When the user is aware of nature of data, currently he has no control over disabling this sort, spill operation.

This is similar to following issue in Hive:

https://issues.apache.org/jira/browse/HIVE-223

https://issues.apache.org/jira/browse/HIVE-291
Bug: Spark Driver Supervise does not work reliably
Status: Open | Priority: Major | Resolution: 
Created: 16/Jun/20 19:40
Affects Version/s: 2.4.4

I have a standalone cluster setup. I DO NOT have a streaming use case. I use AWS EC2 machines to have spark master and worker processes.

*Problem*: If a spark worker machine running some drivers and executor dies, then the driver is not spawned again on other healthy machines.

*Below are my findings:*
||Action/Behaviour||Executor||Driver||
|Worker Machine Stop|Relaunches on an active machine|NO Relaunch|
|kill -9 to process|Relaunches on other machines|Relaunches on other machines|
|kill to process|Relaunches on other machines|Relaunches on other machines|

*Cluster Setup:*
 # I have a spark standalone cluster
 # {{spark.driver.supervise=true}}
 # Spark Master HA is enabled and is backed by zookeeper
 # Spark version = 2.4.4
 # I am using a systemd script for the spark worker process
Bug: Enable codegen for GenerateExec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jun/20 05:08
Affects Version/s: 3.0.0

Codegen had been disabled as part of 

[https://github.com/apache/spark/commit/b5c5bd98ea5e8dbfebcf86c5459bdf765f5ceb53]

There has been lots of improvements in the Generate code flow, after it had been disabled.(https://issues.apache.org/jira/browse/SPARK-21657)

Related discussion:

 [http://apache-spark-developers-list.1001551.n3.nabble.com/GenerateExec-CodegenSupport-and-supportCodegen-flag-off-td22984.html]

I tried enabling codegen and found query very perform with explode kind of operations.

Can we enable codegen by default for GenerateExec?
Bug: use MemoryUsage to control the size of block
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 12/Jun/20 07:19
Affects Version/s: 3.1.0

According to the performance test in https://issues.apache.org/jira/browse/SPARK-31783, the performance gain is mainly related to the nnz of block.

So it maybe reasonable to control the size of block by memory usage, instead of number of rows.

 

note1: param blockSize had already used in ALS and MLP to stack vectors (expected to be dense);

note2: we may refer to the {{Strategy.maxMemoryInMB}} in tree models;

 

There may be two ways to impl:

1, compute the sparsity of input vectors ahead of train (this can be computed with other statistics computation, maybe no extra pass), and infer a reasonable number of vectors to stack;

2, stack the input vectors adaptively, by monitoring the memory usage in a block;
Bug: potential regression if use memoryUsage instead of numRows
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 23/Jun/20 01:47
Affects Version/s: 3.1.0

1, if the `memoryUsage` is improperly set, for example, too small to store a instance;

2,  the blockify+GMM reuse two matrices whose shape is related to current blockSize:
{code:java}
@transient private lazy val auxiliaryProbMat = DenseMatrix.zeros(blockSize, k)
@transient private lazy val auxiliaryPDFMat = DenseMatrix.zeros(blockSize, numFeatures) {code}
When implementing blockify+GMM, I found that if I do not pre-allocate those matrices, there will be seriously regression (maybe 3~4 slower, I fogot the detailed numbers);

3, in MLP, three pre-allocated objects are also related to numRows:
{code:java}
if (ones == null || ones.length != delta.cols) ones = BDV.ones[Double](delta.cols)

// TODO: allocate outputs as one big array and then create BDMs from it
if (outputs == null || outputs(0).cols != currentBatchSize) {
...

// TODO: allocate deltas as one big array and then create BDMs from it
if (deltas == null || deltas(0).cols != currentBatchSize) {
  deltas = new Array[BDM[Double]](layerModels.length)
... {code}
I am not very familiar with the impl of MLP and failed to find some related document about this pro-allocation. But I guess there maybe regression if we disable this pro-allocation, since those objects look relatively big.
Bug: `ALTER TABLE multipartIdentifier RECOVER PARTITIONS` should drop partition if partition specific location is not exist any more
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/20 04:01
Affects Version/s: 3.1.0

Currently when execute
{code:sql}
ALTER TABLE multipartIdentifier RECOVER PARTITIONS
{code}
It will auto add the partition according to the table root location structure. 
And Spark need to add one more step to check if the existing partition specific location exists and then delete the partition？
Bug: ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jun/20 18:32
Affects Version/s: 3.0.0

ExecutorPlugin can't work with Standalone Cluster and Kubernetes
 when a jar which contains plugins and files used by the plugins are added by --jars and --files option with spark-submit.

This is because jars and files added by --jars and --files are not loaded on Executor initialization.
 I confirmed it works with YARN because jars/files are distributed as distributed cache.