av9ash/GitBugs huggingface dataset:
Bug: Flaky Test: CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/20 13:43
Affects Version/s: 3.1.0

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120302/testReport/

{code}
sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: 2 did not equal 3
	at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530)
	at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529)
	at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
	at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:503)
	at org.apache.spark.scheduler.CoarseGrainedSchedulerBackendSuite.$anonfun$new$11(CoarseGrainedSchedulerBackendSuite.scala:186)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:151)
	at org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
	at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:286)
	at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
	at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
{code}
Bug: Unable to use vendor specific datatypes with JDBC (MSSQL)
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 04/Mar/20 15:30
Affects Version/s: 2.4.5

I'm trying to create a table in MSSQL with a time(7) type.

For this I'm using the createTableColumnTypes option like "CallStartTime time(7)", with driver "{color:#212121}com.microsoft.sqlserver.jdbc.SQLServerDriver"{color}

I'm getting an error:  {color:#212121}org.apache.spark.sql.catalyst.parser.ParseException: DataType time(7) is not supported.(line 1, pos 43){color}

{color:#212121}What is then the point of using this option?{color}
Bug: Avro serializer should not fail when a nullable Spark field is written to a non-null Avro column
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 06/Mar/20 16:24
Affects Version/s: 2.4.4

Spark StructType schema are strongly biased towards having _nullable_ fields. In fact, this is what _Encoders.bean()_ does - any non-primitive field is automatically _nullable_. When we attempt to serialize dataframes into *user-supplied* Avro schemas where such corresponding fields are marked as _non-null_ (i.e., they are not of _union_ type) any such attempt will fail with the following exception

 
{code:java}
Caused by: org.apache.avro.AvroRuntimeException: Not a union: "string"
	at org.apache.avro.Schema.getTypes(Schema.java:299)
	at org.apache.spark.sql.avro.AvroSerializer.org$apache$spark$sql$avro$AvroSerializer$$resolveNullableType(AvroSerializer.scala:229)
	at org.apache.spark.sql.avro.AvroSerializer$$anonfun$3.apply(AvroSerializer.scala:209)
 {code}
This seems as rather draconian. We certainly should be able to write a field of the same type and with the same name if it is not a null into a non-nullable Avro column. In fact, the problem is so *severe* that it is not clear what should be done in such situations when Avro schema is given to you as part of API communication contract (i.e., it is non-changeable).

This is an important issue.
Bug: Error in writing a pyspark streaming dataframe created from Kafka source to a csv file 
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 04/Mar/20 18:48
Affects Version/s: 2.4.5

While writing a streaming dataframe created from Kafka source to a csv file gives following error in PySpark.

NOTE : The same streaming dataframe is getting displayed in the console.

sdf.writeStream.format("console").start().awaitTermination()  // Working

sdf.writeStream\
    .format("csv")\
    .option("path", "C://output")\
    .option("checkpointLocation", "C://Checkpoint")\
    .outputMode("append")\
    .start().awaitTermination()    // Not working


Error
---------
 *File "C:\Spark\python\pyspark\sql\utils.py", line 63, in deco
    return f(*a, **kw)
  File "C:\Spark\python\lib\py4j-0.10.7-src.zip\py4j\protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o63.awaitTermination.
: org.apache.spark.sql.streaming.StreamingQueryException: Expected e.g. {"topicA":{"0":23,"1":-1},"topicB":{"0":-2}}, got {"logOffset":1}
=== Streaming Query ===
Identifier: [id = 6718625c-489e-44c8-b273-0da3429e97a8, runId = b64887ba-ca32-499e-9ab5-f839fd44ec26]
Current Committed Offsets: {KafkaV2[Subscribe[test1]]: {"logOffset":1}}
Current Available Offsets: {KafkaV2[Subscribe[test1]]: {"logOffset":1}}

Current State: ACTIVE
Thread State: RUNNABLE*
Bug: approxQuantile function  of spark , not taking List<String> as first parameter
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 05/Mar/20 17:58
Affects Version/s: 2.4.1

0
I am using spark-sql-2.4.1v in my project with java8.

I need to calculate the quantiles on the some of the (calculated) columns (i.e. con_dist_1 , con_dist_2 ) of below given dataframe df.
{{List<String> calcColmns = Arrays.asList("con_dist_1","con_dist_2")}}
When I am trying to use first version of approxQuantile i.e. approxQuantile(List, List, double) as below

Dataset<Row> df = //dataset
{{List<List<Double>> quants = df.stat().approxQuantile(calcColmns , Array(0.0,0.1,0.5),0.0);}}
*It is giving error :*
{quote}The method approxQuantile(String, double[], double) in the type DataFrameStatFunctions is not applicable for the arguments (List, List, double)
{quote}
so what is wrong here , I am doing it in my eclipseIDE. Why it is not invoking List even though i am passing List ??

Really appreciate any help on this.

more details are added here

[https://stackoverflow.com/questions/60550152/issue-with-approxquantile-of-spark-not-recognizing-liststring]
Bug: Spark 2.4.* SQL query with partition columns scans entire AVRO data 
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 06/Mar/20 07:18
Affects Version/s: 2.4.0

Partition Column: dt

SQL Query: select distinct dt from table1 

Table format: AVRO

It is scanning entire avro data in a table to get the distinct dt values
Bug: alter hive column datatype is not supported
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 05/Mar/20 06:38
Affects Version/s: 2.2.2

describe tb2;
Getting log thread is interrupted, since query is done!
+-----------------+-----------------+-----------------+--+
| col_name | data_type | comment |
+-----------------+-----------------+-----------------+--+
| fn | int | NULL |
| ln | string | NULL |
| age | int | NULL |
| # Partition Information | | |
| # col_name | data_type | comment |
| age | int | NULL |
+-----------------+-----------------+-----------------+--+
6 rows selected (0.213 seconds)
 alter table tb2 change fn fn bigint;
Getting log thread is interrupted, since query is done!
Error: org.apache.spark.sql.AnalysisException: ALTER TABLE CHANGE COLUMN is not supported for changing column 'fn' with type 'IntegerType' to 'fn' with type 'LongType'; (state=,code=0)
java.sql.SQLException: org.apache.spark.sql.AnalysisException: ALTER TABLE CHANGE COLUMN is not supported for changing column 'fn' with type 'IntegerType' to 'fn' with type 'LongType';
 at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:296)
 at org.apache.hive.beeline.Commands.execute(Commands.java:848)
 at org.apache.hive.beeline.Commands.sql(Commands.java:713)
 at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
 at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
 at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
 at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
 at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
Bug: Parameter cannot be passed to pandas udf of type map_iter
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 10/Mar/20 10:30
Affects Version/s: 3.0.0

Parameters can only be passed in the following way:

********************************************************************

from pyspark.sql.functions import pandas_udf, PandasUDFType

def map_iter_pandas_udf_example(spark):
    strr = "abcd
    df = spark.createDataFrame([(1, 21),(2,30)],("id", "age")) 

    @pandas_udf(df.schema, PandasUDFType.MAP_ITER)
    def filter_func(batch_iter, x = strr):
        print(x)
        for pdf in batch_iter:
            yield pdf[pdf.id == 1]

    df.mapInPandas(filter_func).show()

*******************************************************************

 

However, if the code edited as follow, error ccurred:

*******************************************************************

from pyspark.sql.functions import pandas_udf, PandasUDFType

def map_iter_pandas_udf_example(spark):
    strr = "abcd
    df = spark.createDataFrame([(1, 21),(2,30)],("id", "age")) 

    @pandas_udf(df.schema, PandasUDFType.MAP_ITER)
    def filter_func(batch_iter, x = strr):
        print(x)
        for pdf in batch_iter:
            yield pdf[pdf.id == 1]

    data = "dbca"

    df.mapInPandas(filter_func(data)).show()

*******************************************************************

ValueError: Invalid udf: the udf argument must be a pandas_udf of type MAP_ITER.

Does anyone know if pandas udf of type map_iter can pass parameters, and if so, how to write the code? Thanks.
Bug: Use multiple external catalogs to speed up metastore access
Status: Open | Priority: Major | Resolution: 
Created: 11/Mar/20 03:18
Affects Version/s: 3.1.0

Now, we use HiveClientImpl to access hive metastore. However,  a long running rpc in hive will block all of the query. Currently, we use the member of externalCatalog in ShardState to access. It's singleton.

Maybe, we can use multiple extrenal catalog instance to speed up metastore access in read-only situation.

Original:

Query 1:

DatabaseExist -> getTable -> getPartiton  (6s)

Query 2:

DatabaseExist -> getTable -> getPartiton   (5s)

Total cost: 11s

Now:

Query 1:

DatabaseExist -> getTable -> getPartiton  (6s)

Query 2:

DatabaseExist -> getTable -> getPartiton   (5s)

Total cost: 6s
Bug: spark on k8s Exception "Database xxx not found" when hive MetaStoreClient lost connection
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 09/Mar/20 03:53
Affects Version/s: 2.4.4

org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke:184 | MetaStoreClient lost connection. Attempting to reconnect.org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
06-03-2020 12:56:09 CST stage1 INFO - Caused by: java.net.SocketException: Connection reset
06-03-2020 12:56:09 CST stage1 INFO - 
06-03-2020 12:56:09 CST stage1 INFO - 	at java.net.SocketInputStream.read(SocketInputStream.java:210)
06-03-2020 12:56:09 CST stage1 INFO - 
06-03-2020 12:56:09 CST stage1 INFO - 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
06-03-2020 12:56:09 CST stage1 INFO - 
06-03-2020 12:56:09 CST stage1 INFO - 	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
06-03-2020 12:56:09 CST stage1 INFO - 
06-03-2020 12:56:09 CST stage1 INFO - 	at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
06-03-2020 12:56:09 CST stage1 INFO - 
06-03-2020 12:56:09 CST stage1 INFO - 	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
06-03-2020 12:56:09 CST stage1 INFO - 
06-03-2020 12:56:09 CST stage1 INFO - 	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
06-03-2020 12:56:09 CST stage1 INFO - 
06-03-2020 12:56:09 CST stage1 INFO - 	... 70 more

06-03-2020 12:56:09 CST stage1 INFO - 2020-03-06 04:56:06 [INFO] -- org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open:376 | Trying to connect to metastore with URI thrift://hive-metastore-server:9083
06-03-2020 12:56:09 CST stage1 INFO - 
06-03-2020 12:56:09 CST stage1 INFO - 2020-03-06 04:56:06 [INFO] -- org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open:472 | Connected to metastore.
06-03-2020 12:56:09 CST stage1 INFO - 
06-03-2020 12:56:09 CST stage1 INFO - 2020-03-06 04:56:14 [WARN] -- org.apache.spark.internal.Logging$class.logWarning:87 | Kubernetes client has been closed (this is expected if the application is shutting down.)
06-03-2020 12:56:09 CST stage1 INFO - 2020-03-06 04:56:14 [WARN] -- org.apache.spark.internal.Logging$class.logWarning:87 | Kubernetes client has been closed (this is expected if the application is shutting down.)
06-03-2020 12:56:09 CST stage1 INFO - 
06-03-2020 12:56:09 CST stage1 INFO - Exception in thread "main" org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'xxx' not found;
06-03-2020 12:56:09 CST stage1 INFO - 
06-03-2020 12:56:09 CST stage1 INFO - 	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.org$apache$spark$sql$catalyst$catalog$SessionCatalog$$requireDbExists(SessionCatalog.scala:178)
Bug: Flaky Test: ExecutorAllocationManagerSuite.interleaving add and remove
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/20 13:42
Affects Version/s: 3.1.0

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120300/testReport/

{code}
sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: 12 did not equal 8
	at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530)
	at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529)
	at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
	at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:503)
	at org.apache.spark.ExecutorAllocationManagerSuite.$anonfun$new$51(ExecutorAllocationManagerSuite.scala:864)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:151)
	at org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
{code}
Bug: Expose the ability for user to cleanup shuffle files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/20 18:51
Affects Version/s: 3.0.0

Dynamic scaling on Kubernetes (introduced in Spark 3) depends on only shutting down executors without shuffle files. However Spark does not aggressively clean up shuffle files (see SPARK-5836) and instead depends on JVM GC on the driver to trigger deletes. We already have a mechanism to explicitly clean up shuffle files from the ALS algorithm where we create a lot of quickly orphaned shuffle files. We should expose this as an advanced developer feature to enable people to better clean-up shuffle files improving dynamic scaling of their jobs on Kubernetes.
Bug: KMeans optimization based on triangle-inequality
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/20 10:01
Affects Version/s: 3.1.0

In current impl, following Lemma is used in KMeans:

0, Let x be a point, let b be a center and o be the origin, then d(x,c) >= |(d(x,o) - d(c,o))| = |norm(x)-norm(c)|

this can be applied in {{EuclideanDistance}}, but not in {{CosineDistance}}

According to [Using the Triangle Inequality to Accelerate K-Means|[https://www.aaai.org/Papers/ICML/2003/ICML03-022.pdf]], we can go futher, and there are another two Lemmas can be used:

1, Let x be a point, and let b and c be centers. If d(b,c)>=2d(x,b) then d(x,c) >= d(x,b);

this can be applied in {{EuclideanDistance}}, but not in {{CosineDistance}}.

However, luckily for CosineDistance we can get a variant in the space of radian/angle.

2, Let x be a point, and let b and c be centers. Then d(x,c) >= max\{0, d(x,b)-d(b,c)};

this can be applied in {{EuclideanDistance}}, but not in {{CosineDistance}}

The application of Lemma 2 is a little complex: It need to cache/update the distance/lower bounds to previous centers, and thus can be only applied in training, not usable in prediction.

So this ticket is mainly for Lemma 1. Its idea is quite simple, if point x is close to center b enough (less than a pre-computed radius), then we can say point x belong to center c without computing the distances between x and other centers. It can be used in both training and predction.
Bug: Revert SPARK-29285 to fix shuffle regression caused by creating temporary file eagerly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Mar/20 09:09
Affects Version/s: 3.0.0

In SPARK-29285, we change to create shuffle temporary eagerly. This is helpful for not to fail the entire task in the scenario of occasional disk failure.

But for the applications that many tasks don't actually create shuffle files, it caused overhead. See the below benchmark:

Env: Spark local-cluster[2, 4, 19968], each queries run 5 round, each round 5 times.
Data: TPC-DS scale=99 generate by spark-tpcds-datagen

Results:
|| ||Base||Revert||
|Q20|Vector(4.096865667, 2.76231748, 2.722007606, 2.514433591, 2.400373579) Median 2.722007606|Vector(3.763185446, 2.586498463, 2.593472842, 2.320522846, 2.224627274) Median 2.586498463|
|Q33|Vector(5.872176321, 4.854397586, 4.568787136, 4.393378146, 4.423996818) Median 4.568787136|Vector(5.38746785, 4.361236877, 4.082311276, 3.867206824, 3.783188024) Median 4.082311276|
|Q52|Vector(3.978870321, 3.225437871, 3.282411608, 2.869674887, 2.644490664) Median 3.225437871|Vector(4.000381522, 3.196025108, 3.248787619, 2.767444508, 2.606163423) Median 3.196025108|
|Q56|Vector(6.238045133, 4.820535173, 4.609965579, 4.313509894, 4.221256227) Median 4.609965579|Vector(6.241611339, 4.225592467, 4.195202502, 3.757085755, 3.657525982) Median 4.195202502|
Bug: validate CTAS table path in SPARK-19724 seems conflict and External table also need to check non-empty
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 29/Mar/20 13:56
Affects Version/s: 3.0.0

In  SessionCatalog.validateTableLocation()
{code:java}
val tableLocation =
  new Path(table.storage.locationUri.getOrElse(defaultTablePath(table.identifier)))

{code}
But in CreateDataSourceTableAsSelect , table location use defaultTablePath
{code:java}
assert(table.schema.isEmpty)
sparkSession.sessionState.catalog.validateTableLocation(table)
val tableLocation = if (table.tableType == CatalogTableType.MANAGED) {
  Some(sessionState.catalog.defaultTablePath(table.identifier))
} else {
  table.storage.locationUri
}

{code}
Bug: Provide Configuration Parameter to select/enforce the Hive Hash for Bucketing
Status: Open | Priority: Major | Resolution: 
Created: 16/Mar/20 03:29
Affects Version/s: 3.1.0

I couldn't find a configuration parameter to choose Hive Hashing instead of Spark's default Murmur Hash when performing Spark BucketBy operation. According to the discussion with @[~maropu] [~hyukjin.kwon], suggested to open a new JIRA.
Bug: Support Hive on DataSourceV2
Status: Open | Priority: Major | Resolution: 
Created: 25/Mar/20 04:02
Affects Version/s: 3.1.0

There are 3 reasons why we need to support Hive on DataSourceV2.
1. Hive itself is one of Spark data sources.
2. HiveTable is essentially a FileTable with its own input and output
formats, it works fine with FileTable.
3. HiveTable should be stateless, and users can freely read or write Hive
using batch or microbatch.
Bug: Support for BEGIN/COMMIT/ROLLBACK TRANSACTION in SparkSQL
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 26/Mar/20 16:44
Affects Version/s: 2.4.5

Looking for support for atomic transactions: BEGIN/COMMIT/ROLLBACK.

 

Such as here:

BEGIN TRANSACTION: [https://docs.microsoft.com/en-us/sql/t-sql/language-elements/begin-transaction-transact-sql?view=sql-server-ver15]

COMMIT TRANSACTION: [https://docs.microsoft.com/en-us/sql/t-sql/language-elements/commit-transaction-transact-sql?view=sql-server-ver15]

ROLLBACK TRANSACTION: [https://docs.microsoft.com/en-us/sql/t-sql/language-elements/rollback-transaction-transact-sql?view=sql-server-ver15]
Bug: Structured streaming The offset in the checkpoint is suddenly reset to the earliest
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 30/Mar/20 10:44
Affects Version/s: 2.4.0

Spark structured Streaming -Kafka, ListOffsetRequest to earliest(-2) for existing partition:

Issue identified as spark has been sending ListOffsetRequest for existing partition as "earliest(-2)" instead of latest(-1). This has been occuring all of sudden after couple of hours application got started. 
As per the documentaion, spark should request as earliest for only new partitions but in this request with earliest is being send to existing particular partition. Due to this, offsets are being set to LSO(Stable offset) and duplicate data is being consumed.

Below are the logs at consumer in DEBUG mode. Duplicates have flown from partition-20 and at this point, below 2 lines are bit different from normal logs. Also no error message in logs.This behaviour looks peculiar .

29/03/20 18:27:53 DEBUG internals.Fetcher: [Consumer clientId=consumer-1, groupId=spark-kafka-source-bca38026-e65e-4e24-8d86-aaa-driver-0]
 Sending ListOffsetRequest (type=ListOffsetRequest, replicaId=-1, partitionTimestamps=\{Hid1328.gnpp-raw-20=-2}, isolationLevel=READ_UNCOMMITTED) to broker famescpolyfi5.teliacompany.net:9095 (id: 5 rack: null)

29/03/20 18:27:53 DEBUG internals.Fetcher: [Consumer clientId=consumer-1, groupId=spark-kafka-source-bca38026-e65e-4e24-8d86-9daf0d34b406--1757490130-driver-0] Sending ListOffsetRequest (type=ListOffsetRequest, replicaId=-1, partitionTimestamps=\{Hid1328.gnpp-raw-5=-1, Hid1328.gnpp-raw-20=-1, Hid1328.gnpp-raw-25=-1, Hid1328.gnpp-raw-10=-1, Hid1328.gnpp-raw-15=-1, Hid1328.gnpp-raw-0=-1}, isolationLevel=READ_UNCOMMITTED) to broker famescpolyfi5.teliacompany.net:9095 (id: 5 rack: null)
Bug: RDDs in GradientBoostedTress can be unpersisted earlier for saving memory
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 22/Mar/20 15:04
Affects Version/s: 2.4.5

In ml.tree.impl.GradientBoostedTrees.boost(), _predErrorCheckpointer_ and _input_ are unpersisted at last. Actually, they can be unpersisted before the if(validate) block, so it can save memory for the computation in if block.
Bug: GracefulShutdown does not work when application is terminated from RestSubmissionClient or YarnClient
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 25/Mar/20 08:57
Affects Version/s: 2.4.3

While starting the Spark Application "*spark.streaming.stopGracefullyOnShutdown*" is set to true

try to terminate the application programatically using JAVA API

1 using RestSubmissionClient client = new RestSubmissionClient(masterUrl);
             SubmitRestProtocolResponse statusResponse = client.killSubmission(submissionId);  

 

2. using getYarnClient().killApplication(appId);

 

In both the cases Application dose not stop gracefully

 

But killing the Application using  

 

Kill -SIGTERM <pid>  will shutdown the application gracefully.


Expected : Application should have terminated gracefully in all cases when spark.streaming.stopGracefullyOnShutdown is set