av9ash/GitBugs huggingface dataset:
Bug: Adding support to set an initial GMM in spark.ml
Status: In Progress | Priority: Minor | Resolution: 
Created: 13/Nov/20 11:36
Affects Version/s: 3.0.1

Add support to set the initial model parameter instead of a random initialization for a Gaussian Mixture. This feature exists in mllib but was never implemented in ml.

 

Note: Even though issue SPARK-15785 claims to solve this issue, it doesn't. The current version of the GasussianMixture.scala contains the following TODO:
{code:java}
// TODO: SPARK-15785 Support users supplied initial GMM.{code}
see: https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala#L407
Bug: SparkUI / Executor Page doesn't work where under external reverse proxy
Status: In Progress | Priority: Minor | Resolution: 
Created: 10/Nov/20 17:07
Affects Version/s: 2.4.7

When using SparkUI under an external reverse proxy (NGinx, Træfik, ... ), you can configure SparkUI to be correctly displayed (html page and all static resources). But in the executors page, when calling the /api/, the URL don't use the Spark proxyBase configuration.

 

Here is an example :

I use Træfik to access to the SparkUI using the path : /sparkui (a proxy redirect is made when I call [http://acme.com/sparkui] to the SparkUI) and I use this Spark config : 
{code:java}
spark.ui.proxyBase /sparkui
spark.ui.proxyRedirectUri /{code}
using this config, when calling [http://acme.com/sparkui] I'll be redirected to [http://acme.com/sparkui/jobs/] and all is currently serve (HTML and all static files)

 

But I go on [http://acme.com/sparkui/executors/] a XHR call is made using this URL [http://acme.com/api/v1/applications] so it doesn't use the reverse proxy rule ... the correct URL should be : [http://acme.com/sparkui/api/v1/applications] 

 

I discover on the code (here for example [https://github.com/apache/spark/blob/branch-3.0/core/src/main/resources/org/apache/spark/ui/static/utils.js#L193)] that SparkUI use the location.origin (in my case [http://acme.com|http://acme.com%29./]). I think it should use the complete URL (under the last part like /executors/) instead or receive the proxyBase configuration to add it after the location.origin ...
Bug: Credentials are not passed in the `doFetchFile` when running spark-submit with https url
Status: In Progress | Priority: Minor | Resolution: 
Created: 11/Nov/20 15:28
Affects Version/s: 3.0.1

I'm running spark-submit https url containing username and password. It's said in the documentation - [https://spark.a|https://spark.apache.org/docs/latest/submitting-applications.html] [pache.org/docs/latest/submitting-applications.html|https://spark.apache.org/docs/latest/submitting-applications.html]
{quote}(Note that credentials for password-protected repositories can be supplied in some cases in the repository URI, such as in {{https://user:password@host/...}}. Be careful when supplying credentials this way.)
{quote}
However, when using that, I receive the following error:

 
{code:java}

INFO - 20/11/11 12:59:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
INFO - Exception in thread "main" java.io.IOException: Server returned HTTP response code: 401 for URL: https://username:*****@host.com/my_app/pipeline.jar
INFO - at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1924)
INFO - at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
INFO - at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
INFO - at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:729)
INFO - at org.apache.spark.deploy.DependencyUtils$.downloadFile(DependencyUtils.scala:138)
INFO - at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$8(SparkSubmit.scala:376)
INFO - at scala.Option.map(Option.scala:230)
INFO - at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:376)
INFO - at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
INFO - at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
INFO - at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
INFO - at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
INFO - at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
INFO - at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
INFO - at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
 

When downloading my file manually using wget, at first I receive a 401 error but then there's a retry with credentials:
{code:java}

HTTP request sent, awaiting response... 401 Unauthorized
Authentication selected: Basic realm="Restricted"
Reusing existing connection to host.com:443.
HTTP request sent, awaiting response... 200 OK

{code}
When I do use ` --auth-no-challenge` in wget the credentials are passed directly in the first request and I receive 200 OK. The problem with the first wget is that, it tries to download a file without passing credentials and after 401 it's challenged to pass credentials so it goes in two steps. That is similar to my issue where credentials are not passed in the first query.
Bug: Correct the behaviour of query filters in TPCDSQueryBenchmark 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Nov/20 03:25
Affects Version/s: 2.4.8

This PR intends to fix the behaviour of query filters in TPCDSQueryBenchmark. We can use an option --query-filter for selecting TPCDS queries to run, e.g., --query-filter q6,q8,q13. But, the current master has a weird behaviour about the option. For example, if we pass --query-filter q6 so as to run the TPCDS q6 only, TPCDSQueryBenchmark runs q6 and q6-v2.7 because the filterQueries method does not respect the name suffix. So, there is no way now to run the TPCDS q6 only.
Bug: Delete temporary file when cancelling writing to final path even underlying stream throwing error
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Nov/20 18:11
Affects Version/s: 3.1.0

In {{RenameBasedFSDataOutputStream.cancel}}, we do two things: closing underlying stream and delete temporary file, in a single try/catch block. Closing {{OutputStream}} could possibly throw {{IOException}} so we possibly missing deleting temporary file.
Bug: Fix procedure-like declaration compilation warning in Scala 2.13
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Nov/20 02:47
Affects Version/s: 3.1.0

Similar to spark-29291, just to track Spark 3.1.0.

There are two similar compilation warnings about procedure-like declaration in Scala 2.13.3:

 
{code:java}
[WARNING] [Warn] /spark/core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala:70: procedure syntax is deprecated for constructors: add `=`, as in method definition

[WARNING] [Warn] /spark/core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala:211: procedure syntax is deprecated: instead, add `: Unit =` to explicitly declare `run`'s return type
{code}
 

For constructors method definition should be `this(...) = \{ }` not `this(...) \{ }`, for without 

`return type` methods definition should be `def methodName(...): Unit = {}` not `def methodName(...) {}`
Bug: Move the table commands from ddl.scala to table.scala
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 06/Nov/20 05:06
Affects Version/s: 3.1.0

The table commands can be moved from `ddl.scala` to `table.scala` per the following:
https://github.com/apache/spark/blob/4941b7ae18d4081233953cc11328645d0b4cf208/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala#L137

This change will result in table-related commands in the same file `table.scala`.
Bug: [CORE] Add config spark.yarn.pyspark.archives
Status: In Progress | Priority: Minor | Resolution: 
Created: 06/Nov/20 19:44
Affects Version/s: 3.0.1

This config provides similar function as `spark.yarn.archive`, but instead passing location of pyspark.zip and py4j.zip. It has below benefits:

1. If I have different version of `spark.yarn.archive` from current machine, then there is no way to   provide matched pyspark version. This happens in version upgrade.
2. This can save effort of uploading pyspark.zip and py4j.zip every time.
Bug: In the threadDump page, when a thread is blocked by anther thread, the blocking thread name and url were both displayed incorrect, causing the url to fail to jump.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Nov/20 11:40
Affects Version/s: 3.0.1

In the threadDump page, when a thread is blocked by anther thread, the blocking thread name and url were both displayed incorrect, causing the url to fail to jump.

such as *Thread 73*  but shows `*Thread some(73)*`
Bug: foreachBatch sink outputs wrong metrics
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 05/Nov/20 13:58
Affects Version/s: 3.0.0

I created 2 similar jobs, 
1) First job reading from kafka and writing to console sink in append mode
2) Second job reading from kafka and writing to foreachBatch sink (which then writes in parquet format to S3).

The metrics in the log for console shows correct values for numInputRows and numOutputRows whereas they are wrong for foreachBatch.

With foreachBatch:
numInputRows is +1 more than what is actually present

numOutputRows is always -1.
///Console sink
//====================================20/11/05 13:36:21 INFO MicroBatchExecution: Streaming query made progress: {
  "id" : "775aa543-58bf-4cf7-b274-390da640b6ae",
  "runId" : "e5eac4ca-0b29-4ed4-be35-b70bd20906d5",
  "name" : null,
  "timestamp" : "2020-11-05T13:36:08.921Z",
  "batchId" : 0,
  "numInputRows" : 10,
  "processedRowsPerSecond" : 0.7759757895553658,
  "durationMs" : {
    "addBatch" : 7735,
    "getBatch" : 152,
    "latestOffset" : 2037,
    "queryPlanning" : 1010,
    "triggerExecution" : 12886,
    "walCommit" : 938
  },
  "stateOperators" : [ ],
  "sources" : [ {
    "description" : "KafkaV2[Subscribe[testedr7]]",
    "startOffset" : null,
    "endOffset" : {
      "testedr7" : {
        "0" : 10
      }
    },
    "numInputRows" : 10,
    "processedRowsPerSecond" : 0.7759757895553658
  } ],
  "sink" : {
    "description" : "org.apache.spark.sql.execution.streaming.ConsoleTable$@38c3a814",
    "numOutputRows" : 10
  }
}


///ForEachBatch Sink
//====================================20/11/05 13:43:38 INFO MicroBatchExecution: Streaming query made progress: {
  "id" : "789f9a00-2f2a-4f75-b643-fea201088b4a",
  "runId" : "b5e695c5-3a2e-4ad2-9dbf-11b69f368f61",
  "name" : null,
  "timestamp" : "2020-11-05T13:43:15.421Z",
  "batchId" : 0,
  "numInputRows" : 11,
  "processedRowsPerSecond" : 0.4833252779120348,
  "durationMs" : {
    "addBatch" : 17689,
    "getBatch" : 135,
    "latestOffset" : 2121,
    "queryPlanning" : 880,
    "triggerExecution" : 22758,
    "walCommit" : 876
  },
  "stateOperators" : [ ],
  "sources" : [ {
    "description" : "KafkaV2[Subscribe[testedr7]]",
    "startOffset" : null,
    "endOffset" : {
      "testedr7" : {
        "0" : 10
      }
    },
    "numInputRows" : 11,
    "processedRowsPerSecond" : 0.4833252779120348
  } ],
  "sink" : {
    "description" : "ForeachBatchSink",
    "numOutputRows" : -1
  }
}
Bug: Cache dependencies for Coursier with new sbt in GitHub Actions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Nov/20 05:02
Affects Version/s: 3.1.0

SPARK-33226 upgraded sbt to 1.4.1.
As of 1.3.0, sbt uses Coursier as the dependency resolver / fetcher.
So let's change the dependency cache configuration for the GitHub Actions job.
Bug: Ability to set Dynamodb table name while reading from Kinesis
Status: Open | Priority: Minor | Resolution: 
Created: 02/Nov/20 13:31
Affects Version/s: 3.1.0

* Need the ability to set dynamodb table while reading data from kinesis. The KCL library provides the ability to set dynamodb table name. example: [https://aws.amazon.com/premiumsupport/knowledge-center/kinesis-kcl-apps-dynamodb-table/] . We would like to have a similar interface to pass the dynamodb table.
Bug: Document spark.conf.isModifiable()
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 12/Nov/20 17:22
Affects Version/s: 3.1.0

PySpark's docs make no mention of {{conf.isModifiable()}}, though it appears to be a public method introduced in SPARK-24761.

http://spark.apache.org/docs/3.0.1/api/python/pyspark.sql.html#pyspark.sql.SparkSession.conf
Bug: Accessibility: in spark history server page, The <html> element does not have a lang attribute, user depending on Assistive Technology get affected as lang attribute.
Status: Open | Priority: Trivial | Resolution: 
Created: 25/Nov/20 08:25
Affects Version/s: 3.0.1

in spark history server page, The <html> elements do not have a lang attribute, user depending on Assistive Technology get affected as lang attribute.
Bug: Accessibility: "Spark 2.4.5" image link is not defined 
Status: Open | Priority: Trivial | Resolution: 
Created: 25/Nov/20 08:30
Affects Version/s: 2.4.5

In spark history server page, screen reader user will get confuse with the purpose of the image link of the "Spark 2.4.5" and can not access the image link efficiently if alt of the image is not defined.
Bug: RESET can clear StaticSQLConfs
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 24/Sep/20 22:13
Affects Version/s: 3.0.1

The RESET command can clear a sessions' static SQL configurations, when that static SQL configuration was set on a SparkSession that uses a pre-existing SparkContext. Here is repro:
{code:java}
// Blow away any pre-existing session thread locals
org.apache.spark.sql.SparkSession.clearDefaultSession()
org.apache.spark.sql.SparkSession.clearActiveSession()

// Create new session and explicitly set a spark context
val newSession = org.apache.spark.sql.SparkSession.builder
 .sparkContext(sc)
 .config("spark.sql.globalTempDatabase", "bob")
 .getOrCreate()
assert(newSession.conf.get("spark.sql.globalTempDatabase") == "bob")
newSession.sql("reset")
assert(newSession.conf.get("spark.sql.globalTempDatabase") == "bob") // Boom!
{code}
The problem is that RESET assumes it can use the SparkContext's configurations as its default.
Bug: scheduler with blacklisting doesn't appear to pick up new executor added
Status: Open | Priority: Critical | Resolution: 
Created: 29/Sep/20 19:55
Affects Version/s: 3.0.0

I was running a test with blacklisting  standalone mode and all the executors were initially blacklisted.  Then one of the executors died and we got allocated another one. The scheduler did not appear to pick up the new one and try to schedule on it though.

You can reproduce this by starting a master and slave on a single node, then launch a shell like where you will get multiple executors (in this case I got 3)

$SPARK_HOME/bin/spark-shell --master spark://yourhost:7077 --executor-cores 4 --conf spark.blacklist.enabled=true

From shell run:
{code:java}
import org.apache.spark.TaskContext
val rdd = sc.makeRDD(1 to 1000, 5).mapPartitions { it =>
 val context = TaskContext.get()
 if (context.attemptNumber() < 2) {
 throw new Exception("test attempt num")
 }
 it
}
rdd.collect(){code}
 

Note that I tried both with and without dynamic allocation enabled.

 

You can see screen shot related on https://issues.apache.org/jira/browse/SPARK-33029
Bug: Left semi stream-stream join
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Sep/20 07:45
Affects Version/s: 3.1.0

Current stream-stream join supports inner, left outer and right outer join ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala#L166] ). We do see internally a lot of users are using left semi stream-stream join (not spark structured streaming), e.g. I want to get the ad impression (join left side) which has click (joint right side), but I don't care how many clicks per ad (left semi semantics).

 

Left semi stream-stream join will work as followed:

(1).for left side input row, check if there's a match on right side state store

  (1.1). if there's a match, output the left side row.

  (1.2). if there's no match, put the row in left side state store (with "matched" field to set to false in state store).

(2).for right side input row, check if there's a match on left side state store. If there's a match, update left side row state with "matched" field to set to true. Put the right side row in right side state store.

(3).for left side row needs to be evicted from state store, output the row if "matched" field is true.

(4).for right side row needs to be evicted from state store, doing nothing.
Bug: cannot group by with window in sql statement for structured streaming with watermark
Status: Open | Priority: Major | Resolution: 
Created: 08/Sep/20 10:08
Affects Version/s: 3.1.0

current only support dsl style as below: 

{code}
import spark.implicits._
val words = ... // streaming DataFrame of schema { timestamp: Timestamp, word: String }
// Group the data by window and word and compute the count of each group
val windowedCounts = words.groupBy(window($"timestamp", "10 minutes", "5 minutes"),$"word").count()
{code}
 
but not support group by with window in sql style as below:

{code}
select ts_field,count(\*) as cnt over window(ts_field, '1 minute', '1 minute') with watermark 1 minute from tableX group by ts_field
{code}
Bug: Why am I receiving this warning?
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 25/Sep/20 22:03
Affects Version/s: 3.0.1

I am running Apache Spark Core using Scala 2.12.12 on IntelliJ IDEA 2020.2 with Docker 2.3.0.5
I am running Windows 10 build 2004

Can somebody explain me why am I receiving this warning and what can I do about it?
I tried googling this warning but, all I found was people asking about it and no answers.

[screenshot|https://user-images.githubusercontent.com/1548352/94319642-c8102c80-ff93-11ea-9fea-f58de8da2268.png]

{code:scala}
WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a result reporting of ProcessTree metrics is stopped
{code}

Thanks in advance!
Bug: Duplicate Columns in a csv file
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Sep/20 16:13
Affects Version/s: 2.4.3

Imagine a csv file shaped like:

========================================================

Id,Product,Sale_Amount,Sale_Units,Sale_Amount2,Sale_Amount,Sale_Price
1,P,"6,40,728","6,40,728","6,40,728","6,40,728","6,40,728"
2,P,"5,81,644","5,81,644","5,81,644","5,81,644","5,81,644"

=========================================================

Reading this with the header=True will result in a stacktrace.
Bug: Spark can not cast long value from Kafka
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 18/Sep/20 13:59
Affects Version/s: 3.0.0

Spark seems to be unable to cast the key (or value) part from Kafka to a _{color:#172b4d}long{color}_ value and throws
{code:java}
org.apache.spark.sql.AnalysisException: cannot resolve 'CAST(`key` AS BIGINT)' due to data type mismatch: cannot cast binary to bigint;;{code}
 

{color:#172b4d}See this repo for further investigation:{color} [https://github.com/maseiler/spark-kafka- casting-bug|https://github.com/maseiler/spark-kafka-casting-bug]
Bug: Add withField function to SparkR
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/20 05:16
Affects Version/s: 3.1.0

Already present in [Scala|https://issues.apache.org/jira/browse/SPARK-31317] and  [PySpark|https://issues.apache.org/jira/browse/SPARK-32835].
Bug: Sparksql authentication
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 17/Sep/20 12:30
Affects Version/s: 2.3.0

At present, we want to implement the same authentication module as hive, mainly to realize the blood relationship between table and column (field name). According to the survey, spark is used to resolve the corresponding relationship based on antrl4. However, the test shows that it does not support multi-layer nested query well, and can not accurately locate the corresponding relationship between table and column. We hope that the predecessors can provide help and relative Should the reference material, I will be very grateful, thank you
Bug: Support push-based shuffle in multiple deployment environments
Status: Open | Priority: Major | Resolution: 
Created: 17/Sep/20 18:23
Affects Version/s: 3.1.0

Create this ticket outside of SPARK-30602, since this is outside of the scope of the immediate deliverables in that SPIP. Want to use this ticket to discuss more about how to further improve push-based shuffle in different environments.

The tasks created under SPARK-30602 would enable push-based shuffle on YARN in a compute/storage colocated cluster. However, there are other deployment environments that are getting more popular these days. We have seen 2 as we discussed with other community members on the idea of push-based shuffle:
 * Spark on K8S in a compute/storage colocated cluster. Because of the limitation of concurrency of read/write of a mounted volume in K8S, multiple executor pods on the same node in a K8S cluster cannot concurrently access the same mounted disk volume. This creates some different requirements for supporting external shuffle service as well as push-based shuffle.
 * Spark on a compute/storage disaggregate cluster. Such a setup is more typical in cloud environments, where the compute cluster has little/no local storage, and the shuffle intermediate data needs to be stored in remote disaggregate storage cluster.

Want to use this ticket to discuss ways to support push-based shuffle in these different deployment environments.
Bug: Pass all `external/avro` module UTs in Scala 2.13
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Sep/20 08:14
Affects Version/s: 3.1.0

There are 14 test failed of `external/avro` module as follow:
 * AvroV1Suite(7 FAILED)
 * AvroV2Suite(7 FAILED