av9ash/GitBugs huggingface dataset:
Bug: The .schema() API behaves incorrectly for nested schemas that have column duplicates in case-insensitive mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jul/20 15:20
Affects Version/s: 2.4.6

The code below throws org.apache.spark.sql.AnalysisException: Found duplicate column(s) in the data schema: `camelcase`; for multiple file formats due to a duplicate column in the requested schema.
{code:java}
import org.apache.spark.sql.types._
spark.conf.set("spark.sql.caseSensitive", "false")
val formats = Seq("parquet", "orc", "avro", "json")
val caseInsensitiveSchema = new StructType().add("LowerCase", LongType).add("camelcase", LongType).add("CamelCase", LongType)
formats.map{ format =>
    val path = s"/tmp/$format"
    spark
    .range(1L)
    .selectExpr("id AS lowercase", "id + 1 AS camelCase")
    .write.mode("overwrite").format(format).save(path) 
    spark.read.schema(caseInsensitiveSchema).format(format).load(path).show
}
{code}
Similar code with nested schema behaves inconsistently across file formats and sometimes returns incorrect results:
{code:java}
import org.apache.spark.sql.types._
spark.conf.set("spark.sql.caseSensitive", "false")
val formats = Seq("parquet", "orc", "avro", "json")
val caseInsensitiveSchema = new StructType().add("StructColumn", new StructType().add("LowerCase", LongType).add("camelcase", LongType).add("CamelCase", LongType))
formats.map{ format =>
    val path = s"/tmp/$format"
    spark
    .range(1L)
    .selectExpr("NAMED_STRUCT('lowercase', id, 'camelCase', id + 1) AS StructColumn")
    .write.mode("overwrite").format(format).save(path)
    
    spark.read.schema(caseInsensitiveSchema).format(format).load(path).show
}
{code}
The desired behavior likely should be returning an exception just like in the flat schema scenario.
Bug: Shuffle Storage API: Dynamic updates of shuffle metadata
Status: Open | Priority: Major | Resolution: 
Created: 31/Jul/20 00:09
Affects Version/s: 3.0.0

When using external storage for shuffles as part of the shuffle storage API mechanism, it is often desirable to update the metadata associated with shuffles that we have enabled plugin systems to implement via https://issues.apache.org/jira/browse/SPARK-31801. For example:
 # If data is stored in some replicated manner, and the number of replicas is updated - then we want the metadata stored on the driver to reflect the new number of replicas and where they are located.
 # If data is stored on the mapper's local disk, but is asynchronously backed up to some external storage medium, then we want to know when a backup is available externally.

To achieve this, we would need to pass a hook to updating the shuffle metadata to the shuffle executor components at the root of the plugin tree on the executor side. The executor would establish an RPC connection with the driver and send messages to update shuffle metadata accordingly.
Bug: Clear shuffle state when decommissioned nodes/executors are finally lost
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jul/20 20:35
Affects Version/s: 3.1.0

While every effort has been made to try to migrate the cached and shuffle blocks out of a decommissioned node – its still possible that there are lingering references for some blocks on a decommissioned node. These will result in a fetch failures – that will not only take time to detect but can also cause job failure.

This is a bit tricky in terms of when to clear the shuffle state ? Ideally you want to clear it the millisecond before the shuffle service on the node dies (or the executor dies when there is no external shuffle service) – too soon and it could lead to some wastage and too late would lead to fetch failures.

There are only very few cases where we precisely know when the shuffle data will start being unavailable – perhaps during a cloud spot kill that gives some advance warning. The next best thing is to clear this state lazily at the first sign: ie, when the first fetch failure is observed on a decommissioned entity (node or executor). We take that as a hint that finally the entity has gone away.

What we care about here is whether the shuffle data is going away: ie, if there is an (external) shuffle service resident on the node being decommissioned, or when the shuffle service is embedded inside an executor and the executor is being decommissioned.

This clearing need not be done if the shuffle data is truly remote in certain disaggregated environments.
Bug: Don't fail running jobs when decommissioned executors finally go away
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jul/20 20:25
Affects Version/s: 3.1.0

When a decommissioned executor is finally lost, its death shouldn't fail running jobs. 

A decommissioned executor will eventually die, and in response to its heartbeat failure we will generate a `SlaveLost` message. This SlaveLost message should be treated specially for decommissioned executors: It should not be deemed that this loss is due to the running application. Decommissioning is an exogenous event and the running application shouldn't be penalized for it.
Bug: Pass all `core` module UTs in Scala 2.13
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jul/20 06:43
Affects Version/s: 3.1.0

{code}
$ dev/change-scala-version.sh 2.13
$ build/mvn test -pl core --am -Pscala-2.13
...
Tests: succeeded 2612, failed 3, canceled 1, ignored 8, pending 0
*** 3 TESTS FAILED ***
{code}

*AFTER*
{code}
Tests: succeeded 2615, failed 0, canceled 1, ignored 8, pending 0
All tests passed.
{code}
Bug: Fix the slicing logic in createDataFrame when converting pandas dataframe to arrow table
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 29/Jul/20 09:51
Affects Version/s: 3.1.0

h1. Problem:

In [https://github.com/databricks/runtime/blob/84a952313ae73e3df32f065eb00cc0bcb024af14/python/pyspark/sql/pandas/conversion.py#L418|https://github.com/databricks/runtime/blob/84a952313ae73e3df32f065eb00cc0bcb024af14/python/pyspark/sql/pandas/conversion.py#L418,] , the slicing logic may result in less partitions than specified.
h1. Example:

Assume:
{noformat}
length = 100 -> [0, 1, ..., 99]
num_slices = 99 = self.sparkContext.defaultParallelism{noformat}
Old method:

step = math.ceil(length / num_slices) = 2
 start = i * step, end = (i + 1) * step:
 output: [0,1] [2,3] [4,5] ... [98,99] -> 50 slices != num_slices

 
h1. Solution:

We can use a silimar logic as in [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/ParallelCollectionRDD.scala#L125]
{code:python}
# replace conversion.py#L418
pdf_slices = (pdf.iloc[i * length // num_slices: (i + 1) * length // num_slices] for i in xrange(0, num_slices))
{code}
New method:
 start = i * length // num_slices, end = (i + 1) * length // num_slices:
 output: [0] [1] [2] ... [98,99] -> 99 slices
Bug: Support reset one particular configuration
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/20 07:54
Affects Version/s: 3.1.0

Hive:
|reset <key>|Resets the value of a particular configuration variable (key) to the default value.
*Note:* If you misspell the variable name, Beeline will not show an error.|

 

PostgreSQL:

[https://www.postgresql.org/docs/9.1/sql-reset.html]

 

Looks good to have our reset command to support this
Bug: Remove javax.ws.rs.NotFoundException from `import` in StagesResource/OneApplicationResource
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/20 21:30
Affects Version/s: 3.1.0

To use `org.apache.spark.status.api.v1.NotFoundException`, we need the following.
{code}
-import javax.ws.rs._
+import javax.ws.rs.{NotFoundException => _, _}
{code}
Bug: How do I get the SPARK shuffle monitoring indicator?
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 28/Jul/20 09:12
Affects Version/s: 2.2.1

I want to monitor the spark task through graphite_export and prometheus, I have collected execution, driver cpu and memory related indicators, but there is no shuffle read and shuffle write related indicators, how should I configure it?

metrics.properties:
master.source.jvm.class=org.apache.spark.metrics.source.JvmSource
worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource
shuffleService.source.jvm.class=org.apache.spark.metrics.source.JvmSource
Bug: Use === instead IndexSeqView
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/20 03:44
Affects Version/s: 3.1.0

This issue aims to fix `SorterSuite` and `RadixSortSuite` in Scala 2.13 by using `sameElements` instead of `IndexSeqView.==`.

Scala 2.13 reimplements `IndexSeqView` and the behavior is different.
- https://docs.scala-lang.org/overviews/core/collections-migration-213.html

{code}
Welcome to Scala 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_262).
Type in expressions for evaluation. Or try :help.

scala> Seq(1,2,3).toArray.view == Seq(1,2,3).toArray.view
res0: Boolean = true
{code}

{code}
Welcome to Scala 2.13.3 (OpenJDK 64-Bit Server VM, Java 1.8.0_262).
Type in expressions for evaluation. Or try :help.

scala> Seq(1,2,3).toArray.view == Seq(1,2,3).toArray.view
val res0: Boolean = false
{code}

{code}
$ dev/change-scala-version.sh 2.13
$ build/mvn test -pl core --am -Pscala-2.13 -Dtest=none -DwildcardSuites=org.apache.spark.util.collection.unsafe.sort.RadixSortSuite
...
Tests: succeeded 9, failed 36, canceled 0, ignored 0, pending 0
*** 36 TESTS FAILED ***

$ build/mvn test -pl core --am -Pscala-2.13 -Dtest=none -DwildcardSuites=org.apache.spark.util.collection.SorterSuite
...
Tests: succeeded 3, failed 1, canceled 0, ignored 2, pending 0
*** 1 TEST FAILED ***
{code}
Bug: Standalone Mode allow setting CUDA_VISIBLE_DEVICES on executor launch
Status: Open | Priority: Major | Resolution: 
Created: 24/Jul/20 13:23
Affects Version/s: 3.0.0

It would be nice if standalone mode could allow users to set CUDA_VISIBLE_DEVICES before launching an executor.  This has multiple benefits. 
 * kind of an isolation in that the executor can only see the GPUs set there. 
 * If your GPU application doesn't support explicitly setting the GPU device id, setting this will make any GPU look like the default (id 0) and things generally just work without any explicit setting
 * New features are being added on newer GPUs that require explicit setting of CUDA_VISIBLE_DEVICES like MIG ([https://www.nvidia.com/en-us/technologies/multi-instance-gpu/])

The code changes to just set this are very small, once we set them we would also possibly need to change the gpu addresses as it changes them to start from device id 0 again.

The easiest implementation would just specifically support this and have it behind a config and set when the config is on and GPU resources are allocated. 

Note we probably want to have this same thing set when we launch a python process as well so that it gets same env.
Bug: Improve MLlib BLAS native acceleration docs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jul/20 02:03
Affects Version/s: 3.1.0

The document of enabling BLAS native acceleration in ML guide ([https://spark.apache.org/docs/latest/ml-guide.html#dependencies]) is incomplete and unclear to the user. 

We will rewrite it to a clearer and complete guide.
Bug: UDF regression of WrappedArray supporting caused by SPARK-31826
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/20 12:55
Affects Version/s: 3.1.0

 
{code:java}
test("WrappedArray") {
  val myUdf = udf((a: WrappedArray[Int]) =>
    WrappedArray.make[Int](Array(a.head + 99)))
  checkAnswer(Seq(Array(1))
    .toDF("col")
    .select(myUdf(Column("col"))),
    Row(ArrayBuffer(100)))
}{code}
Execute the above test in master branch, we'll hit the error:


{code:java}
[info]   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, 192.168.101.3, executor driver): java.lang.RuntimeException: Error while decoding: java.lang.ClassCastException: scala.collection.mutable.ArrayBuffer cannot be cast to scala.collection.mutable.WrappedArray[info]  {code}
However, the test can be executed successfully in branch-3.0.

 

This's actually a regression caused by SPARK-31826. And the regression happens after we changed the catalyst-to-scala converter from CatalystTypeConverters to ExpressionEncoder.deserializer .
Bug: Automatic type promotion to ArrayType in defined schema in from_json is broken
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 05/Jul/20 21:05
Affects Version/s: 3.0.0

 

In spark 2.4, I'm able to read data where I have data in mixed types, by defining col "stats" as StringType and later parse the inner data
  
 stats_def = StructType().add("hour",IntegerType(),True).add("hits",IntegerType(),True)
 df2 = df.select(f.col("stats"),f.from_json(f.col("stats"),ArrayType(stats_def)).alias("stats_array"))
 df2.show(5,False)
 df2.printSchema
  
||stats||stats_array||
|[\{"hour":3,"hits":1},\{"hour":4,"hits":1}]|[[3, 1], [4, 1]]|
|{"hits":20}|[[, 20]]|

<bound method DataFrame.printSchema of DataFrame[*stats: string, stats_array: array<struct<hour:int,hits:int>>*]>
  
 In spark 3.0.0 it throws error -
 java.lang.ClassCastException: java.lang.Integer cannot be cast to org.apache.spark.sql.catalyst.util.ArrayData
  
 I think it was an important feature and should be supported, maybe with the help of from_json options.
Bug: PySpark regexp_replace not replacing JSON çlçl
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 10/Jul/20 00:29
Affects Version/s: 2.4.4

This is an example of the column value:

```hocon

'\{someVersion=8.3.2-hmg-dev, someUnitName=IB, someMessage=Test. [BL056], someOrigin=MOBILE, someStatus=TEST, duration=3500, someNumber=9872329, someAppOrigin=APP_PADRAO, someId=c3ASAUSQTiWvl_YA9DYpDV:APA91bGfVcLNNGL20hfmaDDS0D8TuzJDuCjj4tgbRNcJcYASIBRVEE2FnA4exnE4ZWTuupRX7FQkdcJiMWkNEatk8lktkFcpR7P7mehb4r_SVnabIabGInjagGZ6pGyweDkxW2JUGK8g, someType=00001, someOriginOpen=null, someOS=null, eventSubType=TESTLOGON, someToken=, ip=error, somePair=0.4220043,-1.084015, eventType=SUCESSO, someMag=aWg4V01qSxDMjAvWmlEWGJ6aExnc2nZJbWZVPQ==, macAddress=33d94a3f7d2f8aff, someJSON={"ip":"error","hostname":null,"type":null,"concode":null,"continent":null,"country":null,"country_name":null,"code":null,"name":null,"city":null,"zip":null,"latitude":null,"longitude":null,"anotherJSON":{"id":null,"capital":null,"languages":null,"flag":null,"flag_emoji":null,"flag_emoji_unicode":null,"calling_code":null,"is_eu":null},"time_zone":\{"id":null,"current_time":null,"gmt_offset":null,"code":null,"is_daylight_saving":null},"currency":\{"code":null,"name":null,"plural":null,"symbol":null,"symbol_native":null},"connection":\{"asn":null,"isp":null},"security":\{"is_proxy":null,"proxy_type":null,"is_crawler":null,"crawler_name":null,"crawler_type":null,"is_tor":null,"threat_level":null,"threat_types":null}}, organization=IBPF, codigoCliente=440149, device=Android SDK built for x86, eventDate=6/1/20 4:03 PM}'

```
Bug: PySpark regexp_replace not replacing JSON çlçlçl
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 10/Jul/20 00:29
Affects Version/s: 2.4.4

This is an example of the column value:

```hocon

'\{someVersion=8.3.2-hmg-dev, someUnitName=IB, someMessage=Test. [BL056], someOrigin=MOBILE, someStatus=TEST, duration=3500, someNumber=9872329, someAppOrigin=APP_PADRAO, someId=c3ASAUSQTiWvl_YA9DYpDV:APA91bGfVcLNNGL20hfmaDDS0D8TuzJDuCjj4tgbRNcJcYASIBRVEE2FnA4exnE4ZWTuupRX7FQkdcJiMWkNEatk8lktkFcpR7P7mehb4r_SVnabIabGInjagGZ6pGyweDkxW2JUGK8g, someType=00001, someOriginOpen=null, someOS=null, eventSubType=TESTLOGON, someToken=, ip=error, somePair=0.4220043,-1.084015, eventType=SUCESSO, someMag=aWg4V01qSxDMjAvWmlEWGJ6aExnc2nZJbWZVPQ==, macAddress=33d94a3f7d2f8aff, someJSON={"ip":"error","hostname":null,"type":null,"concode":null,"continent":null,"country":null,"country_name":null,"code":null,"name":null,"city":null,"zip":null,"latitude":null,"longitude":null,"anotherJSON":{"id":null,"capital":null,"languages":null,"flag":null,"flag_emoji":null,"flag_emoji_unicode":null,"calling_code":null,"is_eu":null},"time_zone":\{"id":null,"current_time":null,"gmt_offset":null,"code":null,"is_daylight_saving":null},"currency":\{"code":null,"name":null,"plural":null,"symbol":null,"symbol_native":null},"connection":\{"asn":null,"isp":null},"security":\{"is_proxy":null,"proxy_type":null,"is_crawler":null,"crawler_name":null,"crawler_type":null,"is_tor":null,"threat_level":null,"threat_types":null}}, organization=IBPF, codigoCliente=440149, device=Android SDK built for x86, eventDate=6/1/20 4:03 PM}'

```
Bug: PySpark regexp_replace not replacing JSON çlçlçlçl
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 10/Jul/20 00:29
Affects Version/s: 2.4.4

This is an example of the column value:

```hocon

'\{someVersion=8.3.2-hmg-dev, someUnitName=IB, someMessage=Test. [BL056], someOrigin=MOBILE, someStatus=TEST, duration=3500, someNumber=9872329, someAppOrigin=APP_PADRAO, someId=c3ASAUSQTiWvl_YA9DYpDV:APA91bGfVcLNNGL20hfmaDDS0D8TuzJDuCjj4tgbRNcJcYASIBRVEE2FnA4exnE4ZWTuupRX7FQkdcJiMWkNEatk8lktkFcpR7P7mehb4r_SVnabIabGInjagGZ6pGyweDkxW2JUGK8g, someType=00001, someOriginOpen=null, someOS=null, eventSubType=TESTLOGON, someToken=, ip=error, somePair=0.4220043,-1.084015, eventType=SUCESSO, someMag=aWg4V01qSxDMjAvWmlEWGJ6aExnc2nZJbWZVPQ==, macAddress=33d94a3f7d2f8aff, someJSON={"ip":"error","hostname":null,"type":null,"concode":null,"continent":null,"country":null,"country_name":null,"code":null,"name":null,"city":null,"zip":null,"latitude":null,"longitude":null,"anotherJSON":{"id":null,"capital":null,"languages":null,"flag":null,"flag_emoji":null,"flag_emoji_unicode":null,"calling_code":null,"is_eu":null},"time_zone":\{"id":null,"current_time":null,"gmt_offset":null,"code":null,"is_daylight_saving":null},"currency":\{"code":null,"name":null,"plural":null,"symbol":null,"symbol_native":null},"connection":\{"asn":null,"isp":null},"security":\{"is_proxy":null,"proxy_type":null,"is_crawler":null,"crawler_name":null,"crawler_type":null,"is_tor":null,"threat_level":null,"threat_types":null}}, organization=IBPF, codigoCliente=440149, device=Android SDK built for x86, eventDate=6/1/20 4:03 PM}'

```
Bug: Introduce UnresolvedTableOrPermanentView for commands that support a table/view but not a temporary view
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 15/Jul/20 21:48
Affects Version/s: 3.1.0

We should have UnresolvedTableOrPermanentView for commands that do support a table or a view, but not a temporary view, such that an analysis can fail if an identifier is resolved to a temporary view for those commands 

 

For example, SHOW TBLPROPERTIES should not support a temp view since it always returns an empty result, which could be misleading.
Bug: Support skew handling on join that has one side with no query stage
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Jul/20 01:22
Affects Version/s: 3.0.0

In our production environment, there are many bucket tables, which are used to join with other tables. And there are some skewed joins now and then. While, in current implementation, the skew join handling can only applied when both sides of a SMJ are QueryStages. So skew join handling is not able to deal with such cases.
Bug: Preserve shuffled hash join build side partitioning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/20 05:42
Affects Version/s: 3.1.0

Currently `ShuffledHashJoin.outputPartitioning` inherits from `HashJoin.outputPartitioning`, which only preserves stream side partitioning:

`HashJoin.scala`
{code:java}
override def outputPartitioning: Partitioning = streamedPlan.outputPartitioning
{code}
This loses build side partitioning information, and causes extra shuffle if there's another join / group-by after this join.

Example:

 
{code:java}
// code placeholder
withSQLConf(
    SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "50",
    SQLConf.SHUFFLE_PARTITIONS.key -> "2",
    SQLConf.PREFER_SORTMERGEJOIN.key -> "false") {
  val df1 = spark.range(10).select($"id".as("k1"))
  val df2 = spark.range(30).select($"id".as("k2"))
  Seq("inner", "cross").foreach(joinType => {
    val plan = df1.join(df2, $"k1" === $"k2", joinType).groupBy($"k1").count()
      .queryExecution.executedPlan
    assert(plan.collect { case _: ShuffledHashJoinExec => true }.size === 1)
    // No extra shuffle before aggregate
    assert(plan.collect { case _: ShuffleExchangeExec => true }.size === 2)
  })
}{code}
 

Current physical plan (having an extra shuffle on `k1` before aggregate)

 
{code:java}
*(4) HashAggregate(keys=[k1#220L], functions=[count(1)], output=[k1#220L, count#235L])
+- Exchange hashpartitioning(k1#220L, 2), true, [id=#117]
   +- *(3) HashAggregate(keys=[k1#220L], functions=[partial_count(1)], output=[k1#220L, count#239L])
      +- *(3) Project [k1#220L]
         +- ShuffledHashJoin [k1#220L], [k2#224L], Inner, BuildLeft
            :- Exchange hashpartitioning(k1#220L, 2), true, [id=#109]
            :  +- *(1) Project [id#218L AS k1#220L]
            :     +- *(1) Range (0, 10, step=1, splits=2)
            +- Exchange hashpartitioning(k2#224L, 2), true, [id=#111]
               +- *(2) Project [id#222L AS k2#224L]
                  +- *(2) Range (0, 30, step=1, splits=2){code}
 

Ideal physical plan (no shuffle on `k1` before aggregate)
{code:java}
 *(3) HashAggregate(keys=[k1#220L], functions=[count(1)], output=[k1#220L, count#235L])
+- *(3) HashAggregate(keys=[k1#220L], functions=[partial_count(1)], output=[k1#220L, count#239L])
   +- *(3) Project [k1#220L]
      +- ShuffledHashJoin [k1#220L], [k2#224L], Inner, BuildLeft
         :- Exchange hashpartitioning(k1#220L, 2), true, [id=#107]
         :  +- *(1) Project [id#218L AS k1#220L]
         :     +- *(1) Range (0, 10, step=1, splits=2)
         +- Exchange hashpartitioning(k2#224L, 2), true, [id=#109]
            +- *(2) Project [id#222L AS k2#224L]
               +- *(2) Range (0, 30, step=1, splits=2){code}
 

This can be fixed by overriding `outputPartitioning` method in `ShuffledHashJoinExec`, similar to `SortMergeJoinExec`.
Bug: Fix TaskSetManagerSuite by hiding `org.apache.spark.FakeSchedulerBackend`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jul/20 04:19
Affects Version/s: 3.1.0

There exists two FakeSchedulerBackend.
{code}
$ git grep "class FakeSchedulerBackend"
core/src/test/scala/org/apache/spark/HeartbeatReceiverSuite.scala:private class FakeSchedulerBackend(
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala:class FakeSchedulerBackend extends SchedulerBackend {
{code}

Although `TaskSetManagerSuite` is inside `org.apache.spark.scheduler` package, `import org.apache.spark._` makes Scala 2.13 confused and causes 4 UT failures.
{code}
$ build/mvn test -pl core --am -Pscala-2.13 -Dtest=none -DwildcardSuites=org.apache.spark.scheduler.TaskSetManagerSuite
...
Tests: succeeded 48, failed 4, canceled 0, ignored 0, pending 0
*** 4 TESTS FAILED ***
{code}

This issue aims to fix that by hiding that.
{code}
import org.apache.spark.{FakeSchedulerBackend => _, _}
{code}