av9ash/GitBugs huggingface dataset:
Bug: Gaussian can not handle data close to MaxDouble
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 07/Aug/20 20:52
Affects Version/s: 3.0.0

Running Gaussian from Apache Spark MLlib with [this dataset|[https://user.informatik.uni-goettingen.de/~sherbol/MaxDouble.arff]] containing values close to MaxDouble (values >10^306) results in the error below. KMeans and Bisecting KMeans can both handle the same dataset which for me raises the question, if this would be a bug or to be expected behavior.

Stacktrace:

org.apache.spark.SparkException: Failed to execute user defined function(GaussianMixtureModel$$Lambda$2841/0x00000001003ab040: (struct<type:tinyint,size:int,indices:array<int>,values:array<double>>) => struct<type:tinyint,size:int,indices:array<int>,values:array<double>>)
 at org.apache.spark.sql.catalyst.expressions.ScalaUDF.eval(ScalaUDF.scala:1070)
 at org.apache.spark.sql.catalyst.expressions.Alias.eval(namedExpressions.scala:156)
 at org.apache.spark.sql.catalyst.expressions.InterpretedMutableProjection.apply(InterpretedMutableProjection.scala:83)

at org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation$$anonfun$apply$17.applyOrElse(Optimizer.scala:1502)
 at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$1(TreeNode.scala:286)

at org.apache.spark.ml.clustering.ClusteringSummary.clusterSizes$lzycompute(ClusteringSummary.scala:49)
 at org.apache.spark.ml.clustering.GaussianMixture.fit(GaussianMixture.scala:374)


 Caused by: breeze.linalg.NotConvergedException: 
 at breeze.linalg.eigSym$.breeze$linalg$eigSym$$doEigSym(eig.scala:164)
 at breeze.linalg.eigSym$EigSym_DM_Impl$.apply(eig.scala:111)
 at breeze.linalg.eigSym$EigSym_DM_Impl$.apply(eig.scala:109)
 at breeze.generic.UFunc.apply(UFunc.scala:46)
 at breeze.generic.UFunc.apply$(UFunc.scala:45)
 at breeze.linalg.eigSym$.apply(eig.scala:106)
 at org.apache.spark.ml.stat.distribution.MultivariateGaussian.calculateCovarianceConstants(MultivariateGaussian.scala:117)
 at org.apache.spark.ml.stat.distribution.MultivariateGaussian.x$1$lzycompute(MultivariateGaussian.scala:58)
 at org.apache.spark.ml.stat.distribution.MultivariateGaussian.x$1(MultivariateGaussian.scala:58)


 at org.apache.spark.ml.clustering.GaussianMixtureModel$.computeProbabilities(GaussianMixture.scala:287)
 at org.apache.spark.ml.clustering.GaussianMixtureModel.predictProbability(GaussianMixture.scala:171)
Bug: --py-files option is appended without passing value for it
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Aug/20 00:58
Affects Version/s: 3.0.0

Submitted application passing --py-files option in a hardcoded manner for a Mesos Cluster in cluster mode using REST Submission API. It is causing a simple Java-based SparkPi job to fail.

This Bug is introduced by SPARK-26466.

Here is the example job submission:
{code:bash}
curl -X POST http://localhost:7077/v1/submissions/create --header "Content-Type:application/json" --data '{
"action": "CreateSubmissionRequest",
"appResource": "file:///opt/spark-3.0.0-bin-3.2.0/examples/jars/spark-examples_2.12-3.0.0.jar",
"clientSparkVersion": "3.0.0",
"appArgs": ["30"],
"environmentVariables": {},
"mainClass": "org.apache.spark.examples.SparkPi",
"sparkProperties": {
  "spark.jars": "file:///opt/spark-3.0.0-bin-3.2.0/examples/jars/spark-examples_2.12-3.0.0.jar",
  "spark.driver.supervise": "false",
  "spark.executor.memory": "512m",
  "spark.driver.memory": "512m",
  "spark.submit.deployMode": "cluster",
  "spark.app.name": "SparkPi",
  "spark.master": "mesos://localhost:5050"
}}'
{code}
Expected Driver log would contain:
{code:bash}
20/08/20 20:19:57 WARN DependencyUtils: Local jar /var/lib/mesos/slaves/e6779377-08ec-4765-9bfc-d27082fbcfa1-S0/frameworks/e6779377-08ec-4765-9bfc-d27082fbcfa1-0000/executors/driver-20200820201954-0002/runs/d9d734e8-a299-4d87-8f33-b134c65c422b/spark.driver.memory=512m does not exist, skipping.
Error: Failed to load class org.apache.spark.examples.SparkPi.
20/08/20 20:19:57 INFO ShutdownHookManager: Shutdown hook called
{code}
Bug: Logging and Swallowing the Exception Per Entry in History Server
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Aug/20 06:50
Affects Version/s: 3.0.0

As discussed in [https://github.com/apache/spark/pull/29350]

To avoid any entry affect others while History server scanning log dir, we would like to add a try catch to log and swallow the exception per entry.
Bug: Check if the same ExprId refers to the unique attribute in logical plans
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Aug/20 12:28
Affects Version/s: 3.1.0

Some plan transformations (e.g., `RemoveNoopOperators`) implicitly assume the same `ExprId` refers to the unique attribute. But, `RuleExecutor` does not check this integrity between logical plan transformations. So, this ticket targets at adding this check in `isPlanIntegral` of `Analyzer`/`Optimizer`.

This PR comes from the talk with @cloud-fan @viirya in https://github.com/apache/spark/pull/29485#discussion_r475346278
Bug: Upgrade to jQuery 3.5.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Aug/20 06:09
Affects Version/s: 3.0.0

Spark 3.0, Spark 2.4.x uses JQuery version < 3.5 which has known security vulnerability in Spark Master UI and Spark Worker UI.

Can we please upgrade JQuery to 3.5 and above ?

 [https://www.tenable.com/plugins/nessus/136929]

??According to the self-reported version in the script, the version of JQuery hosted on the remote web server is greater than or equal to 1.2 and prior to 3.5.0. It is, therefore, affected by multiple cross site scripting vulnerabilities.??
Bug: Fix the data issue of inserted DPP on non-atomic type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Aug/20 08:41
Affects Version/s: 3.0.0

DPP has data issue when pruning on non-atomic type. for example:
{noformat}
 spark.range(1000)
 .select(col("id"), col("id").as("k"))
 .write
 .partitionBy("k")
 .format("parquet")
 .mode("overwrite")
 .saveAsTable("df1");

spark.range(100)
 .select(col("id"), col("id").as("k"))
 .write
 .partitionBy("k")
 .format("parquet")
 .mode("overwrite")
 .saveAsTable("df2")

spark.sql("set spark.sql.optimizer.dynamicPartitionPruning.fallbackFilterRatio=2")
spark.sql("set spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly=false")
spark.sql("SELECT df1.id, df2.k FROM df1 JOIN df2 ON struct(df1.k) = struct(df2.k) AND df2.id < 2").show
{noformat}


 It should return two records, but it returns empty.
Bug: PageRank not sending the correct values in Pergel sendMessage
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 10/Aug/20 06:20
Affects Version/s: 2.3.0

The core sendMessage method is incorrect:
{code:java}
def sendMessage(edge: EdgeTriplet[(Double, Double), Double]) = {
 if (edge.srcAttr._2 > tol) {
   Iterator((edge.dstId, edge.srcAttr._2 * edge.attr))
                          // *** THIS ^ ***
 } else {
   Iterator.empty
 }
}{code}
 

Instead of using the source PR value, it's using the PR delta (2nd tuple arg). This is not the documented behavior, nor a valid PR algorithm AFAIK.

This is a 7 years old code, all versions affected.
Bug: Broadcast block pieces may memory leak
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Aug/20 11:58
Affects Version/s: 2.4.6

We use Spark thrift-server as a long-running service. A bad query submitted a heavy BroadcastNestLoopJoin operation and made driver full GC. We killed the bad query but we found the driver's memory usage was still high and full GCs had very frequency. By investigating with GC dump and log, we found the broadcast may memory leak.

2020-08-19T18:54:02.824-0700: [Full GC (Allocation Failure) 2020-08-19T18:54:02.824-0700: [Class Histogram (before full gc):
116G->112G(170G), 184.9121920 secs]
[Eden: 32.0M(7616.0M)->0.0B(8704.0M) Survivors: 1088.0M->0.0B Heap: 116.4G(170.0G)->112.9G(170.0G)], [Metaspace: 177285K->177270K(182272K)]

num #instances #bytes class name
----------------------------------------------
1: 676531691 72035438432 [B
2: 676502528 32472121344 org.apache.spark.sql.catalyst.expressions.UnsafeRow
3: 99551 12018117568 [Ljava.lang.Object;
4: 26570 4349629040 [I
5: 6 3264536688 [Lorg.apache.spark.sql.catalyst.InternalRow;
6: 1708819 256299456 [C
7: 2338 179615208 [J
8: 1703669 54517408 java.lang.String
9: 103860 34896960 org.apache.spark.status.TaskDataWrapper
10: 177396 25545024 java.net.URI
...
Bug: Add an optimizer rule to split an Expand into multiple Expands for aggregates
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 05/Aug/20 08:51
Affects Version/s: 3.0.0

Split an expand into several small Expand, which contains the Specified number of projections.
For instance, like this sql.select a, b, c, d, count(1) from table1 group by a, b, c, d with cube. It can expand 2^4 times of original data size.
Now we specify the spark.sql.optimizer.projections.size=4.The Expand will be split into 2^4/4 smallExpand.It can reduce the shuffle pressure and improve performance in multidimensional analysis when data is huge.
Bug: Maintain the order of expressions in AttributeSet and ExpressionSet 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Aug/20 14:40
Affects Version/s: 3.1.0

Expressions identity is based on the ExprId which is an auto-incremented number. This means that the same query can yield a query plan with different expression ids in different runs. AttributeSet and ExpressionSet internally use a HashSet as the underlying data structure, and therefore cannot guarantee the a fixed order of operations in different runs. This can be problematic in cases we like to check for plan changes in different runs.

We change do the following changes to AttributeSet and ExpressionSet to maintain the insertion order of the elements:
 * We change the underlying data structure of AttributeSet from HashSet to LinkedHashSet to maintain the insertion order.
 * ExpressionSet already uses a list to keep track of the expressions, however, since it is extending Scala's immutable.Set class, operations such as map and flatMap are delegated to the immutable.Set itself. This means that the result of these operations is not an instance of ExpressionSet anymore, rather it's a implementation picked up by the parent class. We also remove this inheritance from immutable.Set and implement the needed methods directly. ExpressionSet has a very specific semantics and it does not make sense to extend immutable.Set anyway.
 * We change the PlanStabilitySuite to not sort the attributes, to be able to catch changes in the order of expressions in different runs.
Bug: Reset the numPartitions metric when DPP is enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Aug/20 00:20
Affects Version/s: 3.0.0

This pr reset the {{numPartitions}} metric when DPP is enabled. Otherwise, it is always a [static value|https://github.com/apache/spark/blob/18cac6a9f0bf4a6d449393f1ee84004623b3c893/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L215].
Bug: WidenSetOperationTypes in subquery  attribute  missing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/20 11:16
Affects Version/s: 2.3.4

I am migrating sql from mysql to spark sql, meet a very strange case. Below is code to reproduce the exception:

 
{code:java}
val spark = SparkSession.builder()
 .master("local")
 .appName("Word Count")
 .getOrCreate()
spark.sparkContext.setLogLevel("TRACE")
val DecimalType = DataTypes.createDecimalType(20, 2)
val schema = StructType(List(
 StructField("a", DecimalType, true)
))
val dataList = new util.ArrayList[Row]()


val df=spark.createDataFrame(dataList,schema)
df.printSchema()
df.createTempView("test")
val sql=
 """
 |SELECT t.kpi_04 FROM
 |(
 | SELECT a as `kpi_04` FROM test
 | UNION ALL
 | SELECT a+a as `kpi_04` FROM test
 |) t
 |
 """.stripMargin
spark.sql(sql)
{code}
 

Exception Message:

 
{code:java}
Exception in thread "main" org.apache.spark.sql.AnalysisException: Resolved attribute(s) kpi_04#2 missing from kpi_04#4 in operator !Project [kpi_04#2]. Attribute(s) with the same name appear in the operation: kpi_04. Please check if the right attribute(s) are used.;;
!Project [kpi_04#2]
+- SubqueryAlias t
 +- Union
 :- Project [cast(kpi_04#2 as decimal(21,2)) AS kpi_04#4]
 : +- Project [a#0 AS kpi_04#2]
 : +- SubqueryAlias test
 : +- LocalRelation <empty>, [a#0]
 +- Project [kpi_04#3]
 +- Project [CheckOverflow((promote_precision(cast(a#0 as decimal(21,2))) + promote_precision(cast(a#0 as decimal(21,2)))), DecimalType(21,2)) AS kpi_04#3]
 +- SubqueryAlias test
 +- LocalRelation <empty>, [a#0]{code}
 

 

Base the trace log ,seemly the WidenSetOperationTypes add new outer project layer. It caused the parent query lose the reference to subquery. 

 

 
{code:java}
 
=== Applying Rule org.apache.spark.sql.catalyst.analysis.TypeCoercion$WidenSetOperationTypes ===
!'Project [kpi_04#2] !Project [kpi_04#2]
!+- 'SubqueryAlias t +- SubqueryAlias t
! +- 'Union +- Union
! :- Project [a#0 AS kpi_04#2] :- Project [cast(kpi_04#2 as decimal(21,2)) AS kpi_04#4]
! : +- SubqueryAlias test : +- Project [a#0 AS kpi_04#2]
! : +- LocalRelation <empty>, [a#0] : +- SubqueryAlias test
! +- Project [CheckOverflow((promote_precision(cast(a#0 as decimal(21,2))) + promote_precision(cast(a#0 as decimal(21,2)))), DecimalType(21,2)) AS kpi_04#3] : +- LocalRelation <empty>, [a#0]
! +- SubqueryAlias test +- Project [kpi_04#3]
! +- LocalRelation <empty>, [a#0] +- Project [CheckOverflow((promote_precision(cast(a#0 as decimal(21,2))) + promote_precision(cast(a#0 as decimal(21,2)))), DecimalType(21,2)) AS kpi_04#3]
! +- SubqueryAlias test
! +- LocalRelation <empty>, [a#0]
{code}
 

  in the source code ,WidenSetOperationTypes.scala. it is  a intent behavior, but  possibly  miss this edge case. 

I hope someone can help me out to fix it . 

 

 
{code:java}
if (targetTypes.nonEmpty) {
 // Add an extra Project if the targetTypes are different from the original types.
 children.map(widenTypes(_, targetTypes))
} else {
 // Unable to find a target type to widen, then just return the original set.
 children
}{code}
Bug: Deduplicating and repartitioning the same column create duplicate rows with AQE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Aug/20 06:51
Affects Version/s: 3.0.0

To reproduce:
{code:java}
spark.range(10).union(spark.range(10)).createOrReplaceTempView("v1")
val df = spark.sql("select id from v1 group by id distribute by id") 
println(df.collect().toArray.mkString(","))
println(df.queryExecution.executedPlan)

// With AQE
[4],[0],[3],[2],[1],[7],[6],[8],[5],[9],[4],[0],[3],[2],[1],[7],[6],[8],[5],[9]
AdaptiveSparkPlan(isFinalPlan=true)
+- CustomShuffleReader local
   +- ShuffleQueryStage 0
      +- Exchange hashpartitioning(id#183L, 10), true
         +- *(3) HashAggregate(keys=[id#183L], functions=[], output=[id#183L])
            +- Union
               :- *(1) Range (0, 10, step=1, splits=2)
               +- *(2) Range (0, 10, step=1, splits=2)

// Without AQE
[4],[7],[0],[6],[8],[3],[2],[5],[1],[9]
*(4) HashAggregate(keys=[id#206L], functions=[], output=[id#206L])
+- Exchange hashpartitioning(id#206L, 10), true
   +- *(3) HashAggregate(keys=[id#206L], functions=[], output=[id#206L])
      +- Union
         :- *(1) Range (0, 10, step=1, splits=2)
         +- *(2) Range (0, 10, step=1, splits=2){code}
Bug: Support local property propagation in SubqueryBroadcastExec
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 31/Aug/20 05:09
Affects Version/s: 3.0.0

Since SPARK-22590, local property propagation is supported through `SQLExecution.withThreadLocalCaptured` in both `BroadcastExchangeExec` and `SubqueryExec` when computing `relationFuture`.

The propagation is missed in `SubqueryBroadcastExec`.
Bug: Add Application attemptId support to SQL Rest API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/20 19:22
Affects Version/s: 3.1.0

Currently, Spark Public Rest APIs support Application attemptId except SQL API. This causes *no such app: application_X* issue when the application has *{{attemptId}}*.

Please find existing and supported Rest endpoints with {{attemptId}}.
{code:java}
// Existing Rest Endpoints
applications/{appId}/sql
applications/{appId}/sql/{executionId}

// Rest Endpoints required support
applications/{appId}/{attemptId}/sql 
applications/{appId}/{attemptId}/sql/{executionId}{code}
Bug: SPARK SQL JDBC truncates last value of seconds for datetime2 values for Azure SQL DB 
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 17/Aug/20 10:50
Affects Version/s: 2.4.1

SPARK jdbc is truncating TIMESTAMP values for the microsecond when datetime2 datatype is used for Microsoft SQL Server JDBC driver.

 

Source data(datetime2) : '2007-08-08 12:35:29.1234567'

 

After loading to target using SPARK dataframes

 

Target data(datetime2) : '2007-08-08 12:35:29.1234560'
Bug: RDD actions in DStream.transfrom delays batch submission
Status: In Progress | Priority: Major | Resolution: 
Created: 29/Aug/20 14:28
Affects Version/s: 3.0.0

h4. Issue

Some spark applications have batch creation delay after running for some time. For instance, Batch 10:03 is submitted at 10:06. In spark UI, the latest batch doesn't match current time.
  
||Clock||BatchTime||
|10:00|10:00|
|10:02|10:01|
|10:04|10:02|
|10:06|10:03|
h4. Investigation

We observe such applications share a commonality that rdd actions exist in dstream.transfrom. Those actions will be executed in dstream.compute, which is called by JobGenerator. JobGenerator runs with a single thread event loop so any synchronized operations will block event processing.
h4. Proposal

delegate dstream.compute to JobSchduler

 
{code:java}
// class ForEachDStream

override def generateJob(time: Time): Option[Job] = {
  val jobFunc = () => createRDDWithLocalProperties(time, displayInnerRDDOps) {
    parent.getOrCompute(time).foreach(rdd => foreachFunc(rdd, time))
  }
  Some(new Job(time, jobFunc))
}
{code}
Bug: RDD actions in DStream.transfrom don't show at batch page
Status: In Progress | Priority: Major | Resolution: 
Created: 29/Aug/20 14:43
Affects Version/s: 3.0.0

h4. Issue
{code:java}
val lines = ssc.socketTextStream("localhost", 9999)
val words = lines.flatMap(_.split(" "))
val mappedStream= words.transform(rdd => {
  val c = rdd.count();
  rdd.map(x => s"$c x")}
)
mappedStream.foreachRDD(rdd => rdd.foreach(x => println(x))){code}
Every batch two spark jobs are created. Only the second one is associated with the streaming output operation and shows at batch page.
h4. Investigation

The first action rdd.count() is invoked by JobGenerator.generateJobs. Batch time and output op id are not available in spark context because they are set in JobScheduler later.
h4. Proposal

delegate dstream.getOrCompute to JobScheduler so that all rdd actions can run in spark context with correct local properties.
Bug: Alias breaks for interval typed literals
Status: In Progress | Priority: Major | Resolution: 
Created: 31/Aug/20 06:21
Affects Version/s: 3.0.0

Cases we found：

{code:java}
+-- !query
+select interval '1 day' as day
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.parser.ParseException
+
+no viable alternative at input 'as'(line 1, pos 24)
+
+== SQL ==
+select interval '1 day' as day
+------------------------^^^
+
+
+-- !query
+select interval '1 day' day
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.parser.ParseException
+
+Error parsing ' 1 day day' to interval, unrecognized number 'day'(line 1, pos 16)
+
+== SQL ==
+select interval '1 day' day
+----------------^^^
+
+
+-- !query
+select interval '1-2' year as y
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.parser.ParseException
+
+Error parsing ' 1-2 year' to interval, invalid value '1-2'(line 1, pos 16)
+
+== SQL ==
+select interval '1-2' year as y
+----------------^^^
{code}
Bug: Replace getClass.getName with getClass.getCanonicalName in CodegenContext.addReferenceObj
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Aug/20 13:33
Affects Version/s: 3.0.0

{code:java}
scala> Array[Byte](1, 2).getClass.getName
res13: String = [B

scala> Array[Byte](1, 2).getClass.getCanonicalName
res14: String = byte[]
{code}

{{[B}} is not a correct java type. We should use {{byte[]}}. Otherwise will hit compile issue:
{noformat}
20:49:54.885 ERROR org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator: 
...
/* 029 */     if (!isNull_2) {
/* 030 */       value_1 = org.apache.spark.sql.catalyst.util.TypeUtils.compareBinary(value_2, (([B) references[0] /* min */)) >= 0 && org.apache.spark.sql.catalyst.util.TypeUtils.compareBinary(value_2, (([B) references[1] /* max */)) <= 0 ).mightContainBinary(value_2);
...

20:49:54.886 WARN org.apache.spark.sql.catalyst.expressions.Predicate: Expr codegen error and falling back to interpreter mode
java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 30, Column 81: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 30, Column 81: Unexpected token "[" in primary
	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306)
{noformat}
Bug: Refactor DataFrameStatPerfSuite to use main method
Status: Open | Priority: Major | Resolution: 
Created: 31/Aug/20 05:30
Affects Version/s: 3.1.0

[https://github.com/apache/spark/blob/be2eca22e9888d2e2ea61b159a215c1f42f9261f/sql/core/src/test/scala/org/apache/spark/sql/DataFrameStatSuite.scala#L502-L534]
Bug: Add an option to reject block migrations when under disk pressure
Status: Open | Priority: Major | Resolution: 
Created: 31/Aug/20 02:57
Affects Version/s: 3.1.0

If we have a very low minimum number of nodes, we may fill up the disks of executors by migrating shuffle files to them. We should add a configuration option to only accept remote shuffle blocks when there is sufficient free disk space.
Bug: Fix double caching in KMeans/BiKMeans
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Aug/20 03:27
Affects Version/s: 3.0.0

In the .mllib side, if the storageLevel of input {{data}} is always ignored and cached twice:
{code:java}
@Since("0.8.0")
def run(data: RDD[Vector]): KMeansModel = {
  val instances = data.map(point => (point, 1.0))
  runWithWeight(instances, None)
}
 {code}
{code:java}
private[spark] def runWithWeight(
    data: RDD[(Vector, Double)],
    instr: Option[Instrumentation]): KMeansModel = {

  // Compute squared norms and cache them.
  val norms = data.map { case (v, _) =>
    Vectors.norm(v, 2.0)
  }

  val zippedData = data.zip(norms).map { case ((v, w), norm) =>
    new VectorWithNorm(v, norm, w)
  }

  if (data.getStorageLevel == StorageLevel.NONE) {
    zippedData.persist(StorageLevel.MEMORY_AND_DISK)
  }
  val model = runAlgorithmWithWeight(zippedData, instr)
  zippedData.unpersist()

  model
} {code}
Bug: Add tests for arrays/maps of nested structs to ReadSchemaSuite to test structs reuse
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Aug/20 21:50
Affects Version/s: 3.0.0

Splitting tests originally posted in [PR|[https://github.com/apache/spark/pull/29352]] for SPARK-32531. The added tests cover cases for maps and arrays of nested structs for different file formats. Eg, [https://github.com/apache/spark/pull/29353] and [https://github.com/apache/spark/pull/29354] add object reuse when reading ORC and Avro files. However, for dynamic data structures like arrays and maps, we do not know just by looking at the schema what the size of the data structure will be so it has to be allocated when reading the data points. The added tests provide coverage so that objects are not accidentally reused when encountering maps and arrays.

AFAIK this is not covered by existing tests.