Bug: Some tests failed when cherry-picking https://github.com/apache/spark/pull/43494 back to  branch 3.5
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 15/Jan/24 09:03
Affects Version/s: 4.0.0

When cherry-picking https://github.com/apache/spark/pull/43494 back to branch 3.5 https://github.com/apache/spark/pull/44690, 
I ran into the issue that some tests for Scala 2.12 failed when comparing two maps. It turned out that the function [compareMaps](https://github.com/apache/spark/pull/43494/files#diff-f205431247dd9446f4ce941e5a4620af438c242b9bdff6e7faa7df0194db49acR129) is not so robust for scala 2.12 and scala 2.13.

- scala 2.13

``` scala
Welcome to Scala 2.13.12 (OpenJDK 64-Bit Server VM, Java 17.0.9).
Type in expressions for evaluation. Or try :help.

scala> def compareMaps(lhs: Map[String, Double], rhs: Map[String, Double],
     |                   eps: Double = 0.00000001): Boolean = {
     |     lhs.size == rhs.size &&
     |       lhs.zip(rhs).forall { case ((lName, lAmount), (rName, rAmount)) =>
     |         lName == rName && (lAmount - rAmount).abs < eps
     |       }
     | }
     |   
     | import scala.collection.mutable.HashMap
     | val resources = Map("gpu" -> Map("a" -> 1.0, "b" -> 2.0, "c" -> 3.0, "d"-> 4.0))
     | val mapped = resources.map { case (rName, addressAmounts) =>
     |  rName -> HashMap(addressAmounts.toSeq.sorted: _*) 
     | }
     | 
     | compareMaps(resources("gpu"), mapped("gpu").toMap)
def compareMaps(lhs: Map[String,Double], rhs: Map[String,Double], eps: Double): Boolean
import scala.collection.mutable.HashMap
val resources: scala.collection.immutable.Map[String,scala.collection.immutable.Map[String,Double]] = Map(gpu -> Map(a -> 1.0, b -> 2.0, c -> 3.0, d -> 4.0))
val mapped: scala.collection.immutable.Map[String,scala.collection.mutable.HashMap[String,Double]] = Map(gpu -> HashMap(a -> 1.0, b -> 2.0, c -> 3.0, d -> 4.0))
val res0: Boolean = true
```

- scala 2.12

``` scala
Welcome to Scala 2.12.14 (OpenJDK 64-Bit Server VM, Java 17.0.9).
Type in expressions for evaluation. Or try :help.

scala> def compareMaps(lhs: Map[String, Double], rhs: Map[String, Double],
     |                   eps: Double = 0.00000001): Boolean = {
     |     lhs.size == rhs.size &&
     |       lhs.zip(rhs).forall { case ((lName, lAmount), (rName, rAmount)) =>
     |         lName == rName && (lAmount - rAmount).abs < eps
     |       }
     | }
compareMaps: (lhs: Map[String,Double], rhs: Map[String,Double], eps: Double)Boolean

scala> import scala.collection.mutable.HashMap
import scala.collection.mutable.HashMap

scala> val resources = Map("gpu" -> Map("a" -> 1.0, "b" -> 2.0, "c" -> 3.0, "d"-> 4.0))
resources: scala.collection.immutable.Map[String,scala.collection.immutable.Map[String,Double]] = Map(gpu -> Map(a -> 1.0, b -> 2.0, c -> 3.0, d -> 4.0))

scala> val mapped = resources.map { case (rName, addressAmounts) =>
     |   rName -> HashMap(addressAmounts.toSeq.sorted: _*) 
     | }
mapped: scala.collection.immutable.Map[String,scala.collection.mutable.HashMap[String,Double]] = Map(gpu -> Map(b -> 2.0, d -> 4.0, a -> 1.0, c -> 3.0))

scala> compareMaps(resources("gpu"), mapped("gpu").toMap)
res0: Boolean = false
```

The same code bug got different results for Scala 2.12 and Scala 2.13.  This PR tried to rework compareMaps to make tests pass for both scala 2.12 and scala 2.13