av9ash/GitBugs huggingface dataset:
Bug: Port error class data to automation-friendly format
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Jan/24 00:06
Affects Version/s: 4.0.0

As described in SPARK-46810, we have several types of error data captured in our code and documentation.

Unfortunately, a good chunk of this data is in a Markdown table that is not friendly to automation (e.g. to generate documentation, or run tests).

[https://github.com/apache/spark/blob/d1fbc4c7191aafadada1a6f7c217bf89f6cae49f/common/utils/src/main/resources/error/README.md#L121]

We should migrate this error data to an automation-friendly format.
Bug: Improve error message for spark.table when argument type is wrong
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Jan/24 02:07
Affects Version/s: 4.0.0

{code}
>>> spark.table(None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../spark/python/pyspark/sql/session.py", line 1710, in table
    return DataFrame(self._jsparkSession.table(tableName), self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.../spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
  File "/.../spark/python/pyspark/errors/exceptions/captured.py", line 215, in deco
    return f(*a, **kw)
           ^^^^^^^^^^^
  File "/.../spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o27.table.
: java.lang.NullPointerException: Cannot invoke "String.length()" because "s" is null
	at org.antlr.v4.runtime.CharStreams.fromString(CharStreams.java:222)
	at org.antlr.v4.runtime.CharStreams.fromString(CharStreams.java:212)
	at org.apache.spark.sql.catalyst.parser.AbstractParser.parse(parsers.scala:58)
	at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:55)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseMultipartIdentifier(AbstractSqlParser.scala:54)
	at org.apache.spark.sql.DataFrameReader.table(DataFrameReader.scala:681)
	at org.apache.spark.sql.SparkSession.table(SparkSession.scala:619)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
	at py4j.Gateway.invoke(Gateway.java:282)
{code}
Bug: Disable memory profiler for pandas UDFs with iterators
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Jan/24 22:12
Affects Version/s: 3.5.0

When using pandas UDFs with iterators, if users enable the profiling spark conf, a warning indicating non-support should be raised, and profiling should be disabled.

However, currently, after raising the not-supported warning, the memory profiler is still being enabled.

We should fix that.
Bug: Add customizable TaskScheduling param, to avoid randomly choosing executor for tasks, and downscale on low micro-batches activity
Status: Open | Priority: Minor | Resolution: 
Created: 10/Jan/24 19:44
Affects Version/s: 3.4.0

When using dynamicAllocation (but not spark.decommission.enabled=true) with a micro-batches activity, very small tasks are arriving at regular interval, and are processed extremely quickly.
The flow of events that are processed may consume less than 1% of the cpu of the cluster.
But globally, the number of executors stay at a high level (spark.dynamicAllocation.maxExecutors) eventhough they are all 99% of the time IDDLE.

Unfortunatly, in the current code, tasks are assigned randomly to executors, so a constant flow of very small tasks maintain artificially in an "active" status all the executors: 
all executors are receiving tasks from time to time, so strictly speaking, they are never considered as IDDLE during a duration longer than "spark.dynamicAllocation.executorIdleTimeout". 

Therefore, executors are never marked as candidate for decommissioning, and they continue to receive tasks forever, while thoses tasks could easily be assigned to any other executor (chosen not randomly).


The proposition is therefore to add a new configuration property to suppress the random shuffling of assignable offers for task.

see this code [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L773|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L773]


{code:java}
  /**
   * Shuffle offers around to avoid always placing tasks on the same workers.  Exposed to allow
   * overriding in tests, so it can be deterministic.
   */
  protected def shuffleOffers(offers: IndexedSeq[WorkerOffer]): IndexedSeq[WorkerOffer] = {
    Random.shuffle(offers)
  }
{code}

It could be replaced simply by

{code:java}
    val SKIP_RANDOMIZE_WORKER_OFFERS =  ConfigBuilder("spark.task.skipRandomizeWorkerOffers")
      .version("3.6.0")
      .booleanConf
      .createWithDefault(false)
..

    val skipRandomizeWorkerOffers = conf.get(SKIP_RANDOMIZE_WORKER_OFFERS)

..

  protected def shuffleOffers(offers: IndexedSeq[WorkerOffer]): IndexedSeq[WorkerOffer] = {
    if (skipRandomizeWorkerOffers) {
       offers
    } else {
       Random.shuffle(offers)
    }
  }
{code}


Another even better alternative would be to sort offers, using a deterministic sorting order on the Node server name. In that way, spark executors to maintain alive would concentrate on some Node VMS, so that other nodes could be released (scaled-down) by the NodePool, using the "Kubernetes Node autoscaler".

The sort should return first all the nodes whoses names contains "spot", then remaining names.
In that way, spark executors to maintain alive would be preferably the cheapest ones, on "spot" VMs (for AWS, Azure, GCP)
Bug: Add a test regarding to backward compatibility check for Scala StreamingQueryListener
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Jan/24 03:57
Affects Version/s: 3.5.0

When we added onQueryIdle to StreamingQueryListener, we missed to add backward compatibility test(s) into StreamingQueryListenerSuite.

We added the compatibility test in pyspark when we fixed an issue on pyspark specific via https://issues.apache.org/jira/browse/SPARK-45631, but we didn't add the symmetric test in Scala side. Worth noting that the issue SPARK-45631 does not apply to Scala, but worth to have compatibility test so that we do not miss it.
Bug: Refactor `ExecutorFailureTracker#maxNumExecutorFailures` to avoid unnecessary computations when `MAX_EXECUTOR_FAILURES` is configured
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Jan/24 12:15
Affects Version/s: 4.0.0

{code:java}
def maxNumExecutorFailures(sparkConf: SparkConf): Int = {
  val effectiveNumExecutors =
    if (Utils.isStreamingDynamicAllocationEnabled(sparkConf)) {
      sparkConf.get(STREAMING_DYN_ALLOCATION_MAX_EXECUTORS)
    } else if (Utils.isDynamicAllocationEnabled(sparkConf)) {
      sparkConf.get(DYN_ALLOCATION_MAX_EXECUTORS)
    } else {
      sparkConf.get(EXECUTOR_INSTANCES).getOrElse(0)
    }
  // By default, effectiveNumExecutors is Int.MaxValue if dynamic allocation is enabled. We need
  // avoid the integer overflow here.
  val defaultMaxNumExecutorFailures = math.max(3,
    if (effectiveNumExecutors > Int.MaxValue / 2) Int.MaxValue else 2 * effectiveNumExecutors)

  sparkConf.get(MAX_EXECUTOR_FAILURES).getOrElse(defaultMaxNumExecutorFailures)
} {code}
The result of defaultMaxNumExecutorFailures is calculated first, even if {{MAX_EXECUTOR_FAILURES}} is configured now
Bug: RemoveRedundantAliases does not account for SubqueryExpression when removing aliases
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Jan/24 22:45
Affects Version/s: 4.0.0

`RemoveRedundantAliases{{{}`{}}} does not take into account the outer attributes of a `SubqueryExpression` aliases, potentially removing them if it thinks they are redundant.

This can cause scenarios where a subquery expression has conditions like `a#x = a#x` i.e. both the attribute names and the expression ID(s) are the same. This can then lead to conflicting expression ID(s) error.

In `RemoveRedundantAliases`, we have an excluded AttributeSet argument denoting the references for which we should not remove aliases. For a query with a subquery expression, adding the references of this subquery in the excluded set prevents such rewrite from happening.
Bug: Make lxml as an optional testing dependency in test_session
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Jan/24 00:02
Affects Version/s: 4.0.0

{code}
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/__w/spark/spark/python/pyspark/sql/tests/test_session.py", line 22, in <module>
    from lxml import etree
ModuleNotFoundError: No module named 'lxml'
{code}
Bug: Add customizable property spark.dynamicAllocation.lastExecutorIdleTimeout for last remaining executor, defaulting to spark.dynamicAllocation.executorIdleTimeout
Status: Open | Priority: Minor | Resolution: 
Created: 10/Jan/24 20:17
Affects Version/s: 3.5.0

when using dynamicAllocation, the parameter "spark.dynamicAllocation.executorIdleTimeout" is used for any executor, regardless whether it is the last running one, or any other useless one.

However, it might be interresting to preserve the last executor running longer when it is the last remaining one, so that any incoming new task would be immediatly processed faster, instead of waiting for a complete restart of executors that may take >= 30 secondes.

This is particularly frequent in scenario when using Spark Streaming, and when polling for micro-batches. Preserving 1 alive executors help responding faster, while still allowing dynamic alllocation for 2,3..N executors.

In practise, this might change only the following source code lines 

In [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/internal/config/package.scala#L647-L653|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/internal/config/package.scala#L647-L653]
to add
{code:java}
  private[spark] val DYN_ALLOCATION_LAST_EXECUTOR_IDLE_TIMEOUT =
    ConfigBuilder("spark.dynamicAllocation.lastExecutorIdleTimeout")
      .version("3.6.0")
      .timeConf(TimeUnit.SECONDS)
      .checkValue(_ >= 0L, "Last Timeout must be >= 0 (and preferrably >= spark.dynamicAllocation.executorIdleTimeout)")
      .createWithDefault(60)
{code}



In  [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/dynalloc/ExecutorMonitor.scala#L46|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/dynalloc/ExecutorMonitor.scala#L46]

{code:java}
  private val idleTimeoutNs = TimeUnit.SECONDS.toNanos(
    conf.get(DYN_ALLOCATION_EXECUTOR_IDLE_TIMEOUT))
{code}
to add
{code:java}
  private val idleTimeoutNs = TimeUnit.SECONDS.toNanos(
    conf.get(DYN_ALLOCATION_EXECUTOR_IDLE_TIMEOUT))
  private val lastIdleTimeoutNs = TimeUnit.SECONDS.toNanos(
    conf.get(DYN_ALLOCATION_EXECUTOR_LAST_IDLE_TIMEOUT))
{code}


and replace (insert if-condition) in
[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/dynalloc/ExecutorMonitor.scala#L573|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/dynalloc/ExecutorMonitor.scala#L573]

{code:java}
    def updateTimeout(): Unit = {
...
             val timeout = Seq(_cacheTimeout, _shuffleTimeout, idleTimeoutNs).max
...
{code}

to be something like

{code:java}
    def updateTimeout(): Unit = {
...
          val isOnlyOneLastExecutorRemaining = ....
          val currIddleTimeoutNs = if (isOnlyOneLastExecutorRemaining) lastIdleTimeoutNs else idleTimeoutNs
          val timeout = Seq(_cacheTimeout, _shuffleTimeout, currIddleTimeoutNs).max
...
{code}
Bug: Run PyPy 3 and Python 3.10 tests independently
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Jan/24 03:19
Affects Version/s: 4.0.0

https://github.com/apache/spark/actions/runs/7462843546/job/20306241275

Seems like it terminates in the middle because of OOM. we should split
Bug: Address null from Exception.getMessage in Py4J captured exception
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jan/24 10:06
Affects Version/s: 4.0.0

If JVM throws an exception without a message, the message becomes null and returns:

{code}
  File "/.../pyspark/errors/exceptions/captured.py", line 88, in __str__
    desc = desc + "\n\nJVM stacktrace:\n%s" % self._stackTrace
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
{code}
Bug: Use `exists` instead of `filter + nonEmpty` to get `showResourceColumn` in `MasterPage.scala`
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 02/Jan/24 06:55
Affects Version/s: 4.0.0

{code:java}
def render(request: HttpServletRequest): Seq[Node] = {
  val state = getMasterState

  val showResourceColumn = state.workers.filter(_.resourcesInfoUsed.nonEmpty).nonEmpty{code}
we can use `exists` instead of `workers.filter(_.resourcesInfoUsed.nonEmpty).nonEmpty`
Bug: Spark Connect add property in ExecutePlanRequest for job description
Status: Open | Priority: Trivial | Resolution: 
Created: 27/Jan/24 08:04
Affects Version/s: 3.5.0

A representation of the execution request is added as the job description when using Spark Connect.

I would like to add a property to the ExecutePlanRequest that allows client applications to set the string instead of the description being generated by the Spark Connect server.

Having the new property means that we can get things like the client app's stack trace in the job description, which would match the point of the client application.

I have created a small wip PR (I need to figure out how to test it).

The new property could be optional to be backwards compatible and not a breaking change.
Bug: Performance improvement regressed String to Decimal cast
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 24/Nov/21 04:27
Affects Version/s: 3.1.0

A performance improvement to how Spark casts Strings to Decimal in this [PR title|https://issues.apache.org/jira/browse/SPARK-32706], has introduced a regression


{noformat}

scala> :paste 
// Entering paste mode (ctrl-D to finish)

import org.apache.spark.sql.types._
import org.apache.spark.sql.Row
spark.conf.set("spark.sql.legacy.allowNegativeScaleOfDecimal", true)
spark.conf.set("spark.rapids.sql.castStringToDecimal.enabled", true)
spark.conf.set("spark.rapids.sql.castDecimalToString.enabled", true)
val data = Seq(Row("7.836725755512218E38"))
val schema=StructType(Array(StructField("a", StringType, false)))
val df =spark.createDataFrame(spark.sparkContext.parallelize(data), schema)
df.select(col("a").cast(DecimalType(37,-17))).show


// Exiting paste mode, now interpreting.

+--------------------+
|                   a|
+--------------------+
|7.836725755512218...|
+--------------------+

scala> spark.version
res2: String = 3.0.1


scala> :paste
// Entering paste mode (ctrl-D to finish)

import org.apache.spark.sql.types._
import org.apache.spark.sql.Row
spark.conf.set("spark.sql.legacy.allowNegativeScaleOfDecimal", true)
spark.conf.set("spark.rapids.sql.castStringToDecimal.enabled", true)
spark.conf.set("spark.rapids.sql.castDecimalToString.enabled", true)
val data = Seq(Row("7.836725755512218E38"))
val schema=StructType(Array(StructField("a", StringType, false)))
val df =spark.createDataFrame(spark.sparkContext.parallelize(data), schema)
df.select(col("a").cast(DecimalType(37,-17))).show

// Exiting paste mode, now interpreting.

+----+
|   a|
+----+
|null|
+----+

import org.apache.spark.sql.types._
import org.apache.spark.sql.Row
data: Seq[org.apache.spark.sql.Row] = List([7.836725755512218E38])
schema: org.apache.spark.sql.types.StructType = StructType(StructField(a,StringType,false))
df: org.apache.spark.sql.DataFrame = [a: string]

scala> spark.version
res1: String = 3.1.1
{noformat}
Bug: Add Target Encoding to ml.feature
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Nov/21 12:38
Affects Version/s: 3.2.0

Target Encoding is a mechanism of converting categorical features to continues features based on the posterior probability __ calculated from values of the label (target) column.

 

Target Encoding can help to improve accuracy of machine learning algorithms when columns with high cardinality are used as features during training phase.
Bug: Multiple optimizations related to CurrentOrigin.withOrigin
Status: In Progress | Priority: Major | Resolution: 
Created: 23/Nov/21 08:42
Affects Version/s: 4.0.0

This PR / Jira proposes multiple optimizations related to CurrentOrigin.withOrigin, including:

*Avoid CurrentOrigin.withOrigin calls in tree transformation methods when partial function isn't defined*

`CurrentOrigin.withOrigin \{ block }` performs a ThreadLocal.get() call to retrieve the current value of the `origin` thread local so it can be restored after `block` exits.

TreeNode has transform methods which apply PartialFunctions to every node in the tree. By checking `rule.isDefinedAt` before applying a rule we can skip the `CurrentOrigin.withOrigin` call in cases where we know the partial function isn't defined. This avoids the performance overhead associated with thread-local reads and sets.

*Small optimizations to withOrigin itself to save a thread-local* write.

See PR for full details.
Bug: Add ExtendedLevelDBTest and disable LevelDB tests on Apple Silicon
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Nov/21 05:16
Affects Version/s: 3.3.0

Java 17 officially support Apple Silicon.
- JEP 391: macOS/AArch64 Port
- https://bugs.openjdk.java.net/browse/JDK-8251280

Oracle Java, Azul Zulu, and Eclipse Temurin Java 17 supports Apple Silicon natively.
{code}
/Users/dongjoon/.jenv/versions/oracle17/bin/java: Mach-O 64-bit executable arm64
/Users/dongjoon/.jenv/versions/zulu17/bin/java: Mach-O 64-bit executable arm64
/Users/dongjoon/.jenv/versions/temurin17/bin/java: Mach-O 64-bit executable arm64
{code}

Since LevelDBJNI still doesn't support Apple Silicon natively, the test cases fail on M1.
Bug: Add CONTAINS() function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/21 14:02
Affects Version/s: 3.3.0

{{contains()}} is a common convenient function supported by a number of database systems:
 # [https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#contains_substr]
 # [https://docs.snowflake.com/en/_static/apple-touch-icon.png!CONTAINS — Snowflake Documentation|https://docs.snowflake.com/en/sql-reference/functions/contains.html]

Proposed syntax:
{code:java}
contains(haystack, needle)
return type: boolean {code}

It is semantically equivalent to {{haystack like '%needle%'}}
Bug: Spark-on-K8S: Allow disabling of resources.limits.memory in executor pod spec
Status: Open | Priority: Major | Resolution: 
Created: 17/Nov/21 16:42
Affects Version/s: 3.2.0

When spark creates an executor pod on my Kubernetes cluster, it adds the following resources definition:

{{      resources:}}
{{        limits:}}
{{          memory: 896Mi}}
{{        requests:}}
{{          cpu: '4'}}
{{          memory: 896Mi}}
Note that resources.limits.cpu is not set. This is controlled by the spark.kubernetes.driver.limit.cores setting (which we intentionally do not set).

We'd like to be able to omit the resources.limit.memory setting as well to let the spark worker expand its memory as necessary.

However, this isn't possible. The scala code in BasicExecutorFeatureStep.scala is as follows:
{{{}.editOrNewResources(){}}}{{{}.addToRequests("memory", executorMemoryQuantity){}}}{{{}.addToLimits("memory", executorMemoryQuantity){}}}{{{}.addToRequests("cpu", executorCpuQuantity){}}}{{{}.addToLimits(executorResourceQuantities.asJava){}}}{{{}.endResources(){}}}{{{{}}{}}}
 
i.e. it always adds the memory limit, and there's no way to stop it.

Oh - most of our code is in python, so it is not bound by the JVM memory settings,
Bug: Pull out dynamic partition and bucket sort from FileFormatWriter
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Nov/21 09:57
Affects Version/s: 3.3.0

`FileFormatWriter.write` now is used by all V1 write which includes datasource and hive table. However it contains a sort which is based on dynamic partition and bucket columns that can not be seen in plan directly.

V2 write has a better approach that it satisfies the order or even distribution by using rule `V2Writes`.

V1 write should do the similar thing with V2 write.
Bug: Consolidate whole stage and non-whole stage subexpression elimination
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Nov/21 00:49
Affects Version/s: 3.3.0

Currently there are separate subexpression elimination paths for whole stage and non-whole stage codegen. Consolidating these into a single code path would make it simpler to add further enhancements, such as supporting lambda functions  (https://issues.apache.org/jira/browse/SPARK-37466).
Bug: CollectMetrics is executed twice if it is followed by a sort
Status: In Progress | Priority: Major | Resolution: 
Created: 29/Nov/21 13:24
Affects Version/s: 3.3.0

It is best examplified by this new UT in DataFrameCallbackSuite:
{code}
  test("SPARK-37487: get observable metrics with sort by callback") {
    val df = spark.range(100)
      .observe(
        name = "my_event",
        min($"id").as("min_val"),
        max($"id").as("max_val"),
        // Test unresolved alias
        sum($"id"),
        count(when($"id" % 2 === 0, 1)).as("num_even"))
      .observe(
        name = "other_event",
        avg($"id").cast("int").as("avg_val"))
      .sort($"id".desc)

    validateObservedMetrics(df)
  }
{code}

The count and sum aggregate report twice the number of rows:
{code}
[info] - SPARK-37487: get observable metrics with sort by callback *** FAILED *** (169 milliseconds)
[info]   [0,99,9900,100] did not equal [0,99,4950,50] (DataFrameCallbackSuite.scala:342)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info]   at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
[info]   at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
[info]   at org.apache.spark.sql.util.DataFrameCallbackSuite.checkMetrics$1(DataFrameCallbackSuite.scala:342)
[info]   at org.apache.spark.sql.util.DataFrameCallbackSuite.validateObservedMetrics(DataFrameCallbackSuite.scala:350)
[info]   at org.apache.spark.sql.util.DataFrameCallbackSuite.$anonfun$new$21(DataFrameCallbackSuite.scala:324)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
{code}

I could not figure out how this happes. Hopefully the UT can help with debugging