av9ash/GitBugs huggingface dataset:
Bug: Caught Hive MetaException attempting to get partition metadata by filter from Hive
Status: Open | Priority: Critical | Resolution: 
Created: 31/Aug/23 23:01
Affects Version/s: 2.4.0

We are integrating Spark 2.4 with our AWS Glue ETL job.
And we recently realized that a lot of our jobs are failed because of the below error:


{{Exception in User Class: java.lang.RuntimeException : Caught Hive MetaException attempting to get partition metadata by filter from Hive. You can set the Spark configuration setting spark.sql.hive.manageFilesourcePartitions to false to work around this problem, however this will result in degraded performance. Please report a bug: }}{{https://issues.apache.org/jira/browse/SPARK
}}

This error first happens from Aug 30th, and it occurs from time to time. It is gone for several hours and occurs again. During the time that the error occurs, most of the jobs fail, only few succeed. 

And we tried to set {{spark.sql.hive.manageFilesourcePartitions}} to false but it did not work out. Some other issue is coming out.

Can you look into the error and let me know if there is any work around to mitigate the issue? 

Let me know if you need anything from my end.
Bug: Downgrade Hadoop to 3.3.4
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 04/Aug/23 16:07
Affects Version/s: 3.5.0

There is a community report on S3A committer performance regression. Although it's one liner fix, there is no available Hadoop release with that fix at this time.

HADOOP-18757: Bump corePoolSize of HadoopThreadPoolExecutor in s3a committer
Bug: Improve Python UDTF arrow serializer performance
Status: Open | Priority: Major | Resolution: 
Created: 17/Aug/23 19:09
Affects Version/s: 3.5.0

Currently, there is a lot of overhead in the arrow serializer for Python UDTFs. The overhead is largely from converting arrow batches into pandas series and converting UDTF's results back to a pandas dataframe.

We should try directly converting Python object into arrow and vice versa to avoid the expensive pandas conversion. Similar to this converter: [https://github.com/apache/spark/blob/be04ac1ace91f6da34b08a1510e41d3ab6f0377b/python/pyspark/sql/connect/conversion.py#L56]
Bug: Flaky Test with NPE: track allocated resources by taskId
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Aug/23 02:02
Affects Version/s: 4.0.0

{code:java}
- track allocated resources by taskId *** FAILED *** (76 milliseconds)
28782[info]   java.lang.NullPointerException:
28783[info]   at org.apache.spark.executor.CoarseGrainedExecutorBackend.statusUpdate(CoarseGrainedExecutorBackend.scala:267)
28784[info]   at org.apache.spark.executor.CoarseGrainedExecutorBackendSuite.$anonfun$new$22(CoarseGrainedExecutorBackendSuite.scala:347)
28785[info]   at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
28786[info]   at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
28787[info]   at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
28788[info]   at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
28789[info]   at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:69)
28790[info]   at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:155)
28791[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
28792[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
28793[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
28794[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
28795[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
28796[info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
28797[info]   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:227)
28798[info]   at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
28799[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
28800[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
28801[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
28802[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
28803[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:69)
28804[info]   at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
28805[info]   at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
28806[info]   at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:69)
28807[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
28808[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413) {code}
Bug: Spark doesn't create SUCCESS file in Spark 3.3.0+ when partitionOverwriteMode is dynamic
Status: Open | Priority: Major | Resolution: 
Created: 20/Aug/23 12:34
Affects Version/s: 3.3.0

The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0 (tested with 3.4.1 as well)

Code to reproduce the issue

 
{code:java}
scala> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic") 
scala> val DF = Seq(("test1", 123)).toDF("name", "num")
scala> DF.write.option("path", "gs://test_bucket/table").mode("overwrite").partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1") {code}
 

The above code succeeds and creates external Hive table, but {*}there is no SUCCESS file generated{*}.

Adding the content of the bucket after table creation

!image-2023-08-25-13-01-42-137.png|width=500,height=130!

 The same code when running with spark 2.4.0 (with or without external path), generates the SUCCESS file.
{code:java}
scala> DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1"){code}
!image-2023-08-20-18-46-53-342.png|width=465,height=166!
Bug: Fix `RELEASE` file to have the correct information in Docker images
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Aug/23 21:23
Affects Version/s: 2.4.8

{code}
$ docker run -it --rm apache/spark:latest ls -al /opt/spark/RELEASE
-rw-r--r-- 1 spark spark 0 Jun 25 03:13 /opt/spark/RELEASE

$ docker run -it --rm apache/spark:v3.1.3 ls -al /opt/spark/RELEASE | tail -n1
-rw-r--r-- 1 root root 0 Feb 21  2022 /opt/spark/RELEASE
{code}
Bug: Migrate Log4j 2.x in Spark 3.4.1 to Logback
Status: Open | Priority: Major | Resolution: 
Created: 02/Aug/23 22:39
Affects Version/s: 3.4.1

Hi,

We are working on the spark 3.4.1 upgrade from spark 3.1.3, in our logging system, we are using logback framework, it is working with spark 3.1.3 since it is using log4j 1.x. However, when we upgrade spark to 3.4.1, based on the [release notes|https://spark.apache.org/docs/latest/core-migration-guide.html], spark is migrating from log4j 2.x from log4j 1.x, the way we are replacing the log4j with logback is causing build failures in spark master start process.

Error: Unable to initialize main class org.apache.spark.deploy.master.Master
Caused by: java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/Filter

In our current approach, we are using log4j-over-slf4j to replace the log4j-core, it is only applicable to log4j 1.x library. And there is no log4j-over-slf4j for log4j 2.x out there yet. (please correct me if I am wrong). 

I am also curious that why spark choose to use log4j 2.x instead of using SPI, which gives the users less flexibility to choose whatever logger implementation they want to use.

I want to share this issue and see if anyone else has been reported this and if there is any work-around or alternative solutions for it. Any suggestions are appreciated, thanks.
Bug: Add SSL/TLS support for RPC and Shuffle communications
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Aug/23 23:12
Affects Version/s: 4.0.0

Add support for SSL/TLS based communication for Spark RPCs and block transfers - providing an alternative to the existing encryption / authentication implementation documented at [https://spark.apache.org/docs/latest/security.html#spark-rpc-communication-protocol-between-spark-processes]



This is a superset of the functionality discussed in https://issues.apache.org/jira/browse/SPARK-6373
Bug: High On-heap memory usage is detected while doing parquet-file reading with Off-Heap memory mode enabled on spark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Aug/23 11:55
Affects Version/s: 3.4.1

I see the high use of on-heap memory usage while doing the parquet file reading when the off-heap memory mode is enabled. This is caused by the memory-mode for the column vector for the vectorized reader is configured by different flag, and the default value is always set to On-Heap.

Conf to reproduce the issue:

{{spark.memory.offHeap.size 1000000}}
{{spark.memory.offHeap.enabled true}}

Enabling these configurations only will not change the memory mode used for parquet-reading by the vectorized reader to Off-Heap.

 

Proposed PR: https://github.com/apache/spark/pull/42394
Bug: Correlated EXISTS subqueries in join ON condition unsupported and fail with internal error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/23 20:44
Affects Version/s: 3.4.0

They are not handled in decorrelation, and we also don’t have any checks to block them, so these queries have outer references in the query plan leading to internal errors:
{code:java}
CREATE TEMP VIEW x(x1, x2) AS VALUES (0, 1), (1, 2);
CREATE TEMP VIEW y(y1, y2) AS VALUES (0, 2), (0, 3);
CREATE TEMP VIEW z(z1, z2) AS VALUES (0, 2), (0, 3);
select * from x left join y on x1 = y1 and exists (select * from z where z1 = x1)

Error occurred during query planning: 
org.apache.spark.sql.catalyst.plans.logical.Filter cannot be cast to org.apache.spark.sql.execution.SparkPlan {code}
PullupCorrelatedPredicates#apply and RewritePredicateSubquery only handle subqueries in UnaryNode, it seems to assume that they cannot occur elsewhere, like in a join ON condition.

We will need to decide whether to block them properly in analysis (i.e. give a proper error for them), or see if we can add support for them.

Also note, scalar subqueries in the ON condition are unsupported too but return a proper error.
Bug: Enhance raise_error() to exploit the new error framework
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Aug/23 22:55
Affects Version/s: 3.5.0

raise_error() and assert_true() do not presently utilize the new error framework.
We want to generalize raise_error() to take an error class, sqlstate and message parameters as arguments to compose a well-formed error condition.
The existing assert_true(0 and raise_error() versions should return an error class
Bug: Log a warning (or automatically disable) when shuffle tracking is enabled along side another DA supported mechanism
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Aug/23 19:03
Affects Version/s: 4.0.0

Some people enable both shuffle tracking and another mechanism (like migration) and then are confused when their jobs don't scale down.

 

We should at least log a warning here (or automatically disable shuffle tracking?) when it is configured alongside another DA supported mechanism.
Bug: Handle shuffle fetch failures in decommissions
Status: Open | Priority: Major | Resolution: 
Created: 02/Aug/23 09:36
Affects Version/s: 4.0.0

Spark's decommission feature supports migration of shuffle data. However shuffle data fetcher will only look at the location (`BlockManagerId`) when it is initialized. This can lead to shuffle fetch failures when the shuffle read tasks are long.

 

To mitigate this, shuffle data fetchers should be able to look for the updated locations after decommissions, and fetch from there instead.
Bug: Drop legacy Hive-based ORC file format
Status: Open | Priority: Major | Resolution: 
Created: 04/Aug/23 15:18
Affects Version/s: 4.0.0

Currently, Spark allows to use spark.sql.orc.impl=native/hive to switch the ORC FileFormat implementation.

SPARK-23456(2.4) switched the default value of spark.sql.orc.impl from "hive" to "native". and prepared to drop the "hive" implementation in the future.

> ... eventually, Apache Spark will drop old Hive-based ORC code.

The native implementation works well during the whole Spark 3.x period, so it's a good time to consider dropping the "hive" one in Spark 4.0.

Also, we should take care about the backward-compatibility during change.

> BTW, IIRC, there was a different at Hive ORC CHAR implementation before. So, we couldn't remove it for backward-compatibility issues. Since Spark implements many CHAR features, we need to re-verify that {{native}} implementation has all legacy Hive-based ORC features
Bug: Provide context for dataset API errors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Aug/23 15:03
Affects Version/s: 4.0.0

SQL failures already provide nice error context when there is a failure:
{noformat}
org.apache.spark.SparkArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
== SQL(line 1, position 1) ==
a / b
^^^^^

	at org.apache.spark.sql.errors.QueryExecutionErrors$.divideByZeroError(QueryExecutionErrors.scala:201)
	at org.apache.spark.sql.errors.QueryExecutionErrors.divideByZeroError(QueryExecutionErrors.scala)
...
{noformat}
We could add a similar user friendly error context to Dataset APIs.

E.g. consider the following Spark app SimpleApp.scala:
{noformat}
   1  import org.apache.spark.sql.SparkSession
   2  import org.apache.spark.sql.functions._
   3
   4  object SimpleApp {
   5    def main(args: Array[String]) {
   6      val spark = SparkSession.builder.appName("Simple Application").config("spark.sql.ansi.enabled", true).getOrCreate()
   7      import spark.implicits._
   8
   9      val c = col("a") / col("b")
  10
  11      Seq((1, 0)).toDF("a", "b").select(c).show()
  12
  13      spark.stop()
  14    }
  15  }
{noformat}
then the error context could be:
{noformat}
Exception in thread "main" org.apache.spark.SparkArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
== Dataset ==
"div" was called from SimpleApp$.main(SimpleApp.scala:9)

	at org.apache.spark.sql.errors.QueryExecutionErrors$.divideByZeroError(QueryExecutionErrors.scala:201)
	at org.apache.spark.sql.catalyst.expressions.DivModLike.eval(arithmetic.scala:672
...
{noformat}
Bug: Migrate test_reset_index assert_eq to use assertDataFrameEqual
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Aug/23 22:20
Affects Version/s: 3.5.0

Migrate assert_eq to assertDataFrameEqual in this file: [python/pyspark/pandas/tests/indexes/test_reset_index.py|https://github.com/apache/spark/blob/42e5daddf3ba16ff7d08e82e51cd8924cc56e180/python/pyspark/pandas/tests/indexes/test_reset_index.py#L46]
Bug: SPIP: Incremental Stats Collection
Status: Open | Priority: Major | Resolution: 
Created: 15/Aug/23 15:44
Affects Version/s: 3.5.0

Spark's Cost Based Optimizer is dependent on the table and column statistics.

After every execution of DML query, table and column stats are invalidated if auto update of stats collection is not turned on. To keep stats updated we need to run `ANALYZE TABLE COMPUTE STATISTICS` command which is very expensive. It is not feasible to run this command after every DML query.

Instead, we can incrementally update the stats during each DML query run itself. This way our table and column stats would be fresh at all the time and CBO benefits can be applied. Initially, we can only update table level stats and gradually start updating column level stats as well.

*Pros:*

1. Optimize queries over table which is updated frequently.
2. Saves Compute cycles by removing dependency over `ANALYZE TABLE COMPUTE STATISTICS` for updating stats.

[SPIP Document |https://docs.google.com/document/d/1CNPWg_L1fxfB4d2m6xfizRyYRoWS2uPCwTKzhL2fwaQ/edit?usp=sharing]
Bug: Spark wrongly map the BOOLEAN Type to BIT(1) in Snowflake
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Aug/23 09:20
Affects Version/s: 3.4.1

In Snowflake the Boolean type is represented by the Boolean data type ([https://docs.snowflake.com/en/sql-reference/data-types-logical]), but Spark rely on the default JdbcDialect to generate the mapping which maps _Boolean_ to _BIT(1)_

This should be probably handled by a dialect specific to Snowflake.
Bug: Improve Spark Dynamic Allocation
Status: Open | Priority: Major | Resolution: 
Created: 24/Aug/23 19:01
Affects Version/s: 4.0.0

For Spark 4 we should aim to improve Spark's dynamic allocation. Some potential ideas here includes the following:
 * Plug-gable DEA algorithms
 * How to reduce wastage on the RM side? Sometimes the driver asks for some units of resources. But when RM provisions them, the driver cancels it. 
 * Support for "warm" executor pools which are not tied to a particular driver but start and wait for a driver to connect to them to "claim" them.
 * More explicit Cost Vs AppRunTime confiugration: A good DEA algo should allow the developer to choose between cost and runtime. Sometimes developers might be ok to pay higher costs for faster execution.
 * Use previous run information to inform future runs
 * Better selection of executors to be scaled down
Bug: NoClassDefFoundError when using Hive UDF
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Aug/23 12:18
Affects Version/s: 3.5.0

How to reproduce:
{noformat}
spark-sql (default)> add jar /Users/yumwang/Downloads/HiveUDFs-1.0-SNAPSHOT.jar;
Time taken: 0.413 seconds
spark-sql (default)> CREATE TEMPORARY FUNCTION long_to_ip as 'net.petrabarus.hiveudfs.LongToIP';
Time taken: 0.038 seconds
spark-sql (default)> SELECT long_to_ip(2130706433L) FROM range(10);
23/08/08 20:17:58 ERROR SparkSQLDriver: Failed in [SELECT long_to_ip(2130706433L) FROM range(10)]
java.lang.NoClassDefFoundError: org/codehaus/jackson/map/type/TypeFactory
	at org.apache.hadoop.hive.ql.udf.UDFJson.<clinit>(UDFJson.java:64)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
...
{noformat}
Bug: Resolve testing timeout issue from Spark Connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Aug/23 06:43
Affects Version/s: 4.0.0

DiffFramesParitySetItemSeriesTests.test_series_iloc_setitem is failing on Spark Connect due to unexpected timeout issue: https://github.com/itholic/spark/actions/runs/5850534247/job/15860127608
Bug: Refactor Spark Connect Docs to incorporate Scala setup
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Aug/23 09:29
Affects Version/s: 3.5.0

The current Spark Connect [overview|https://spark.apache.org/docs/latest/spark-connect-overview.html] does not include instructions to setup the Scala REPL as well using the Scala client in applications.
Bug: Relax constraint for columnar shuffle check in AQE
Status: Open | Priority: Major | Resolution: 
Created: 03/Aug/23 19:27
Affects Version/s: 3.4.1

Currently in AQE, after evaluating the columnar rules, Spark will check if the top operator of the stage is still a shuffle operator, and throw exception if it doesn't.

{code}
        val optimized = e.withNewChildren(Seq(optimizeQueryStage(e.child, isFinalStage = false)))
        val newPlan = applyPhysicalRules(
          optimized,
          postStageCreationRules(outputsColumnar = plan.supportsColumnar),
          Some((planChangeLogger, "AQE Post Stage Creation")))
        if (e.isInstanceOf[ShuffleExchangeLike]) {
          if (!newPlan.isInstanceOf[ShuffleExchangeLike]) {
            throw SparkException.internalError(
              "Custom columnar rules cannot transform shuffle node to something else.")
          }
{code}

However, once a shuffle operator is transformed into a custom columnar shuffle operator, the {{supportsColumnar}} of the new shuffle operator will return true, and therefore the columnar rules will insert {{ColumnarToRow}} on top of it. This means the {{newPlan}} is likely no longer a {{ShuffleExchangeLike}} but a {{ColumnarToRow}}, and exception will be thrown, even though the use case is valid.

This JIRA proposes to relax the check by allowing the above case.
Bug: Avro connector: convert a union of a single primitive type to a StructType
Status: Open | Priority: Major | Resolution: 
Created: 22/Aug/23 22:19
Affects Version/s: 3.4.1

Spark Avro data source schema converter currently converts union with a single primitive type to a Spark primitive type instead of a StructType.

While for more complex union types that consists of multiple primitive types, the schema converter translate them into StructTypes.

For example, 
import scala.collection.JavaConverters._
import org.apache.avro._
import org.apache.spark.sql.avro._

// ["string", "null"]
SchemaConverters.toSqlType(
  Schema.createUnion(Seq(Schema.create(Schema.Type.STRING), Schema.create(Schema.Type.NULL)).asJava)
).dataType

// ["string", "int", "null"]
SchemaConverters.toSqlType(
  Schema.createUnion(Seq(Schema.create(Schema.Type.STRING), Schema.create(Schema.Type.INT), Schema.create(Schema.Type.NULL)).asJava)
).dataType
The first one would return StringType, the second would return StructType(StringType, IntegerType).
 
We hope to add a new configuration to control the conversion behavior. The default behavior would still be the same. When the config is altered, a union with single primitive type would be translated into StructType.
Bug: Push filters through intersect
Status: Open | Priority: Major | Resolution: 
Created: 15/Aug/23 04:42
Affects Version/s: 3.4.1

For following SQL
{code:sql}
select a from (select a from tl intersect select x from tr) where a > 123 {code}
The physical plan is
{code:bash}
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- HashAggregate(keys=[a#8L], functions=[])
   +- Exchange hashpartitioning(a#8L, 200), ENSURE_REQUIREMENTS, [plan_id=133]
      +- HashAggregate(keys=[a#8L], functions=[])
         +- BroadcastHashJoin [coalesce(a#8L, 0), isnull(a#8L)], [coalesce(x#24L, 0), isnull(x#24L)], LeftSemi, BuildRight, false
            :- Filter (isnotnull(a#8L) AND (a#8L > 123))
            :  +- FileScan json [a#8L] ...
            +- BroadcastExchange HashedRelationBroadcastMode(List(coalesce(input[0, bigint, true], 0), isnull(input[0, bigint, true])),false), [plan_id=129]
               +- FileScan json [x#24L] ... {code}
We can find the filter {color:#ff8b00}_a > 123_{color} is not pushed to right table.

 

 

Further more, for following SQL
{code:sql}
select a from (select a from tl intersect select x from tr) join trr on a = y{code}
The physical plan is
{code:bash}
*(3) Project [a#8L]
+- *(3) BroadcastHashJoin [a#8L], [y#114L], Inner, BuildRight, false
   :- *(3) HashAggregate(keys=[a#8L], functions=[])
   :  +- Exchange hashpartitioning(a#8L, 200), ENSURE_REQUIREMENTS, [plan_id=506]
   :     +- *(1) HashAggregate(keys=[a#8L], functions=[])
   :        +- *(1) BroadcastHashJoin [coalesce(a#8L, 0), isnull(a#8L)], [coalesce(x#24L, 0), isnull(x#24L)], LeftSemi, BuildRight, false
   :           :- *(1) Filter isnotnull(a#8L)
   :           :  +- FileScan json [a#8L] ...
   :           +- BroadcastExchange HashedRelationBroadcastMode(List(coalesce(input[0, bigint, true], 0), isnull(input[0, bigint, true])),false), [plan_id=490]
   :              +- FileScan json [x#24L] ...
   +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, false]),false), [plan_id=512]
      +- *(2) Filter isnotnull(y#114L)
         +- FileScan json [y#114L] ...{code}
There should be a filter _{color:#ff8b00}isnotnull( x ){color}_ for table tr, while it's not pushed down.