Bug: Plan canonicalization test serializes/deserializes class that is not serializable
Status: Open | Priority: Minor | Resolution: 
Created: 20/Feb/24 18:15
Affects Version/s: 4.0.0

The test
{code:java}
test("SPARK-23731 plans should be canonicalizable after being (de)serialized"){code}
serializes and deserializes 
{code:java}
FileSourceScanExec{code}
which is not actually serializable. In particular, FileSourceScanExec.relation is not serializable.
The test still passes though.

The test below derived from the above shows the issue - 
{code:java}
test("verify FileSourceScanExec (de)serialize") {
  withSQLConf(SQLConf.USE_V1_SOURCE_LIST.key -> "parquet") {
    withTempPath { path =>
      spark.range(1).write.parquet(path.getAbsolutePath)
      val df = spark.read.parquet(path.getAbsolutePath)
      val fileSourceScanExec =
        df.queryExecution.sparkPlan.collectFirst { case p:      FileSourceScanExec => p }.get
      val serializer = SparkEnv.get.serializer.newInstance()
      val relation = serializer.serialize(fileSourceScanExec.relation)
      assert(relation != null)
      val deserialized =
      serializer.deserialize[FileSourceScanExec(serializer.serialize(fileSourceScanExec))
      assert(deserialized.relation != null)
    }
  }
}{code}
 
The test fails with -
{code:java}
(file:/private/var/folders/bz/gg_fqnmj4c17j2c7mdn8ps1m0000gn/T/spark-d534d738-64f1-4eaa-9d9e-8c33374b60f1))
    - field (class: org.apache.spark.sql.execution.datasources.HadoopFsRelation, name: location, type: interface org.apache.spark.sql.execution.datasources.FileIndex)
    - object (class org.apache.spark.sql.execution.datasources.HadoopFsRelation, parquet)
java.io.NotSerializableException: org.apache.spark.sql.execution.datasources.InMemoryFileIndex
Serialization stack:
    - object not serializable (class: org.apache.spark.sql.execution.datasources.InMemoryFileIndex, value: org.apache.spark.sql.execution.datasources.InMemoryFileIndex(file:/private/var/folders/bz/gg_fqnmj4c17j2c7mdn8ps1m0000gn/T/spark-d534d738-64f1-4eaa-9d9e-8c33374b60f1))
    - field (class: org.apache.spark.sql.execution.datasources.HadoopFsRelation, name: location, type: interface org.apache.spark.sql.execution.datasources.FileIndex)
    - object (class org.apache.spark.sql.execution.datasources.HadoopFsRelation, parquet)
    at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:41)
    at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:49)
    at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:115)
    at org.apache.spark.sql.execution.SparkPlanSuite.$anonfun$new$11(SparkPlanSuite.scala:54)
    at org.apache.spark.sql.execution.SparkPlanSuite.$anonfun$new$11$adapted(SparkPlanSuite.scala:48)
    at org.apache.spark.sql.catalyst.plans.SQLHelper.withTempPath(SQLHelper.scala:69)
    at org.apache.spark.sql.catalyst.plans.SQLHelper.withTempPath$(SQLHelper.scala:66)
    at org.apache.spark.sql.QueryTest.withTempPath(QueryTest.scala:33)
    at org.apache.spark.sql.execution.SparkPlanSuite.$anonfun$new$10(SparkPlanSuite.scala:48)
    at org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf(SQLHelper.scala:54)
    at org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf$(SQLHelper.scala:38)
    at org.apache.spark.sql.execution.SparkPlanSuite.org$apache$spark$sql$test$SQLTestUtilsBase$$super$withSQLConf(SparkPlanSuite.scala:32)
    at org.apache.spark.sql.test.SQLTestUtilsBase.withSQLConf(SQLTestUtils.scala:266)
    at org.apache.spark.sql.test.SQLTestUtilsBase.withSQLConf$(SQLTestUtils.scala:264)
    at org.apache.spark.sql.execution.SparkPlanSuite.withSQLConf(SparkPlanSuite.scala:32)
    at org.apache.spark.sql.execution.SparkPlanSuite.$anonfun$new$9(SparkPlanSuite.scala:48) {code}