Bug: Number check for InputFileBlockSources is missing for V2 source (BatchScan) ?
Status: Open | Priority: Major | Resolution: 
Created: 10/Nov/23 08:57
Affects Version/s: 3.2.3

When doing a join with the "input_file_name()" function, it will blow up with a

*AnalysisException* if using the v1 data source (FileSourceScan). That's ok.

 

But if we change to use the v2 data source (BatchScan), the expected exception is gone, and the join passes.

Is this number check for InputFileDataSources mssing for V2 data source ? or is it by design ?

 

Repro steps:
{code:java}
scala> spark.range(100).withColumn("const1", lit("from_t1")).write.parquet("/data/tmp/t1")
 
scala> spark.range(100).withColumn("const2", lit("from_t2")).write.parquet("/data/tmp/t2")
 
scala> spark.conf.set("spark.sql.sources.useV1SourceList", "parquet")
 
scala> spark.read.parquet("/data/tmp/t1").join(spark.read.parquet("/data/tmp/t2"), "id", "inner").selectExpr("*", "input_file_name()").show(5, false)
org.apache.spark.sql.AnalysisException: 'input_file_name' does not support more than one sources.; line 1 pos 0;
Project id#376L, const1#377, const2#381, input_file_name() AS input_file_name()#389
+- Project id#376L, const1#377, const2#381
   +- Join Inner, (id#376L = id#380L)
      :- Relation id#376L,const1#377 parquet
      +- Relation id#380L,const2#381 parquet
 
  at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:52)
  at org.apache.spark.sql.execution.datasources.PreReadCheck$.org$apache$spark$sql$execution$datasources$PreReadCheck$$checkNumInputFileBlockSources(rules.scala:476)
  at org.apache.spark.sql.execution.datasources.PreReadCheck$.$anonfun$checkNumInputFileBlockSources$2(rules.scala:472)
  at org.apache.spark.sql.execution.datasources.PreReadCheck$.$anonfun$checkNumInputFileBlockSources$2$adapted(rules.scala:472)
  at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
  at scala.collection.Iterator.foreach(Iterator.scala:943)
  at scala.collection.Iterator.foreach$(Iterator.scala:943)
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
  at scala.collection.IterableLike.foreach(IterableLike.scala:74)
  at scala.collection.IterableLike.foreach$(IterableLike.scala:73)

scala> spark.conf.set("spark.sql.sources.useV1SourceList", "")
 
scala> spark.read.parquet("/data/tmp/t1").join(spark.read.parquet("/data/tmp/t2"), "id", "inner").selectExpr("*", "input_file_name()").show(5, false)
+---+-------+-------+---------------------------------------------------------------------------------------+
|id |const1 |const2 |input_file_name()                                                                      |
+---+-------+-------+---------------------------------------------------------------------------------------+
|91 |from_t1|from_t2|file:///data/tmp/t1/part-00011-a52b9990-4463-447c-9cdf-7a84542de2f7-c000.snappy.parquet|
|92 |from_t1|from_t2|file:///data/tmp/t1/part-00011-a52b9990-4463-447c-9cdf-7a84542de2f7-c000.snappy.parquet|
|93 |from_t1|from_t2|file:///data/tmp/t1/part-00011-a52b9990-4463-447c-9cdf-7a84542de2f7-c000.snappy.parquet|
|94 |from_t1|from_t2|file:///data/tmp/t1/part-00011-a52b9990-4463-447c-9cdf-7a84542de2f7-c000.snappy.parquet|
|95 |from_t1|from_t2|file:///data/tmp/t1/part-00011-a52b9990-4463-447c-9cdf-7a84542de2f7-c000.snappy.parquet|
+---+-------+-------+---------------------------------------------------------------------------------------+
only showing top 5 rows{code}