Bug: Custom Schema with Custom RDD reorders columns when more than 4 added
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 09/Oct/20 19:38
Affects Version/s: 3.0.1

I have a custom RDD written in Java that uses a custom schema.  Everything appears to work fine with using 4 columns, but when i add a 5th column, calling show() fails with 

java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: java.lang.Long is not a valid external type for schema of

here is the schema definition in java:

StructType schema = new StructType() StructType schema = new StructType() .add("recordId", DataTypes.LongType, false) .add("col1", DataTypes.DoubleType, false) .add("col2", DataTypes.DoubleType, false) .add("col3", DataTypes.IntegerType, false) .add("col4", DataTypes.IntegerType, false);

 

Here is the printout of schema.printTreeString();

== Physical Plan ==
*(1) Scan dw [recordId#0L,col1#1,col2#2,col3#3,col4#4] PushedFilters: [], ReadSchema: struct<recordId:bigint,col1:double,col2:double,col3:int,col4:int>

 

I hardcoded a return in my Row object with values matching the schema:

@Override @Override public Object get(int i) \{ switch(i) { case 0: return 0L; case 1: return 1.1911950001644689D; case 2: return 9.100000949955666E9D; case 3: return 476; case 4: return 500; } return 0L; }

 

Here is the output of the show command:

15:30:26.875 ERROR org.apache.spark.executor.Executor - Exception in task 0.0 in stage 0.0 (TID 0)15:30:26.875 ERROR org.apache.spark.executor.Executor - Exception in task 0.0 in stage 0.0 (TID 0)java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: java.lang.Long is not a valid external type for schema of doublevalidateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, col1), DoubleType) AS col1#30validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, recordId), LongType) AS recordId#31Lvalidateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, col2), DoubleType) AS col2#32validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, col3), IntegerType) AS col3#33validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, col4), IntegerType) AS col4#34 at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:215) ~[spark-catalyst_2.12-3.0.1.jar:3.0.1] at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:197) ~[spark-catalyst_2.12-3.0.1.jar:3.0.1] at scala.collection.Iterator$$anon$10.next(Iterator.scala:459) ~[scala-library-2.12.10.jar:?] at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) ~[?:?] at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) ~[spark-sql_2.12-3.0.1.jar:3.0.1] at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729) ~[spark-sql_2.12-3.0.1.jar:3.0.1] at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:340) ~[spark-sql_2.12-3.0.1.jar:3.0.1] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872) ~[spark-core_2.12-3.0.1.jar:3.0.1] at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872) ~[spark-core_2.12-3.0.1.jar:3.0.1] at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.0.1.jar:3.0.1] at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349) ~[spark-core_2.12-3.0.1.jar:3.0.1] at org.apache.spark.rdd.RDD.iterator(RDD.scala:313) ~[spark-core_2.12-3.0.1.jar:3.0.1] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.0.1.jar:3.0.1] at org.apache.spark.scheduler.Task.run(Task.scala:127) ~[spark-core_2.12-3.0.1.jar:3.0.1] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446) ~[spark-core_2.12-3.0.1.jar:3.0.1] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) ~[spark-core_2.12-3.0.1.jar:3.0.1] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449) [spark-core_2.12-3.0.1.jar:3.0.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_265] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_265] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]Caused by: java.lang.RuntimeException: java.lang.Long is not a valid external type for schema of double at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields_0_0$(Unknown Source) ~[?:?] at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) ~[?:?] at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:211) ~[spark-catalyst_2.12-3.0.1.jar:3.0.1] ... 19 more