Bug: AvroSerializer can cause java.lang.ClassCastException at run time
Status: In Progress | Priority: Major | Resolution: 
Created: 02/Feb/22 19:02
Affects Version/s: 3.0.0

{{{}AvroSerializer{}}}'s implementation, at least in {{{}newConverter{}}}, was not 100% based on the {{nternalRow}} and {{SpecializedGetters}} interface. It assumes many implementation details of the interface. 

For example, in 

{code}
      case (TimestampType, LONG) => avroType.getLogicalType match {
          // For backward compatibility, if the Avro type is Long and it is not logical type
          // (the `null` case), output the timestamp value as with millisecond precision.
          case null | _: TimestampMillis => (getter, ordinal) =>
            DateTimeUtils.microsToMillis(timestampRebaseFunc(getter.getLong(ordinal)))
          case _: TimestampMicros => (getter, ordinal) =>
            timestampRebaseFunc(getter.getLong(ordinal))
          case other => throw new IncompatibleSchemaException(errorPrefix +
            s"SQL type ${TimestampType.sql} cannot be converted to Avro logical type $other")
        }
{code}

it assumes the {{InternalRow}} instance encodes {{TimestampType}} as {{{}java.lang.Long{}}}. That's true for {{Unsaferow}} but not for {{{}GenericInternalRow{}}}. 

Hence the above code will end up with runtime exceptions when used on an instance of {{{}GenericInternalRow{}}}, which is the case for Python UDF. 

I didn't get time to dig deeper than that. I got the impression that Spark's optimizer(s) will turn a row into a {{UnsafeRow}} and Python UDF doesn't involve the optimizer(s) and hence each row is a {{{}GenericInternalRow{}}}. 

It would be great if someone can correct me or offer a better explanation. 

 

To reproduce the issue, 

{{git checkout master}} and {{git cherry-pick --no-commit}} [this-commit|https://github.com/Zhen-hao/spark/commit/1ffe8e8f35273b2f3529f6c2d004822f480e4c88]

and run the test {{{}org.apache.spark.sql.avro.AvroSerdeSuite{}}}.

 

You will see runtime exceptions like the following one

\\{code}

Serialize DecimalType to Avro BYTES with logical type decimal *** FAILED ***
  java.lang.ClassCastException: class java.math.BigDecimal cannot be cast to class org.apache.spark.sql.types.Decimal (java.math.BigDecimal is in module java.base of loader 'bootstrap'; org.apache.spark.sql.types.Decimal is in unnamed module of loader 'app')
  at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getDecimal(rows.scala:45)
  at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getDecimal$(rows.scala:45)
  at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getDecimal(rows.scala:195)
  at org.apache.spark.sql.avro.AvroSerializer.$anonfun$newConverter$10(AvroSerializer.scala:136)
  at org.apache.spark.sql.avro.AvroSerializer.$anonfun$newConverter$10$adapted(AvroSerializer.scala:135)
  at org.apache.spark.sql.avro.AvroSerializer.$anonfun$newStructConverter$2(AvroSerializer.scala:283)
  at org.apache.spark.sql.avro.AvroSerializer.serialize(AvroSerializer.scala:60)
  at org.apache.spark.sql.avro.AvroSerdeSuite.$anonfun$new$5(AvroSerdeSuite.scala:82)
  at org.apache.spark.sql.avro.AvroSerdeSuite.$anonfun$new$5$adapted(AvroSerdeSuite.scala:67)
  at org.apache.spark.sql.avro.AvroSerdeSuite.$anonfun$withFieldMatchType$2(AvroSerdeSuite.scala:217)
\\{code}