apache/spark — pull request #3795 by JoshRosen (2014-12-25 05:42:38+00:00)
Title: [SPARK-3847] Raise exception when hashing Java enums

This patch modifies Spark to throw exceptions when attempting to hash-partition Java Enums. Java Enums' hashCodes are machine/JVM-dependent, so it is unsafe to compare enum hashCodes generated in different JVMs; this means that we can't partition RDDs with enum keys.

The fix here is based on a similar fix which prevents Java arrays from being used as keys when repartitioning (https://github.com/mesos/spark/pull/348): at RDD definition time, use reflection to check whether the key class is an enumeration and throw a warning if a HashPartitioner is being used. We do not throw a warning if some other partitioner is used because, in principle, that partitioner could have custom logic for properly handling enums (e.g. by calling `toString()` and using the string's hashcode). There are some corner-cases that this will miss (such as enums that are nested in other objects, like pairs of enums), but I think that this may be the best that we can do without adding a per-record performance overhead or marking changes to the shuffle code.

In case we have to add similar error checks in the future, I've factored the logic into a helper function in the `Partitioner` object. I also improved the warning messages to reference the relevant JIRA issues.