apache/incubator-spark — pull request #512 by tyro89 (2014-01-26 19:54:18+00:00)
Title: Quick fix to pythonRDD max item size problem.

reference JIRA for more context: https://spark-project.atlassian.net/browse/SPARK-1043

This fix was a quick way for me to get things to work again. I'm not sure it's the best way but I didn't think that reimplementing `DataOutputStream.writeUTF` ourself was a good idea and it seemed like a fix like this introduces the least changes in the cross language data contract that we're using.

Essentially I added a custom writeUTF method to PythonRDD and made it replace DataOutputStream.writeUTF so that we can use items with greater size in pyspark. I also modify pyspark MUTF8Deserializer to expect the size of the data blocks to be represented by an integer instead of an unsigned short.

cc: @JoshRosen since you commented on the JIRA :)