Shopify/spark — pull request #7 by tyro89 (2014-01-26 20:43:19+00:00)
Title: Fix pyspark item size limits

@hornairs @honkfestival @bouk @patrickdonovan @udnay for review please.

This is so that we can unblock @bouk asap with his apriori stuff. It's a little hacky but its the best I could come up with without having to change `writeUTF` to some other UTF encoding algorithm or implementing the `writeUTF` algorithm myself. (both of those things seemed really scary to me so I wanted to avoid it and make sure I don't break the current contract between scala and python).

I have this code open in a pr with mainline as well but it might not be the way the maintainer wish to go with the code and it usually takes a while for them to accept a pr anyhow since they are really careful and making sure people have time to take a look at things :) Therefore I suggest we get this into our build asap so that we can keep the spark momentum going! :)