apache/spark — pull request #2378 by davies (2014-09-13 07:27:13+00:00)
Title: [SPARK-3491] [WIP] [MLlib] [PySpark] use pickle to serialize data in MLlib

Currently, we serialize the data between JVM and Python case by case manually, this cannot scale to support so many APIs in MLlib.

This patch will try to address this problem by serialize the data using pickle protocol, using Pyrolite library to serialize/deserialize in JVM. Pickle protocol can be easily extended to support customized class.

In the first step, it can support Double, DenseVector, SparseVector, DenseMatrix, LabeledPoint, Rating, Tuple2 now, the recommendation module had been refactor to use this new protocol.

Later, I will refactor all others to use this protocol.