apache/spark — pull request #2217 by viirya (2014-08-30 19:08:05+00:00)
Title: [SPARK-3327] Make broadcasted value mutable for caching useful information

This PR makes broadcasted value mutable for caching useful information when implementing some algorithms that iteratively run with those information.

Some algorithms can be implemented more efficiently if we can update broadcasted variables and use them in later iterations.

Specifically, we would like to performa operation "A" on each partition of data. Some variables are updated. Then we want to run operation "B" on the data too. "B" operation uses the variables updated by operation "A".

One of the examples is the Liblinear on Spark from Dr. Lin. They discuss the problem in Section IV.D of the paper "Large-scale Logistic Regression and Linear Support Vector Machines Using Spark."

This implementation can make sure that updated broadcasted variables are consistent after parallel data operations (such as map) are done on the broadcasted variables at same node.