apache/spark — pull request #2347 by staple (2014-09-10 16:02:25+00:00)
Title: [SPARK-1484][MLLIB] Warn when running an iterative algorithm on uncached data.

Add warnings to KMeans, GeneralizedLinearAlgorithm, and computeSVD when called with input data that is not cached. KMeans is implemented iteratively, and I believe that GeneralizedLinearAlgorithm’s current optimizers are iterative and its future optimizers are also likely to be iterative. RowMatrix’s computeSVD is iterative against an RDD when run in DistARPACK mode. ALS and DecisionTree are iterative as well, but they implement RDD caching internally so do not require a warning.

I added a warning to GeneralizedLinearAlgorithm rather than inside its optimizers themselves, where the iteration actually occurs, because internally GeneralizedLinearAlgorithm maps its input data to an uncached RDD before passing it to an optimizer. (In other words, the warning would be printed for every GeneralizedLinearAlgorithm run, regardless of whether its input is cached, if the warning were in GradientDescent or other optimizer.) I assume that use of an uncached RDD by GeneralizedLinearAlgorithm is intentional, and that the mapping there (adding label, intercepts and scaling) is a lightweight operation. Arguably a user calling an optimizer such as GradientDescent will be knowledgable enough to cache their data without needing a log warning, so lack of a warning in the optimizers may be ok.

This patch causes all calls to GeneralizedLinearAlgorithm from Python to print a warning, because the implementation in PythonMLLibAPI.trainRegressionModel deserializes the data from python using map(SerDe.deserializeLabeledPoint) to create a deserialized RDD without caching this new RDD. This means that deserialization must occur on every training iteration for RDDs originating in Python. Perhaps the python cache() call from _regression_train_wrapper / _get_unmangled_labeled_point_rdd should be moved to be after deserialization instead of before serialization. There is a similar issue in KMeans.

Some of the documentation examples making use of these iterative algorithms did not cache their training RDDs (while others did). I updated the examples to always cache. I also fixed some (unrelated) minor errors in the documentation examples.