apache/incubator-spark — pull request #476 by yinxusen (2014-01-20 09:32:23+00:00)
Title: add data.cache in GradientDescent

Hi, as we moving forward to test [PR-407](https://github.com/apache/incubator-spark/pull/407), we find that if adding cache on data RDD in `GradientDescent`, it will have 5x acceleration easily, as illustrated in the two graphs below.

In GradientDescent-like methods, data entries are keeping still, while the computations flow over them. If there is no data.cache, Spark will read data from HDFS or local disk at every iteration.

Besides, the sampling `data.sample(false, miniBatchFraction, 42+i)` after `data.cache` will have no effect on the performance.

You may wondering on the Figure 2, because there are nearly 13x performance gains at almost all iterations. This is because I just count the real-work computation regardless of the Spark framework.

Experiment parameters (on `LogisticRegressionWithSGD`):

* Number of iterations: 20
* Other configurations are same with PR-407.

![00](https://f.cloud.github.com/assets/2637239/1952950/f8689748-81b2-11e3-9772-d0adb5388f0d.png)

**Figure 1. Total time comparison between cached version and original version.**

![01](https://f.cloud.github.com/assets/2637239/1952949/f868711e-81b2-11e3-93c7-6df055e1c80c.png)

**Figure 2. Time consumption comparison in each iteration.**