BTW Reynold,

I just wanted to do a quick micro-benchmark with Thyme (
https://github.com/Ichoran/thyme) to see if using a view would actually
help in this case. It looks like the view slows things down quite a bit and
generates more garbage:

scala> val nums = Array.fill[Double](100000)(scala.util.Random.nextDouble)

scala> th.pbench {* nums.zipWithIndex.minBy* { case (n, _) => n } }
Benchmark (2540 calls in 15.48 s)
 * Time:    3.032 ms*   95% CI 2.931 ms - 3.133 ms   (n=20)
  *Garbage: 2.814 ms*   (n=566 sweeps measured)
res4: (Double, Int) = (1.044753784418262E-5,48030)

scala> th.pbench { *nums.view.zipWithIndex.minBy* { case (n, _) => n } }
Benchmark (300 calls in 9.499 s)
  *Time:    9.796 ms *  95% CI 9.229 ms - 10.36 ms   (n=20)
  *Garbage: 21.21 ms*   (n=130 sweeps measured)
res5: (Double, Int) = (1.044753784418262E-5,48030)

That was something I was always curious about.




On Fri, Jul 5, 2013 at 5:13 PM, Reynold Xin <notifications@github.com>wrote:

> In mllib/src/main/scala/spark/ml/clustering/KMeans.scala:
>
> > +                .setInitializationMode(initializationMode)
> > +                .train(data)
> > +  }
> > +
> > +  def train(data: RDD[Array[Double]], k: Int, maxIterations: Int, runs: Int): KMeansModel = {
> > +    train(data, k, maxIterations, runs, K_MEANS_PARALLEL)
> > +  }
> > +
> > +  def train(data: RDD[Array[Double]], k: Int, maxIterations: Int): KMeansModel = {
> > +    train(data, k, maxIterations, 1, K_MEANS_PARALLEL)
> > +  }
> > +
> > +  /**
> > +   * Return the index of the closest point in `centers` to `point`, as well as its distance.
> > +   */
> > +  private[mllib] def findClosest(centers: Array[Array[Double]], point: Array[Double])
>
> Can create a "view" to do that to skip the extra collection - but I think
> the current way is fine as is.
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/mesos/spark/pull/678/files#r5051594>
> .
>