[SPARK-2850] [mllib] MLlib stats examples + small fixes

Added examples for statistical summarization:
* Scala: StatisticalSummary.scala
** Tests: correlation, MultivariateOnlineSummarizer
* python: statistical_summary.py
** Tests: correlation (since MultivariateOnlineSummarizer has no Python API)

Added examples for random and sampled RDDs:
* Scala: RandomAndSampledRDDs.scala
* python: random_and_sampled_rdds.py
* Both test:
** RandomRDDGenerators.normalRDD, normalVectorRDD
** RDD.sample, takeSample, sampleByKey

Added sc.stop() to all examples.

CorrelationSuite.scala
* Added 1 test for RDDs with only 1 value

RowMatrix.scala
* numCols(): Added check for numRows = 0, with error message.
* computeCovariance(): Added check for numRows <= 1, with error message.

Python SparseVector (pyspark/mllib/linalg.py)
* Added toDense() function

python/run-tests script
* Added stat.py (doc test)

CC: @mengxr @dorx  Main changes were examples to show usage across APIs.
------------------ 原始邮件 ------------------
  发件人: "Xiangrui Meng";<notifications@github.com>;
 发送时间: 2014年8月12日(星期二) 中午12:08
 收件人: "apache/spark"<spark@noreply.github.com>; 
 
 主题: Re: [spark] [SPARK-2850] [mllib] MLlib stats examples + small fixes(#1878)

 

 
In python/pyspark/mllib/linalg.py:
> @@ -160,6 +161,15 @@ def squared_distance(self, other): >                  j += 1 >              return result >   > +    def toDense(self):  
toDense -> toArray (compatible with Scala API)?
 
—
Reply to this email directly or view it on GitHub.