apache/spark — pull request #3098 by debasish83 (2014-11-05 01:38:16+00:00)
Title: [MLLIB] SPARK-4231: Add RankingMetrics to examples.MovieLensALS

@mengxr @srowen

To validate ALS enhancements as proposed in https://issues.apache.org/jira/browse/SPARK-2426, RMSE along with the RankingMetrics measures are important to look at.

This PR adds a flag --validateProducts to examples.MovieLensALS.

Default validateProducts is false and we compute RMSE and MAP for test set related to product recommendation.

./bin/spark-submit --master spark://tusca09lmlvt00c.uswin.ad.vzwcorp.com:7077 --jars /Users/v606014/.m2/repository/com/github/scopt/scopt_2.10/3.2.0/scopt_2.10-3.2.0.jar --total-executor-cores 4 --executor-memory 4g --driver-memory 1g --class org.apache.spark.examples.mllib.MovieLensALS ./examples/target/spark-examples_2.10-1.2.0-SNAPSHOT.jar --kryo --lambda 0.065 hdfs://localhost:8020/sandbox/movielens/
2014-11-04 17:15:24.262 java[4568:1903] Unable to load realm mapping info from SCDynamicStore
14/11/04 17:15:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Got 1000209 ratings from 6040 users on 3706 movies.
Training: 799926, test: 200283.
Test RMSE = 0.8965005871008247 MAP = 7.438473265235346.

--validateProducts will validate user recommendation for each product

./bin/spark-submit --master spark://tusca09lmlvt00c.uswin.ad.vzwcorp.com:7077 --jars /Users/v606014/.m2/repository/com/github/scopt/scopt_2.10/3.2.0/scopt_2.10-3.2.0.jar --total-executor-cores 4 --executor-memory 4g --driver-memory 1g --class org.apache.spark.examples.mllib.MovieLensALS ./examples/target/spark-examples_2.10-1.2.0-SNAPSHOT.jar --kryo --lambda 0.065 --validateProducts hdfs://localhost:8020/sandbox/movielens/
2014-11-04 17:16:18.652 java[4635:1903] Unable to load realm mapping info from SCDynamicStore
14/11/04 17:16:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Got 1000209 ratings from 6040 users on 3706 movies.
Training: 800014, test: 200195.
Test RMSE = 0.8986539583457682 MAP = 12.243775391575324.

Sean,

Are we looking at the right numbers here ? MAP for Movielens dataset is around 12.243. You did similar experiments for oryx/myrrix before...

We can perhaps make the test set generation more intelligent but I went with random sampling for now since I was looking at MAP measure..

For prec@k I am not sure what's the right k number to choose at...I do a sweep over k to choose sweet spot internally.