MAHOUT-1636

Started out simplifying driver code and making changes to all drivers to support that. Then ran into the fat job.jar issue of MAHOUT-1636 so created a slimmed down version of the old job.jar by adding excludes to job.xml and changing the name to "dependencies.jar"

The new jar works for spark-itemsimilarity and spark-row-similarity but needs to be tested for the naive bayes drivers. 

The dependencies.jar still contains a lot of stuff from mrlegacy, some is in external projects, like jackson that can be excluded with this mechanism but also a lot of mahout code that is unneeded in this jar. This later case would require some other mechanism than a simple <exclude> clause in the assembly xml file.

I believe the new dependencies.jar is the only thing that needs to be on the classpath when running spark drivers or the spark-shell. I haven't changed this but it is a further refinement we can try.
Sounds good - push whenever u think is good- I don't think there will be any problems...   the only difference that I can think of from the basic structure of the  item-similarity driver is that the NB driver makea calls to a SparkNB object from the spark module which overrides the math-scala implementation and calls spark aggregateByKey(...). So as long as that's available and math-scala is available there shouldn't be any issues. 



Sent from my Verizon Wireless 4G LTE smartphone

<div>-------- Original message --------</div><div>From: Pat Ferrel <notifications@github.com> </div><div>Date:12/29/2014  3:03 PM  (GMT-05:00) </div><div>To: apache/mahout <mahout@noreply.github.com> </div><div>Cc: Andrew Palumbo <ap.dev@outlook.com> </div><div>Subject: Re: [mahout] MAHOUT-1636 (#69) </div><div>
</div>In spark/src/main/assembly/dependencies.xml:

>          <exclude>org.apache.hadoop:hadoop-core</exclude>
> +        <exclude>org.apache.spark:spark-core_${scala.major}</exclude>
> +        <exclude>org.scala-lang:scala-library</exclude>
> +        <exclude>jackson-core-asl</exclude>
> +        <exclude>jackson-mapper-asl</exclude>
> +        <exclude>xstream</exclude>
> +        <exclude>lucene-core</exclude>
> +        <exclude>lucene-analyzers-common</exclude>
>        </excludes>
>      </dependencySet>
This is as many as seem safe. Lots inside mrlegacy that could be excluded but its all in the same artifact so leaving in unless someone knows how to exclude particular partial packages.

Won't change the code to trim things from the classpath in this commit but I suspect the dependencies.jar may be all that is needed for spark-shell and drivers.

@andrewpalumbo there's little chance this will mess up your drivers so I may push this after some more testing on my side.

—
Reply to this email directly or view it on GitHub.