MAHOUT-1636

Started out simplifying driver code and making changes to all drivers to support that. Then ran into the fat job.jar issue of MAHOUT-1636 so created a slimmed down version of the old job.jar by adding excludes to job.xml and changing the name to "dependencies.jar"

The new jar works for spark-itemsimilarity and spark-row-similarity but needs to be tested for the naive bayes drivers. 

The dependencies.jar still contains a lot of stuff from mrlegacy, some is in external projects, like jackson that can be excluded with this mechanism but also a lot of mahout code that is unneeded in this jar. This later case would require some other mechanism than a simple <exclude> clause in the assembly xml file.

I believe the new dependencies.jar is the only thing that needs to be on the classpath when running spark drivers or the spark-shell. I haven't changed this but it is a further refinement we can try.
instead of filtering what is _exluded_ (opt-out) i'd rather determine
what's the minimum opt-in (in the assembly plugin source file). That's
common practice, excludes are tedious, and, most importantly, don't tell
you a bit what exactly you are ending up with

On Mon, Dec 29, 2014 at 12:03 PM, Pat Ferrel <notifications@github.com>
wrote:

> In spark/src/main/assembly/dependencies.xml
> <https://github.com/apache/mahout/pull/69#discussion-diff-22326191>:
>
> >          <exclude>org.apache.hadoop:hadoop-core</exclude>
> > +        <exclude>org.apache.spark:spark-core_${scala.major}</exclude>
> > +        <exclude>org.scala-lang:scala-library</exclude>
> > +        <exclude>jackson-core-asl</exclude>
> > +        <exclude>jackson-mapper-asl</exclude>
> > +        <exclude>xstream</exclude>
> > +        <exclude>lucene-core</exclude>
> > +        <exclude>lucene-analyzers-common</exclude>
> >        </excludes>
> >      </dependencySet>
>
> This is as many as seem safe. Lots inside mrlegacy that could be excluded
> but its all in the same artifact so leaving in unless someone knows how to
> exclude particular partial packages.
>
> Won't change the code to trim things from the classpath in this commit but
> I suspect the dependencies.jar may be all that is needed for spark-shell
> and drivers.
>
> @andrewpalumbo <https://github.com/andrewpalumbo> there's little chance
> this will mess up your drivers so I may push this after some more testing
> on my side.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/mahout/pull/69/files#r22326191>.
>