Are we comfortable with pulling in hadoop-common-2.x.x beta/alpha?


On Fri, Oct 11, 2013 at 10:36 AM, Sangjin Lee <notifications@github.com>wrote:

> In scripts/scald.rb:
>
> > @@ -33,10 +34,18 @@
> >      "namespaces" => { "abj" => "com.twitter.ads.batch.job", "s" => "com.twitter.scalding" },
> >      "hadoop_opts" => { "mapred.reduce.tasks" => 20, #be conservative by default
> >                         "mapred.min.split.size" => "2000000000" }, #2 billion bytes!!!
> > -    "depends" => [ "org.apache.hadoop/hadoop-core/0.20.2",
> > +    "depends" => [ "org.apache.hadoop/hadoop-core/1.1.2",
> > +                   "org.apache.hadoop/hadoop-common/0.23.9",
>
> My point was that hadoop-core 1.1.2 and hadoop-common 0.23.9 (0.23.9 is
> considered "new implementation" and pretty similar to 2.x.x) contain a lot
> of same classes with very different implementations, and some of them are
> incompatible.
>
> I'm not exactly sure what "depends" does, but if both hadoop-core 1.1.2
> and hadoop-common 0.23.9 get on the same classpath, bad things will happen
> as you would have two copies of lots of classes. Only one or the other
> should be on the classpath, but not both.
>
> Perhaps there is a deeper issue that led you to include both which needs
> to be solved differently?
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/twitter/scalding/pull/655/files#r6921435>
> .
>