That dependency is fine for spark-core. The main thing is to document what else users should add to use a newer Hadoop. (E.g. They'd add a newer hadoop-client, but they may also have to add this yarn stuff).

Matei

On Aug 19, 2013, at 12:12 PM, Jey Kottalam <notifications@github.com> wrote:

> In project/SparkBuild.scala:
> 
> > -            "org.apache.hadoop" % "hadoop-core" % HADOOP_VERSION excludeAll(excludeJackson, excludeNetty, excludeAsm),
> > -            "org.apache.hadoop" % "hadoop-client" % HADOOP_VERSION excludeAll(excludeJackson, excludeNetty, excludeAsm)
> > -          )
> > -        }
> > -      } else {
> > -        Seq("org.apache.hadoop" % "hadoop-core" % HADOOP_VERSION excludeAll(excludeJackson, excludeNetty) )
> > -      }),
> > -    unmanagedSourceDirectories in Compile <+= baseDirectory{ _ /
> > -      ( if (HADOOP_YARN && HADOOP_MAJOR_VERSION == "2") {
> > -        "src/hadoop2-yarn/scala"
> > +      if (isYarnMode) {
> > +        // This kludge is needed for 0.23.x
> > +        Seq(
> > +          "org.apache.hadoop" % "hadoop-yarn-api" % hadoopVersion excludeAll(excludeJackson, excludeNetty, excludeAsm),
> > +          "org.apache.hadoop" % "hadoop-yarn-common" % hadoopVersion excludeAll(excludeJackson, excludeNetty, excludeAsm),
> > +          "org.apache.hadoop" % "hadoop-yarn-client" % hadoopVersion excludeAll(excludeJackson, excludeNetty, excludeAsm)
> Nah, because the issue of explicitly linking against the hadoop libs only applies to non-YARN builds. That does bring up another issue though: right now the spark-core artifact will by defaualt be built with dependency on hadoop >= 1.2.1. I'll look into figuring out how to specify a more accurate set of constraints to the POM dependency mechanism
> 
> —
> Reply to this email directly or view it on GitHub.