SPARK-1518: FileLogger: Fix compile against Hadoop trunk

In Hadoop trunk (currently Hadoop 3.0.0), the deprecated
FSDataOutputStream#sync() method has been removed.  Instead, we should
call FSDataOutputStream#hflush, which does the same thing as the
deprecated method used to do.

Author: Colin McCabe <cmccabe@cloudera.com>

Closes #898 from cmccabe/SPARK-1518 and squashes the following commits:

752b9d7 [Colin McCabe] FileLogger: Fix compile against Hadoop trunk
SPARK-1790: Update EC2 scripts to support r3 instance types

Author: Varakhedi Sujeet <svarakhedi@gopivotal.com>

Closes #960 from sujeetv/ec2-r3 and squashes the following commits:

3cb9fd5 [Varakhedi Sujeet] SPARK-1790: Update EC2 scripts to support r3 instance
Minor: Fix documentation error from apache/spark#946

Author: Ankur Dave <ankurdave@gmail.com>

Closes #970 from ankurdave/SPARK-1991_docfix and squashes the following commits:

6d07343 [Ankur Dave] Minor: Fix documentation error from apache/spark#946
Fix issue in ReplSuite with hadoop-provided profile.

When building the assembly with the maven "hadoop-provided"
profile, the executors were failing to come up because Hadoop classes
were not found in the classpath anymore; so add them explicitly to
the classpath using spark.executor.extraClassPath. This is only
needed for the local-cluster mode, but doesn't affect other tests,
so it's added for all of them to keep the code simpler.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #781 from vanzin/repl-test-fix and squashes the following commits:

4f0a3b0 [Marcelo Vanzin] Fix issue in ReplSuite with hadoop-provided profile.
[SPARK-2029] Bump pom.xml version number of master branch to 1.1.0-SNAPSHOT.

Author: Takuya UESHIN <ueshin@happy-camper.st>

Closes #974 from ueshin/issues/SPARK-2029 and squashes the following commits:

e19e8f4 [Takuya UESHIN] Bump version number to 1.1.0-SNAPSHOT.
SPARK-1677: allow user to disable output dir existence checking

https://issues.apache.org/jira/browse/SPARK-1677

For compatibility with older versions of Spark it would be nice to have an option `spark.hadoop.validateOutputSpecs` (default true)  for the user to disable the output directory existence checking

Author: CodingCat <zhunansjtu@gmail.com>

Closes #947 from CodingCat/SPARK-1677 and squashes the following commits:

7930f83 [CodingCat] miao
c0c0e03 [CodingCat] bug fix and doc update
5318562 [CodingCat] bug fix
13219b5 [CodingCat] allow user to disable output dir existence checking
[SPARK-2036] [SQL] CaseConversionExpression should check if the evaluated value is null.

`CaseConversionExpression` should check if the evaluated value is `null`.

Author: Takuya UESHIN <ueshin@happy-camper.st>

Closes #982 from ueshin/issues/SPARK-2036 and squashes the following commits:

61e1c54 [Takuya UESHIN] Add check if the evaluated value is null.
HOTFIX: Remove generated-mima-excludes file after runing MIMA.

This has been causing some false failures on PR's that don't merge
correctly.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #971 from pwendell/mima and squashes the following commits:

1dc80aa [Patrick Wendell] HOTFIX: Remove generated-mima-excludes file after runing MIMA.
sbt 0.13.X should be using sbt-assembly 0.11.X

https://github.com/sbt/sbt-assembly/blob/master/README.md

Author: Kalpit Shah <shahkalpit84@gmail.com>

Closes #555 from kalpit/upgrade/sbtassembly and squashes the following commits:

1fa7324 [Kalpit Shah] sbt 0.13.X should be using sbt-assembly 0.11.X
Remove compile-scoped junit dependency.

This avoids having junit classes showing up in the assembly jar.
I verified that only test classes in the jtransforms package
use junit.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #794 from vanzin/junit-dep-exclusion and squashes the following commits:

274e1c2 [Marcelo Vanzin] Remove junit from assembly in sbt build also.
ad950be [Marcelo Vanzin] Remove compile-scoped junit dependency.
[SPARK-2041][SQL] Correctly analyze queries where columnName == tableName.

Author: Michael Armbrust <michael@databricks.com>

Closes #985 from marmbrus/tableName and squashes the following commits:

3caaa27 [Michael Armbrust] Correctly analyze queries where columnName == tableName.
Use pluggable clock in DAGSheduler #SPARK-2031

DAGScheduler supports pluggable clock like what TaskSetManager does.

Author: CrazyJvm <crazyjvm@gmail.com>

Closes #976 from CrazyJvm/clock and squashes the following commits:

6779a4c [CrazyJvm] Use pluggable clock in DAGSheduler
[SPARK-2025] Unpersist edges of previous graph in Pregel

Due to a bug introduced by apache/spark#497, Pregel does not unpersist replicated vertices from previous iterations. As a result, they stay cached until memory is full, wasting GC time.

This PR corrects the problem by unpersisting both the edges and the replicated vertices of previous iterations. This is safe because the edges and replicated vertices of the current iteration are cached by the call to `g.cache()` and then materialized by the call to `messages.count()`. Therefore no unmaterialized RDDs depend on `prevG.edges`. I verified that no recomputation occurs by running PageRank with a custom patch to Spark that warns when a partition is recomputed.

Thanks to Tim Weninger for reporting this bug.

Author: Ankur Dave <ankurdave@gmail.com>

Closes #972 from ankurdave/SPARK-2025 and squashes the following commits:

13d5b07 [Ankur Dave] Unpersist edges of previous graph in Pregel
SPARK-2043: ExternalAppendOnlyMap doesn't always find matching keys

The current implementation reads one key with the next hash code as it finishes reading the keys with the current hash code, which may cause it to miss some matches of the next key. This can cause operations like join to give the wrong result when reduce tasks spill to disk and there are hash collisions, as values won't be matched together. This PR fixes it by not reading in that next key, using a peeking iterator instead.

Author: Matei Zaharia <matei@databricks.com>

Closes #986 from mateiz/spark-2043 and squashes the following commits:

0959514 [Matei Zaharia] Added unit test for having many hash collisions
892debb [Matei Zaharia] SPARK-2043: don't read a key with the next hash code in ExternalAppendOnlyMap, instead use a buffered iterator to only read values with the current hash code.
[SPARK-2050][SQL] LIKE, RLIKE and IN in HQL should not be case sensitive.

Author: Michael Armbrust <michael@databricks.com>

Closes #989 from marmbrus/caseSensitiveFuncitons and squashes the following commits:

681de54 [Michael Armbrust] LIKE, RLIKE and IN in HQL should not be case sensitive.
[SPARK-1552] Fix type comparison bug in {map,outerJoin}Vertices

In GraphImpl, mapVertices and outerJoinVertices use a more efficient implementation when the map function conserves vertex attribute types. This is implemented by comparing the ClassTags of the old and new vertex attribute types. However, ClassTags store erased types, so the comparison will return a false positive for types with different type parameters, such as Option[Int] and Option[Double].

This PR resolves the problem by requesting that the compiler generate evidence of equality between the old and new vertex attribute types, and providing a default value for the evidence parameter if the two types are not equal. The methods can then check the value of the evidence parameter to see whether the types are equal.

It also adds a test called "mapVertices changing type with same erased type" that failed before the PR and succeeds now.

Callers of mapVertices and outerJoinVertices can no longer use a wildcard for a graph's VD type. To avoid "Error occurred in an application involving default arguments," they must bind VD to a type parameter, as this PR does for ShortestPaths and LabelPropagation.

Author: Ankur Dave <ankurdave@gmail.com>

Closes #967 from ankurdave/SPARK-1552 and squashes the following commits:

68a4fff [Ankur Dave] Undo conserve naming
7388705 [Ankur Dave] Remove unnecessary ClassTag for VD parameters
a704e5f [Ankur Dave] Use type equality constraint with default argument
29a5ab7 [Ankur Dave] Add failing test
f458c83 [Ankur Dave] Revert "[SPARK-1552] Fix type comparison bug in mapVertices and outerJoinVertices"
16d6af8 [Ankur Dave] [SPARK-1552] Fix type comparison bug in mapVertices and outerJoinVertices
[SPARK-1522]: YARN ClientBase throws a NPE if there is no YARN Application specific CP

The current implementation of ClientBase.getDefaultYarnApplicationClasspath inspects
the MRJobConfig class for the field DEFAULT_YARN_APPLICATION_CLASSPATH when it should
be really looking into YarnConfiguration. If the Application Configuration has no
yarn.application.classpath defined a NPE exception will be thrown.

Public API Changes
===========================

* YARN ClientBase getDefault*ApplicationClasspath returns Option[Seq[String]]

This commit depicts how the `ClientBase` API could change the
`getDefaultYarnApplicationClasspath` and `getDefaultMRApplicationClasspath`
to return a `Option[Seq[String]]` while recovering from `NoSuchFieldException`.

Both methods that return the default application's *classpath*, for both *YARN*
as well as *Map Reduce (MR)*, use reflection and per the Java API
documentation they can throw the following exceptions:

* Class:getField(String name):
    * NoSuchFieldException - if a field with the specified name is not found.
    * NullPointerException - if name is null
    * SecurityException - If a security manager, s, is present and any of
      the following conditions is met:
        1. Invocation of s.checkMemberAccess(this, Member.PUBLIC) denies
        access to the field.
        2. The caller's class loader is not the same as or an ancestor of
        the class loader for the current class and invocation of
        `s.checkPackageAccess()` denies access to the package of this class.

* Field:Object get(Object obj):
    * IllegalAccessException - if this Field object is enforcing Java language access
      control and the underlying field is inaccessible.
    * IllegalArgumentException - if the specified object is not an instance of the class
      or interface declaring the underlying field (or a subclass or implementor thereof).
    * NullPointerException - if the specified object is null and the field
      is an instance field.
    * ExceptionInInitializerError - if the initialization provoked
      by this method fails.

**NOTE**: The above is based on the *Java API for JDK 1.7*

An interesting thing to notice is that the official JDK doesn't mention
the occurrence of the `NoSuchFieldError`. This is completely acceptable
per the JDK spec. The reason is that it is an *Error* and as
described by the Java Language Specification and depicted in
the *Error Class* documentation.

    An `Error` "indicates serious problems that a reasonable
    application should not try to catch."

While

    An `Exception` "indicates conditions that a reasonable
    application might want to catch."

If we actually dig deeper according to the *JVM SE7 Specification*

    "While Loading, Linking, and Initializing, if an error occurs during resolution
    of a symbolic reference, then an instance of
    IncompatibleClassChangeError (or a subclass) must be thrown..."

    "If an attempt by the Java Virtual Machine to resolve a symbolic reference fails
    because an error is thrown that is an instance of LinkageError (or a subclass),
    then subsequent attempts to resolve the reference always fail with the same error
    that was thrown as a result of the initial resolution attempt."

Now `NoSuchFieldError` extends `LinkageError` which in turn is a `IncompatibleClassChangeError`
and according to its documentation, the *LinkageError Class*,

    "indicates that a class has some dependency on another class;
    however, the latter class has incompatibly changed after the
    compilation of the former class."

Why all these is important and how it relates with a couple of lines of
code?

Well, the original approach catches the two most probable problems
you might encounter if you access, using reflection, a field that you
are almost sure that if it exist it will be of _public_ access but you
are not sure it will always be there. Interesting enough the original
implementation addresses one of the _exceptions_ as well as a potential
_linkage error_ but as mentioned neglects a documented _security exception_,
probably due its unlikeliness to occur.

Fact is that if an error _bubbles_ up the Spark YARN Client doesn't handle it
will terminate, in a probably obscure fashion. The current call stack is
as follows.

    Client >> run >> runApp >> ClientBase.setupLaunchEnv >> populateClasspath

In my opinion it is questionable to let an exception escape of this context, the _ClientBase Object_.
In my opinion such _ClientBase Object_ should fail gracefully by handling the potential
_exceptions_ and _linkage error_ while providing enough logging
to let a user know and identify what happened. Yet again, in my opinion
the implementation in this commit handles it in a better, more
resilient, manner than the previous implementation while adding logging
that will help clarify the issues in case of an _exception_.

Additional Changes include:
===========================

* Test Suite for ClientBase added
* Coding Style:
    * [Spark Style Guidelines](https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide)
    * [Scala Official Style Guidelines](http://docs.scala-lang.org/style/)
    * [Scalariform](https://github.com/mdr/scalariform)
* Code refactoring and cleanup per review by andrewor14

Ref.
    "JVM SE7 Specification" http://docs.oracle.com/javase/specs/jvms/se7/html/jvms-5.html#jvms-5.4
    "Java API for JDK 1.7" http://docs.oracle.com/javase/7/docs/api/

[ticket: SPARK-1522] : https://issues.apache.org/jira/browse/SPARK-1522

Author      : berngp
Reviewer    : andrewor14, tgravescs
Testing     : ?