SPARK-1518: FileLogger: Fix compile against Hadoop trunk

In Hadoop trunk (currently Hadoop 3.0.0), the deprecated
FSDataOutputStream#sync() method has been removed.  Instead, we should
call FSDataOutputStream#hflush, which does the same thing as the
deprecated method used to do.

Author: Colin McCabe <cmccabe@cloudera.com>

Closes #898 from cmccabe/SPARK-1518 and squashes the following commits:

752b9d7 [Colin McCabe] FileLogger: Fix compile against Hadoop trunk
(cherry picked from commit 1765c8d0ddf6bb5bc3c21f994456eba04c581de4)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
SPARK-1790: Update EC2 scripts to support r3 instance types

Author: Varakhedi Sujeet <svarakhedi@gopivotal.com>

Closes #960 from sujeetv/ec2-r3 and squashes the following commits:

3cb9fd5 [Varakhedi Sujeet] SPARK-1790: Update EC2 scripts to support r3 instance
(cherry picked from commit 11ded3f66f178e4d8d2b23491dd5e0ea23bcf719)

Conflicts:

	ec2/spark_ec2.py
[SPARK-2030] Bump SparkBuild.scala version number of branch-1.0 to 1.0.1-SNAPSHOT.

Author: Takuya UESHIN <ueshin@happy-camper.st>

Closes #975 from ueshin/issues/SPARK-2030 and squashes the following commits:

aae1044 [Takuya UESHIN] Bump version number to 1.0.1-SNAPSHOT.
SPARK-1677: allow user to disable output dir existence checking

https://issues.apache.org/jira/browse/SPARK-1677

For compatibility with older versions of Spark it would be nice to have an option `spark.hadoop.validateOutputSpecs` (default true)  for the user to disable the output directory existence checking

Author: CodingCat <zhunansjtu@gmail.com>

Closes #947 from CodingCat/SPARK-1677 and squashes the following commits:

7930f83 [CodingCat] miao
c0c0e03 [CodingCat] bug fix and doc update
5318562 [CodingCat] bug fix
13219b5 [CodingCat] allow user to disable output dir existence checking
(cherry picked from commit 89cdbb087cb2f0d03be2dd77440300c6bd61c792)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
[SPARK-2036] [SQL] CaseConversionExpression should check if the evaluated value is null.

`CaseConversionExpression` should check if the evaluated value is `null`.

Author: Takuya UESHIN <ueshin@happy-camper.st>

Closes #982 from ueshin/issues/SPARK-2036 and squashes the following commits:

61e1c54 [Takuya UESHIN] Add check if the evaluated value is null.

(cherry picked from commit e4c11eef2f64df0b6a432f40b669486d91ca6352)
Signed-off-by: Michael Armbrust <michael@databricks.com>
[SPARK-2041][SQL] Correctly analyze queries where columnName == tableName.

Author: Michael Armbrust <michael@databricks.com>

Closes #985 from marmbrus/tableName and squashes the following commits:

3caaa27 [Michael Armbrust] Correctly analyze queries where columnName == tableName.

(cherry picked from commit c7a183b2c2bca13565496495b4ae3a3a9f63f9ab)
Signed-off-by: Reynold Xin <rxin@apache.org>
[SPARK-2025] Unpersist edges of previous graph in Pregel

Due to a bug introduced by apache/spark#497, Pregel does not unpersist replicated vertices from previous iterations. As a result, they stay cached until memory is full, wasting GC time.

This PR corrects the problem by unpersisting both the edges and the replicated vertices of previous iterations. This is safe because the edges and replicated vertices of the current iteration are cached by the call to `g.cache()` and then materialized by the call to `messages.count()`. Therefore no unmaterialized RDDs depend on `prevG.edges`. I verified that no recomputation occurs by running PageRank with a custom patch to Spark that warns when a partition is recomputed.

Thanks to Tim Weninger for reporting this bug.

Author: Ankur Dave <ankurdave@gmail.com>

Closes #972 from ankurdave/SPARK-2025 and squashes the following commits:

13d5b07 [Ankur Dave] Unpersist edges of previous graph in Pregel

(cherry picked from commit 9bad0b73722fb359f14db864e69aa7efde3588c5)
Signed-off-by: Reynold Xin <rxin@apache.org>
SPARK-2043: ExternalAppendOnlyMap doesn't always find matching keys

The current implementation reads one key with the next hash code as it finishes reading the keys with the current hash code, which may cause it to miss some matches of the next key. This can cause operations like join to give the wrong result when reduce tasks spill to disk and there are hash collisions, as values won't be matched together. This PR fixes it by not reading in that next key, using a peeking iterator instead.

Author: Matei Zaharia <matei@databricks.com>

Closes #986 from mateiz/spark-2043 and squashes the following commits:

0959514 [Matei Zaharia] Added unit test for having many hash collisions
892debb [Matei Zaharia] SPARK-2043: don't read a key with the next hash code in ExternalAppendOnlyMap, instead use a buffered iterator to only read values with the current hash code.

(cherry picked from commit b45c13e7d798f97b92f1a6329528191b8d779c4f)
Signed-off-by: Matei Zaharia <matei@databricks.com>
[SPARK-2050][SQL] LIKE, RLIKE and IN in HQL should not be case sensitive.

Author: Michael Armbrust <michael@databricks.com>

Closes #989 from marmbrus/caseSensitiveFuncitons and squashes the following commits:

681de54 [Michael Armbrust] LIKE, RLIKE and IN in HQL should not be case sensitive.

(cherry picked from commit 41db44c428a10f4453462d002d226798bb8fbdda)
Signed-off-by: Reynold Xin <rxin@apache.org>
SPARK-2058: use spark-defaults.conf from SPARK_CONF_DIR when defined
SPARK-2058: override the classpath with SPARK_CONF_DIR