fix "set MASTER automatically fails" bug.

spark-shell intends to set MASTER automatically if we do not provide the option when we start the shell , but there's a problem. 
The condition is "if [[ "x" != "x$SPARK_MASTER_IP" && "y" != "y$SPARK_MASTER_PORT" ]];" we sure will set SPARK_MASTER_IP explicitly, the SPARK_MASTER_PORT option, however, we probably do not set just using spark default port 7077. So if we do not set SPARK_MASTER_PORT, the condition will never be true. We should just use default port if users do not set port explicitly I think.
fix some format problem.
Removing docs on akka options
Minor fixes
Bug fix for reporting of spill output
Force use of LZF when spilling data
Fixing speculation bug
Remove shuffle files if they are still present on a machine.
Avoid matching attempt files in the checkpoint
Added StreamingContext.awaitTermination to streaming examples.
Restricting /lib to top level directory in .gitignore

This patch was proposed by Sean Mackrory.
Made run-example respect SPARK_JAVA_OPTS and SPARK_MEM.
Merge pull request #483 from pwendell/gitignore

Restricting /lib to top level directory in .gitignore

This patch was proposed by Sean Mackrory.
Merge pull request #482 from tdas/streaming-example-fix

Added StreamingContext.awaitTermination to streaming examples

StreamingContext.start() currently starts a non-daemon thread which prevents termination of a Spark Streaming program even if main function has exited. Since the expected behavior of a streaming program is to run until explicitly killed, this was sort of fine when spark streaming applications are launched from the command line. However, when launched in Yarn-standalone mode, this did not work as the driver effectively got terminated when the main function exits. So SparkStreaming examples did not work on Yarn.

This addition to the examples ensures that the examples work on Yarn and also ensures that everyone learns that StreamingContext.awaitTermination() being necessary for SparkStreaming programs to wait.

The true bug-fix of making sure all threads by Spark Streaming are daemon threads is left for post-0.9.
Merge pull request #449 from CrazyJvm/master

SPARK-1028 : fix "set MASTER automatically fails" bug.

spark-shell intends to set MASTER automatically if we do not provide the option when we start the shell , but there's a problem.
The condition is "if [[ "x" != "x$SPARK_MASTER_IP" && "y" != "y$SPARK_MASTER_PORT" ]];" we sure will set SPARK_MASTER_IP explicitly, the SPARK_MASTER_PORT option, however, we probably do not set just using spark default port 7077. So if we do not set SPARK_MASTER_PORT, the condition will never be true. We should just use default port if users do not set port explicitly I think.
Adding small code comment
Removed SPARK_MEM from run-examples.
Merge pull request #484 from tdas/run-example-fix

Made run-example respect SPARK_JAVA_OPTS and SPARK_MEM.

bin/run-example scripts was not passing Java properties set through the SPARK_JAVA_OPTS to the example. This is important for examples like Twitter** as the Twitter authentication information must be set through java properties. Hence added the same JAVA_OPTS code in run-example as it is in bin/spark-class script.

Also added SPARK_MEM, in case someone wants to run the example with different amounts of memory. This can be removed if it is not tune with the intended semantics of the run-example scripts.

@matei Please check this soon I want this to go in 0.9-rc4
Style clean-up
Merge pull request #480 from pwendell/0.9-fixes

Handful of 0.9 fixes

This patch addresses a few fixes for Spark 0.9.0 based on the last release candidate.

@mridulm gets credit for reporting most of the issues here. Many of the fixes here are based on his work in #477 and follow up discussion with him.