Merge pull request #928 from jerryshao/fairscheduler-refactor

Refactor FairSchedulableBuilder
(cherry picked from commit 834686b108ce31cbee531d89de6c6e80913448f4)

Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #941 from ilikerps/master

Add "org.apache." prefix to packages in spark-class
(cherry picked from commit f06f2da2cb2fd6bd25b57e5f5fd6f8e5d37ab1a3)

Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #937 from jerryshao/localProperties-fix

Fix PR926 local properties issues in Spark Streaming like scenarios
(cherry picked from commit a2ea069a5f2ed83268109deade456dc0fc9b79ee)

Signed-off-by: Reynold Xin <rxin@apache.org>
Merge remote-tracking branch 'pr/12'

Fix spacing so java.io.tmpdir doesn't run on with SPARK_JAVA_OPTS

(cherry picked from commit 7220e8f90b7d0bf5d5e3e4ad906a6b257ca75c53)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #9 from rxin/limit

Smarter take/limit implementation.

(cherry picked from commit 6566a19b38204d754c5e8f821b4276616e90abc6)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #7 from wannabeast/memorystore-fixes

some minor fixes to MemoryStore

This is a repeat of #5, moved to its own branch in my repo.

This makes all updates to   on ; it skips on synchronizing the reads where it can get away with it.

(cherry picked from commit 560ee5c9bba3f9fde380c831d0c6701343b2fecf)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge branch 'master' of github.com:markhamstra/incubator-spark

(cherry picked from commit ff540a015b970f722ae6aeba7dece481f9ad863e)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #14 from kayousterhout/untangle_scheduler

Improved organization of scheduling packages.

This commit does not change any code -- only file organization.
Please let me know if there was some masterminded strategy behind
the existing organization that I failed to understand!

There are two components of this change:
(1) Moving files out of the cluster package, and down
a level to the scheduling package. These files are all used by
the local scheduler in addition to the cluster scheduler(s), so
should not be in the cluster package. As a result of this change,
none of the files in the local package reference files in the
cluster package.

(2) Moving the mesos package to within the cluster package.
The mesos scheduling code is for a cluster, and represents a
specific case of cluster scheduling (the Mesos-related classes
often subclass cluster scheduling classes). Thus, the most logical
place for it seems to be within the cluster package.

The one thing about the scheduling code that seems a little funny to me
is the naming of the SchedulerBackends.  The StandaloneSchedulerBackend
is not just for Standalone mode, but instead is used by Mesos coarse grained
mode and Yarn, and the backend that *is* just for Standalone mode is instead called SparkDeploySchedulerBackend. I didn't change this because I wasn't sure if there
was a reason for this naming that I'm just not aware of.

(cherry picked from commit 70a0b993d4f1a72a1b38f9a17fcf269acc1b394f)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #930 from holdenk/master

Add mapPartitionsWithIndex
(cherry picked from commit c514cd15879d510406712b1cebdaf0c0db9151c9)

Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #17 from rxin/optimize

Remove -optimize flag

(cherry picked from commit 714fdabd99bbff3a0cdec5dcf06b021a3a3f2da8)
Signed-off-by: Reynold Xin <rxin@apache.org>
Resolving package conflicts with hadoop 0.23.9
Fix UISuite test to handle situations where DEFAULT_PORT is busy on the test machine
Indentation fix
Merge pull request #31 from sundeepn/branch-0.8

Resolving package conflicts with hadoop 0.23.9

Hadoop 0.23.9 is having a package conflict with easymock's dependencies.
Adding Shark 0.7.1 to EC2 scripts
Merge pull request #40 from pwendell/branch-0.8

Adding Shark 0.7.1 to EC2 scripts [branch 0.8 version]

This adds a newer version of Shark to the ec2 scripts. I've tested this for both Hadoop1 and Hadoop2 clusters.
Merge pull request #42 from pwendell/shuffle-read-perf

Fix inconsistent and incorrect log messages in shuffle read path

The user-facing messages generated by the CacheManager are currently wrong and somewhat misleading. This patch makes the messages more accurate. It also uses a consistent representation of the partition being fetched (`rdd_xx_yy`) so that it's easier for users to trace what is going on when reading logs.

(cherry picked from commit ea34c521025d3408d44d45ab5c132fd9791794f6)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #43 from mateiz/kryo-fix

Don't allocate Kryo buffers unless needed

I noticed that the Kryo serializer could be slower than the Java one by 2-3x on small shuffles because it spend a lot of time initializing Kryo Input and Output objects. This is because our default buffer size for them is very large. Since the serializer is often used on streams, I made the initialization lazy for that, and used a smaller buffer (auto-managed by Kryo) for input.

(cherry picked from commit e67d5b962a2adddc073cfc9c99be9012fbb69838)
Signed-off-by: Reynold Xin <rxin@apache.org>
add Function3 and WrappedFunction3
fix formating