add javadoc to JobLogger, and some small format fix
Merge pull request #132 from Mistobaan/doc_fix

fix persistent-hdfs

(cherry picked from commit e7c7b804b529342d6de1c86ee02bc78da81279bc)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #134 from rxin/readme

Fixed a typo in Hadoop version in README.

(cherry picked from commit d407c0732aa7855a3b150975931425fe9ba30519)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #133 from Mistobaan/link_fix

update default github

(cherry picked from commit 41ead7a74533ffdd208a4ba2f7cd38945b4343ec)
Signed-off-by: Reynold Xin <rxin@apache.org>
modify according to comments
Merge pull request #102 from tdas/transform

Added new Spark Streaming operations

New operations
- transformWith which allows arbitrary 2-to-1 DStream transform, added to Scala and Java API
- StreamingContext.transform to allow arbitrary n-to-1 DStream
- leftOuterJoin and rightOuterJoin between 2 DStreams, added to Scala and Java API
- missing variations of join and cogroup added to Scala Java API
- missing JavaStreamingContext.union

Updated a number of Java and Scala API docs

(cherry picked from commit d307db6e554885d33e22ac09b567c66a0e702fd1)
Signed-off-by: Reynold Xin <rxin@apache.org>

Conflicts:
	core/src/main/scala/org/apache/spark/api/java/function/Function3.java
	core/src/main/scala/org/apache/spark/api/java/function/WrappedFunction3.scala
Merge pull request #112 from kayousterhout/ui_task_attempt_id

Display both task ID and task attempt ID in UI, and rename taskId to taskAttemptId

Previously only the task attempt ID was shown in the UI; this was confusing because the job can be shown as complete while there are tasks still running.  Showing the task ID in addition to the attempt ID makes it clear which tasks are redundant.

This commit also renames taskId to taskAttemptId in TaskInfo and in the local/cluster schedulers.  This identifier was used to uniquely identify attempts, not tasks, so the current naming was confusing.  The new naming is also more consistent with map reduce.

(cherry picked from commit aec9bf90606839a5f8ba58ad72af4fafc9efad76)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #118 from JoshRosen/blockinfo-memory-usage

Reduce the memory footprint of BlockInfo objects

This pull request reduces the memory footprint of all BlockInfo objects and makes additional optimizations for shuffle blocks.  For all BlockInfo objects, these changes remove two boolean fields and one Object field.  For shuffle blocks, we additionally remove an Object field and a boolean field.

When storing tens of thousands of these objects, this may add up to significant memory savings.  A ShuffleBlockInfo now only needs to wrap a single long.

This was motivated by a [report of high blockInfo memory usage during shuffles](https://mail-archives.apache.org/mod_mbox/incubator-spark-user/201310.mbox/%3C20131026134353.202b2b9b%40sh9%3E).

I haven't run benchmarks to measure the exact memory savings.

/cc @aarondav

(cherry picked from commit 745dc4290834e8c38d61600e23e4706b0ae47179)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #126 from kayousterhout/local_fix

Fixed incorrect log message in local scheduler

This change is especially relevant at the moment, because some users are seeing this failure, and the log message is misleading/incorrect (because for the tests, the max failures is set to 0, not 4)

(cherry picked from commit dc9ce16f6b695295c24062a76fd10f574d278cac)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #124 from tgravescs/sparkHadoopUtilFix

Pull SparkHadoopUtil out of SparkEnv (jira SPARK-886)

Having the logic to initialize the correct SparkHadoopUtil in SparkEnv prevents it from being used until after the SparkContext is initialized.   This causes issues like https://spark-project.atlassian.net/browse/SPARK-886.  It also makes it hard to use in singleton objects.  For instance I want to use it in the security code.

(cherry picked from commit 33de11c51dd2dbcbbf1801c54d9ce5ffaa324657)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #117 from stephenh/avoid_concurrent_modification_exception

Handle ConcurrentModificationExceptions in SparkContext init.

System.getProperties.toMap will fail-fast when concurrently modified,
and it seems like some other thread started by SparkContext does
a System.setProperty during it's initialization.

Handle this by just looping on ConcurrentModificationException, which
seems the safest, since the non-fail-fast methods (Hastable.entrySet)
have undefined behavior under concurrent modification.

(cherry picked from commit 8f1098a3f0de8c9b2eb9ede91a1b01da10a525ea)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #125 from velvia/2013-10/local-jar-uri

Add support for local:// URI scheme for addJars()

This PR adds support for a new URI scheme for SparkContext.addJars():  `local://file/path`.
The *local* scheme indicates that the `/file/path` exists on every worker node.    The reason for its existence is for big library JARs, which would be really expensive to serve using the standard HTTP fileserver distribution method, especially for big clusters.  Today the only inexpensive method (assuming such a file is on every host, via say NFS, rsync, etc.) of doing this is to add the JAR to the SPARK_CLASSPATH, but we want a method where the user does not need to modify the Spark configuration.

I would add something to the docs, but it's not obvious where to add it.

Oh, and it would be great if this could be merged in time for 0.8.1.

(cherry picked from commit 618c1f6cf3008caae7a8c0202721a6bd77d29a0f)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #129 from velvia/2013-11/document-local-uris

Document & finish support for local: URIs

Review all the supported URI schemes for addJar / addFile to the Cluster Overview page.
Add support for local: URI to addFile.

(cherry picked from commit d6d11c2edbd11d2fde6dceb706711f2a4c3cf39d)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #70 from rxin/hash1

Fast, memory-efficient hash set, hash table implementations optimized for primitive data types.

This pull request adds two hash table implementations optimized for primitive data types. For primitive types, the new hash tables are much faster than the current Spark AppendOnlyMap (3X faster - note that the current AppendOnlyMap is already much better than the Java map) while uses much less space (1/4 of the space).

Details:

This PR first adds a open hash set implementation (OpenHashSet) optimized for primitive types (using Scala's specialization feature). This OpenHashSet is designed to serve as building blocks for more advanced structures. It is currently used to build the following two hash tables, but can be used in the future to build multi-valued hash tables as well (GraphX has this use case). Note that there are some peculiarities in the code for working around some Scala compiler bugs.

Building on top of OpenHashSet, this PR adds two different hash tables implementations:
1. OpenHashSet: for nullable keys, optional specialization for primitive values
2. PrimitiveKeyOpenHashMap: for primitive keys that are not nullable, and optional specialization for primitive values

I tested the update speed of these two implementations using the changeValue function (which is what Aggregator and cogroup would use). Runtime relative to AppendOnlyMap for inserting 10 million items:

Int to Int: ~30%
java.lang.Integer to java.lang.Integer: ~100%
Int to java.lang.Integer: ~50%
java.lang.Integer to Int: ~85%

(cherry picked from commit b5dc3393a586099229fcd293d96f909e596a11e6)
Signed-off-by: Reynold Xin <rxin@apache.org>
Fix build break caused by BlockInfo refactor
Merge pull request #138 from marmbrus/branch-0.8

Fix build break caused by BlockInfo refactor
delete @return if method has no return value
Merge pull request #130 from aarondav/shuffle

Memory-optimized shuffle file consolidation

Reduces overhead of each shuffle block for consolidation from >300 bytes to 8 bytes (1 primitive Long). Verified via profiler testing with 1 mil shuffle blocks, net overhead was ~8,400,000 bytes.

Despite the memory-optimized implementation incurring extra CPU overhead, the runtime of the shuffle phase in this test was only around 2% slower, while the reduce phase was 40% faster, when compared to not using any shuffle file consolidation.

This is accomplished by replacing the map from ShuffleBlockId to FileSegment (i.e., block id to where it's located), which had high overhead due to being a gigantic, timestamped, concurrent map with a more space-efficient structure. Namely, the following are introduced (I have omitted the word "Shuffle" from some names for clarity):
**ShuffleFile** - there is one ShuffleFile per consolidated shuffle file on disk. We store an array of offsets into the physical shuffle file for each ShuffleMapTask that wrote into the file. This is sufficient to reconstruct FileSegments for mappers that are in the file.
**FileGroup** - contains a set of ShuffleFiles, one per reducer, that a MapTask can use to write its output. There is one FileGroup created per _concurrent_ MapTask. The FileGroup contains an array of the mapIds that have been written to all files in the group. The positions of elements in this array map directly onto the positions in each ShuffleFile's offsets array.

In order to locate the FileSegment associated with a BlockId, we have another structure which maps each reducer to the set of ShuffleFiles that were created for it. (There will be as many ShuffleFiles per reducer as there are FileGroups.) To lookup a given ShuffleBlockId (shuffleId, reducerId, mapId), we thus search through all ShuffleFiles associated with that reducer.

As a time optimization, we ensure that FileGroups are only reused for MapTasks with monotonically increasing mapIds. This allows us to perform a binary search to locate a mapId inside a group, and also enables potential future optimization (based on the usual monotonic access order).

(cherry picked from commit 7a26104ab7cb492b347ba761ef1f17ca1b9078e4)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #128 from shimingfei/joblogger-doc

add javadoc to JobLogger, and some small fix

against Spark-941

add javadoc to JobLogger, output more info for RDD, modify recordStageDepGraph to avoid output duplicate stage dependency information
Merge pull request #139 from aarondav/shuffle-next

Never store shuffle blocks in BlockManager

After the BlockId refactor (PR #114), it became very clear that ShuffleBlocks are of no use
within BlockManager (they had a no-arg constructor!). This patch completely eliminates
them, saving us around 100-150 bytes per shuffle block.
The total, system-wide overhead per shuffle block is now a flat 8 bytes, excluding
state saved by the MapOutputTracker.

Note: This should *not* be merged directly into 0.8.0 -- see #138
(cherry picked from commit 81065321c073fa35028b8aa110704385cba6d5b2)
Signed-off-by: Reynold Xin <rxin@apache.org>