GitHub PushEvent at 2014-07-25 16:36:53
Repository: spark
Actor: sryza
Payload: {"shas":[["1407871733176c92c67ac547adf603c41a772f7f","9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","[MLLIB] make Mima ignore updateFeatures (private) in ALS\n\nFix Mima issues in #1521.\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #1533 from mengxr/mima-als and squashes the following commits:\n\n78386e1 [Xiangrui Meng] make Mima ignore updateFeatures (private) in ALS","Xiangrui Meng",true],["85d3596e65512d481f4be54df100be6bdc9c8e29","5a7a6a189b47cd31d24667a751e4003cd04eced5@databricks.com","SPARK-2047: Introduce an in-mem Sorter, and use it to reduce mem usage\n\n### Why and what?\nCurrently, the AppendOnlyMap performs an \"in-place\" sort by converting its array of [key, value, key, value] pairs into a an array of [(key, value), (key, value)] pairs. However, this causes us to allocate many Tuple2 objects, which come at a nontrivial overhead.\n\nThis patch adds a Sorter API, intended for in memory sorts, which simply ports the Android Timsort implementation (available under Apache v2) and abstracts the interface in a way which introduces no more than 1 virtual function invocation of overhead at each abstraction point.\n\nPlease compare our port of the Android Timsort sort with the original implementation: http://www.diffchecker.com/wiwrykcl\n\n### Memory implications\nAn AppendOnlyMap contains N kv pairs, which results in roughly 2N elements within its underlying array. Each of these elements is 4 bytes wide in a [compressed OOPS](https://wikis.oracle.com/display/HotSpotInternals/CompressedOops) system, which is the default.\n\nToday's approach immediately allocates N Tuple2 objects, which take up 24N bytes in total (exposed via YourKit), and undergoes a Java sort. The Java 6 version immediately copies the entire array (4N bytes here), while the Java 7 version has a worst-case allocation of half the array (2N bytes).\nThis results in a worst-case sorting overhead of 24N + 2N = 26N bytes (for Java 7).\n\nThe Sorter does not require allocating any tuples, but since it uses Timsort, it may copy up to half the entire array in the worst case.\nThis results in a worst-case sorting overhead of 4N bytes.\n\nThus, we have reduced the worst-case overhead of the sort by roughly 22 bytes times the number of elements.\n\n### Performance implications\nAs the destructiveSortedIterator is used for spilling in an ExternalAppendOnlyMap, the purpose of this patch is to provide stability by reducing memory usage rather than improve performance. However, because it implements Timsort, it also brings a substantial performance boost over our prior implementation.\n\nHere are the results of a microbenchmark that sorted 25 million, randomly distributed (Float, Int) pairs. The Java Arrays.sort() tests were run **only on the keys**, and thus moved less data. Our current implementation is called \"Tuple-sort using Arrays.sort()\" while the new implementation is \"KV-array using Sorter\".\n\n<table>\n<tr><th>Test</th><th>First run (JDK6)</th><th>Average of 10 (JDK6)</th><th>First run (JDK7)</th><th>Average of 10 (JDK7)</th></tr>\n<tr><td>primitive Arrays.sort()</td><td>3216 ms</td><td>1190 ms</td><td>2724 ms</td><td>131 ms (!!)</td></tr>\n<tr><td>Arrays.sort()</td><td>18564 ms</td><td>2006 ms</td><td>13201 ms</td><td>878 ms</td></tr>\n<tr><td>Tuple-sort using Arrays.sort()</td><td>31813 ms</td><td>3550 ms</td><td>20990 ms</td><td>1919 ms</td></tr>\n<tr><td><b>KV-array using Sorter</b></td><td></td><td></td><td><b>15020 ms</b></td><td><b>834 ms</b></td></tr>\n</table>\n\nThe results show that this Sorter performs exactly as expected (after the first run) -- it is as fast as the Java 7 Arrays.sort() (which shares the same algorithm), but is significantly faster than the Tuple-sort on Java 6 or 7.\n\nIn short, this patch should significantly improve performance for users running either Java 6 or 7.\n\nAuthor: Aaron Davidson <aaron@databricks.com>\n\nCloses #1502 from aarondav/sort and squashes the following commits:\n\n652d936 [Aaron Davidson] Update license, move Sorter to java src\na7b5b1c [Aaron Davidson] fix licenses\n5c0efaf [Aaron Davidson] Update tmpLength\nec395c8 [Aaron Davidson] Ignore benchmark (again) and fix docs\n034bf10 [Aaron Davidson] Change to Apache v2 Timsort\nb97296c [Aaron Davidson] Don't try to run benchmark on Jenkins + private[spark]\n6307338 [Aaron Davidson] SPARK-2047: Introduce an in-mem Sorter, and use it to reduce mem usage","Aaron Davidson",true],["79fe7634f6817eb2443bc152c6790a4439721fda","50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","[SPARK-2615] [SQL] Add Equal Sign \"==\" Support for HiveQl\n\nCurrently, the \"==\" in HiveQL expression will cause exception thrown, this patch will fix it.\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #1522 from chenghao-intel/equal and squashes the following commits:\n\nf62a0ff [Cheng Hao] Add == Support for HiveQl","Cheng Hao",true],["5f7b99168004f06191dcf121fe985e90bf7dcf74","fbb1b1249c4fec363b436c739458ff446a82c033@gmail.com","Graphx example\n\nfix examples\n\nAuthor: CrazyJvm <crazyjvm@gmail.com>\n\nCloses #1523 from CrazyJvm/graphx-example and squashes the following commits:\n\n663457a [CrazyJvm] outDegrees does not take parameters\n7cfff1d [CrazyJvm] fix example for joinVertices","CrazyJvm",true],["ddadf1b00470b9d7bf7386dacf198d41407a0a2b","48d3a847ef684d87177a99cd6c21cdf466f2e98a@qq.com","[YARN][SPARK-2606]:In some cases,the spark UI pages display incorrect\n\nThe issue is caused by #1112 .\n\nAuthor: GuoQiang Li <witgo@qq.com>\n\nCloses #1501 from witgo/webui_style and squashes the following commits:\n\n4b34998 [GuoQiang Li] In some cases, pages display incorrect in WebUI","GuoQiang Li",true],["02e45729472a22a31629cdd17dc3836ba6810189","fdeca1538ffc24c21530c35c328209050e7a4a3c@twitter.com","[YARN] SPARK-2577: File upload to viewfs is broken due to mount point re...\n\nOpting to the option 2 defined in SPARK-2577, i.e., retrieve and pass the correct file system object to addResource.\n\nAuthor: Gera Shegalov <gera@twitter.com>\n\nCloses #1483 from gerashegalov/master and squashes the following commits:\n\n90c9087 [Gera Shegalov] [YARN] SPARK-2577: File upload to viewfs is broken due to mount point resolution","Gera Shegalov",true],["6c2be93f081f33e9e97e1231b0084a6a0eb4fa22","f264844b57730cf91a9068607b05f0fbe2cff97a@gmail.com","Remove GraphX MessageToPartition for compatibility with sort-based shuffle\n\nMessageToPartition was used in `Graph#partitionBy`. Unlike a Tuple2, it marked the key as transient to avoid sending it over the network. However, it was incompatible with sort-based shuffle (SPARK-2045) and represented only a minor optimization: for partitionBy, it improved performance by 6.3% (30.4 s to 28.5 s) and reduced communication by 5.6% (114.2 MB to 107.8 MB).\n\nAuthor: Ankur Dave <ankurdave@gmail.com>\n\nCloses #1537 from ankurdave/remove-MessageToPartition and squashes the following commits:\n\nf9d0054 [Ankur Dave] Remove MessageToPartition\nab71364 [Ankur Dave] Remove unused VertexBroadcastMsg","Ankur Dave",true],["4c7243e109c713bdfb87891748800109ffbaae07","9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","[SPARK-2617] Correct doc and usages of preservesPartitioning\n\nThe name `preservesPartitioning` is ambiguous: 1) preserves the indices of partitions, 2) preserves the partitioner. The latter is correct and `preservesPartitioning` should really be called `preservesPartitioner` to avoid confusion. Unfortunately, this is already part of the API and we cannot change. We should be clear in the doc and fix wrong usages.\n\nThis PR\n\n1. adds notes in `maPartitions*`,\n2. makes `RDD.sample` preserve partitioner,\n3. changes `preservesPartitioning` to false in  `RDD.zip` because the keys of the first RDD are no longer the keys of the zipped RDD,\n4. fixes some wrong usages in MLlib.\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #1526 from mengxr/preserve-partitioner and squashes the following commits:\n\nb361e65 [Xiangrui Meng] update doc based on pwendell's comments\n3b1ba19 [Xiangrui Meng] update doc\n357575c [Xiangrui Meng] fix unit test\n20b4816 [Xiangrui Meng] Merge branch 'master' into preserve-partitioner\nd1caa65 [Xiangrui Meng] add doc to explain preservesPartitioning fix wrong usage of preservesPartitioning make sample preserse partitioning","Xiangrui Meng",true],["25921110fcd5afe568bf0d25fccd232787af7911","a5c478d606458f6c868a5126471748acd4730a6f@gmail.com","[SPARK-2609] Log thread ID when spilling ExternalAppendOnlyMap\n\nIt's useful to know whether one thread is constantly spilling or multiple threads are spilling relatively infrequently. Right now everything looks a little jumbled and we can't tell which lines belong to the same thread. For instance:\n\n```\n06:14:37 ExternalAppendOnlyMap: Spilling in-memory map of 4 MB to disk (194 times so far)\n06:14:37 ExternalAppendOnlyMap: Spilling in-memory map of 4 MB to disk (198 times so far)\n06:14:37 ExternalAppendOnlyMap: Spilling in-memory map of 4 MB to disk (198 times so far)\n06:14:37 ExternalAppendOnlyMap: Spilling in-memory map of 10 MB to disk (197 times so far)\n06:14:38 ExternalAppendOnlyMap: Spilling in-memory map of 9 MB to disk (45 times so far)\n06:14:38 ExternalAppendOnlyMap: Spilling in-memory map of 23 MB to disk (198 times so far)\n06:14:38 ExternalAppendOnlyMap: Spilling in-memory map of 38 MB to disk (25 times so far)\n06:14:38 ExternalAppendOnlyMap: Spilling in-memory map of 161 MB to disk (25 times so far)\n06:14:39 ExternalAppendOnlyMap: Spilling in-memory map of 0 MB to disk (199 times so far)\n06:14:39 ExternalAppendOnlyMap: Spilling in-memory map of 4 MB to disk (166 times so far)\n06:14:39 ExternalAppendOnlyMap: Spilling in-memory map of 4 MB to disk (199 times so far)\n06:14:39 ExternalAppendOnlyMap: Spilling in-memory map of 4 MB to disk (200 times so far)\n```\n\nAuthor: Andrew Or <andrewor14@gmail.com>\n\nCloses #1517 from andrewor14/external-log and squashes the following commits:\n\n90e48bb [Andrew Or] Log thread ID when spilling","Andrew Or",true],["f776bc98878428940b5130c0d7d9b7ee452c0bd3","cee0dece774e6fdeb3cdb57c2c58a8b8592e24fd@qq.com","[CORE] SPARK-2640: In \"local[N]\", free cores of the only executor should be touched by \"spark.task.cpus\" for every finish/start-up of tasks.\n\nMake spark's \"local[N]\" better.\nIn our company, we use \"local[N]\" in production. It works exellentlly. It's our best choice.\n\nAuthor: woshilaiceshide <woshilaiceshide@qq.com>\n\nCloses #1544 from woshilaiceshide/localX and squashes the following commits:\n\n6c85154 [woshilaiceshide] [CORE] SPARK-2640: In \"local[N]\", free cores of the only executor should be touched by \"spark.task.cpus\" for every finish/start-up of tasks.","woshilaiceshide",true],["1b790cf7755cace0d89ac5777717e6df3be7356f","f070f8f0c50caeda5841963fc7b57e73e59dad56@happy-camper.st","[SPARK-2588][SQL] Add some more DSLs.\n\nAuthor: Takuya UESHIN <ueshin@happy-camper.st>\n\nCloses #1491 from ueshin/issues/SPARK-2588 and squashes the following commits:\n\n43d0a46 [Takuya UESHIN] Merge branch 'master' into issues/SPARK-2588\n1023ea0 [Takuya UESHIN] Modify tests to use DSLs.\n2310bf1 [Takuya UESHIN] Add some more DSLs.","Takuya UESHIN",true],["91903e0a50b0efb7217610021a628b3043004d82","bdff7f849ddaeb48cc60ea6d20d05a7336797665@intel.com","SPARK-2277: clear host->rack info properly\n\nHi mridulm, I just think of this issue of [#1212](https://github.com/apache/spark/pull/1212): I added FakeRackUtil to hold the host -> rack mapping. It should be cleaned up after use so that it won't mess up with test cases others may add later.\nReally sorry about this.\n\nAuthor: Rui Li <rui.li@intel.com>\n\nCloses #1454 from lirui-intel/SPARK-2277-fix-UT and squashes the following commits:\n\nf8ea25c [Rui Li] SPARK-2277: clear host->rack info properly","Rui Li",true],["e060d3ee2d910a5a802bb29630dca6f66cc0525d","f1f4455a46e181f5a91079e1a9678b51f3800f4d@redhat.com","SPARK-2226:  [SQL] transform HAVING clauses with aggregate expressions that aren't in the aggregation list\n\nThis change adds an analyzer rule to\n  1. find expressions in `HAVING` clause filters that depend on unresolved attributes,\n  2. push these expressions down to the underlying aggregates, and then\n  3. project them away above the filter.\n\nIt also enables the `HAVING` queries in the Hive compatibility suite.\n\nAuthor: William Benton <willb@redhat.com>\n\nCloses #1497 from willb/spark-2226 and squashes the following commits:\n\n92c9a93 [William Benton] Removed unnecessary import\nf1d4f34 [William Benton] Cleanups missed in prior commit\n0e1624f [William Benton] Incorporated suggestions from @marmbrus; thanks!\n541d4ee [William Benton] Cleanups from review\n5a12647 [William Benton] Explanatory comments and stylistic cleanups.\nc7f2b2c [William Benton] Whitelist HAVING queries.\n29a26e3 [William Benton] Added rule to handle unresolved attributes in HAVING clauses (SPARK-2226)","William Benton",true],["1871574a240e6f28adeb6bc8accc98c851cafae5","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SPARK-2569][SQL] Fix shipping of TEMPORARY hive UDFs.\n\nInstead of shipping just the name and then looking up the info on the workers, we now ship the whole classname.  Also, I refactored the file as it was getting pretty large to move out the type conversion code to its own file.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #1552 from marmbrus/fixTempUdfs and squashes the following commits:\n\nb695904 [Michael Armbrust] Make add jar execute with Hive.  Ship the whole function class name since sometimes we cannot lookup temporary functions on the workers.","Michael Armbrust",true],["efdaeb111917dd0314f1d00ee8524bed1e2e21ca","b40b7796b7556b6727299d1068450ce3524a3b6a@twitter.com","[SPARK-2102][SQL][CORE] Add option for kryo registration required and use a resource pool in Spark SQL for Kryo instances.\n\nAuthor: Ian O Connell <ioconnell@twitter.com>\n\nCloses #1377 from ianoc/feature/SPARK-2102 and squashes the following commits:\n\n5498566 [Ian O Connell] Docs update suggested by Patrick\n20e8555 [Ian O Connell] Slight style change\nf92c294 [Ian O Connell] Add docs for new KryoSerializer option\nf3735c8 [Ian O Connell] Add using a kryo resource pool for the SqlSerializer\n4e5c342 [Ian O Connell] Register the SparkConf for kryo, it gets swept into serialization\n665805a [Ian O Connell] Add a spark.kryo.registrationRequired option for configuring the Kryo Serializer","Ian O Connell",true],["9b763329d968658d2dedee4485b7931646e6392e","38ba40da102c3b82607690471c403766355c3e07@imaginea.com","[SPARK-2549] Functions defined inside of other functions trigger failures\n\nAuthor: Prashant Sharma <prashant.s@imaginea.com>\n\nCloses #1510 from ScrapCodes/SPARK-2549/fun-in-fun and squashes the following commits:\n\n9458bc5 [Prashant Sharma] Tested by removing an inner function from excludes.\nbc03b1c [Prashant Sharma] SPARK-2549 Functions defined inside of other functions trigger failures","Prashant Sharma",true],["60f0ae3d87c84fd96e1f4d0abf5be1f51870e7ab","48d3a847ef684d87177a99cd6c21cdf466f2e98a@qq.com","[SPARK-2484][SQL] Build should not run hivecompatibility tests by default.\n\nAuthor: witgo <witgo@qq.com>\n\nCloses #1403 from witgo/hive_compatibility and squashes the following commits:\n\n4e5ecdb [witgo] The default does not run hive compatibility tests","witgo",true],["2d25e34814f81f288587f3277324cb655a5fb38d","f264844b57730cf91a9068607b05f0fbe2cff97a@gmail.com","Replace RoutingTableMessage with pair\n\nRoutingTableMessage was used to construct routing tables to enable\njoining VertexRDDs with partitioned edges. It stored three elements: the\ndestination vertex ID, the source edge partition, and a byte specifying\nthe position in which the edge partition referenced the vertex to enable\njoin elimination.\n\nHowever, this was incompatible with sort-based shuffle (SPARK-2045). It\nwas also slightly wasteful, because partition IDs are usually much\nsmaller than 2^32, though this was mitigated by a custom serializer that\nused variable-length encoding.\n\nThis commit replaces RoutingTableMessage with a pair of (VertexId, Int)\nwhere the Int encodes both the source partition ID (in the lower 30\nbits) and the position (in the top 2 bits).\n\nAuthor: Ankur Dave <ankurdave@gmail.com>\n\nCloses #1553 from ankurdave/remove-RoutingTableMessage and squashes the following commits:\n\n697e17b [Ankur Dave] Replace RoutingTableMessage with pair","Ankur Dave",true],["9e7725c86e70ffd3d2ff3a563460c2b7d0c9bbee","48d3a847ef684d87177a99cd6c21cdf466f2e98a@qq.com","SPARK-2662: Fix NPE for JsonProtocol\n\nAuthor: GuoQiang Li <witgo@qq.com>\n\nCloses #1511 from witgo/JsonProtocol and squashes the following commits:\n\n2b6227f [GuoQiang Li] Fix NPE for JsonProtocol","GuoQiang Li",true],["78d18fdbaa62d8ed235c29b2e37fd6607263c639","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SPARK-2658][SQL] Add rule for true = 1.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #1556 from marmbrus/fixBooleanEqualsOne and squashes the following commits:\n\nad8edd4 [Michael Armbrust] Add rule for true = 1 and false = 0.","Michael Armbrust",true]],"size":39,"ref":"refs/heads/sandy-spark-2099","head":"3bde0291411e37e3cd11942b854a6988f347925a"}