GitHub PushEvent at 2014-06-09 17:08:17
Repository: spark
Actor: kanzhang
Payload: {"shas":[["c7a183b2c2bca13565496495b4ae3a3a9f63f9ab","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SPARK-2041][SQL] Correctly analyze queries where columnName == tableName.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #985 from marmbrus/tableName and squashes the following commits:\n\n3caaa27 [Michael Armbrust] Correctly analyze queries where columnName == tableName.","Michael Armbrust",true],["3d3f8c8004da110ca97973119e9d9f04f878ee81","fbb1b1249c4fec363b436c739458ff446a82c033@gmail.com","Use pluggable clock in DAGSheduler #SPARK-2031\n\nDAGScheduler supports pluggable clock like what TaskSetManager does.\n\nAuthor: CrazyJvm <crazyjvm@gmail.com>\n\nCloses #976 from CrazyJvm/clock and squashes the following commits:\n\n6779a4c [CrazyJvm] Use pluggable clock in DAGSheduler","CrazyJvm",true],["9bad0b73722fb359f14db864e69aa7efde3588c5","f264844b57730cf91a9068607b05f0fbe2cff97a@gmail.com","[SPARK-2025] Unpersist edges of previous graph in Pregel\n\nDue to a bug introduced by apache/spark#497, Pregel does not unpersist replicated vertices from previous iterations. As a result, they stay cached until memory is full, wasting GC time.\n\nThis PR corrects the problem by unpersisting both the edges and the replicated vertices of previous iterations. This is safe because the edges and replicated vertices of the current iteration are cached by the call to `g.cache()` and then materialized by the call to `messages.count()`. Therefore no unmaterialized RDDs depend on `prevG.edges`. I verified that no recomputation occurs by running PageRank with a custom patch to Spark that warns when a partition is recomputed.\n\nThanks to Tim Weninger for reporting this bug.\n\nAuthor: Ankur Dave <ankurdave@gmail.com>\n\nCloses #972 from ankurdave/SPARK-2025 and squashes the following commits:\n\n13d5b07 [Ankur Dave] Unpersist edges of previous graph in Pregel","Ankur Dave",true],["b45c13e7d798f97b92f1a6329528191b8d779c4f","c481c5e387de8107aa9fe99c851e11e1ff44d0c3@databricks.com","SPARK-2043: ExternalAppendOnlyMap doesn't always find matching keys\n\nThe current implementation reads one key with the next hash code as it finishes reading the keys with the current hash code, which may cause it to miss some matches of the next key. This can cause operations like join to give the wrong result when reduce tasks spill to disk and there are hash collisions, as values won't be matched together. This PR fixes it by not reading in that next key, using a peeking iterator instead.\n\nAuthor: Matei Zaharia <matei@databricks.com>\n\nCloses #986 from mateiz/spark-2043 and squashes the following commits:\n\n0959514 [Matei Zaharia] Added unit test for having many hash collisions\n892debb [Matei Zaharia] SPARK-2043: don't read a key with the next hash code in ExternalAppendOnlyMap, instead use a buffered iterator to only read values with the current hash code.","Matei Zaharia",true],["41db44c428a10f4453462d002d226798bb8fbdda","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SPARK-2050][SQL] LIKE, RLIKE and IN in HQL should not be case sensitive.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #989 from marmbrus/caseSensitiveFuncitons and squashes the following commits:\n\n681de54 [Michael Armbrust] LIKE, RLIKE and IN in HQL should not be case sensitive.","Michael Armbrust",true],["8d85359f84cc67996b4bcf1670a8a98ab4f914a2","f264844b57730cf91a9068607b05f0fbe2cff97a@gmail.com","[SPARK-1552] Fix type comparison bug in {map,outerJoin}Vertices\n\nIn GraphImpl, mapVertices and outerJoinVertices use a more efficient implementation when the map function conserves vertex attribute types. This is implemented by comparing the ClassTags of the old and new vertex attribute types. However, ClassTags store erased types, so the comparison will return a false positive for types with different type parameters, such as Option[Int] and Option[Double].\n\nThis PR resolves the problem by requesting that the compiler generate evidence of equality between the old and new vertex attribute types, and providing a default value for the evidence parameter if the two types are not equal. The methods can then check the value of the evidence parameter to see whether the types are equal.\n\nIt also adds a test called \"mapVertices changing type with same erased type\" that failed before the PR and succeeds now.\n\nCallers of mapVertices and outerJoinVertices can no longer use a wildcard for a graph's VD type. To avoid \"Error occurred in an application involving default arguments,\" they must bind VD to a type parameter, as this PR does for ShortestPaths and LabelPropagation.\n\nAuthor: Ankur Dave <ankurdave@gmail.com>\n\nCloses #967 from ankurdave/SPARK-1552 and squashes the following commits:\n\n68a4fff [Ankur Dave] Undo conserve naming\n7388705 [Ankur Dave] Remove unnecessary ClassTag for VD parameters\na704e5f [Ankur Dave] Use type equality constraint with default argument\n29a5ab7 [Ankur Dave] Add failing test\nf458c83 [Ankur Dave] Revert \"[SPARK-1552] Fix type comparison bug in mapVertices and outerJoinVertices\"\n16d6af8 [Ankur Dave] [SPARK-1552] Fix type comparison bug in mapVertices and outerJoinVertices","Ankur Dave",true],["8d210560be8b143e48abfbaca347f383b5aa4798","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SPARK-2050 - 2][SQL] DIV and BETWEEN should not be case sensitive.\n\nFollowup: #989\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #994 from marmbrus/caseSensitiveFunctions2 and squashes the following commits:\n\n9d9c8ed [Michael Armbrust] Fix DIV and BETWEEN.","Michael Armbrust",true],["41c4a33105c74417192925db355019ba1badeab2","48d3a847ef684d87177a99cd6c21cdf466f2e98a@qq.com"," [SPARK-1841]: update scalatest to version 2.1.5\n\nAuthor: witgo <witgo@qq.com>\n\nCloses #713 from witgo/scalatest and squashes the following commits:\n\nb627a6a [witgo] merge master\n51fb3d6 [witgo] merge master\n3771474 [witgo] fix RDDSuite\n996d6f9 [witgo] fix TimeStampedWeakValueHashMap test\n9dfa4e7 [witgo] merge bug\n1479b22 [witgo] merge master\n29b9194 [witgo] fix code style\n022a7a2 [witgo] fix test dependency\na52c0fa [witgo] fix test dependency\ncd8f59d [witgo] Merge branch 'master' of https://github.com/apache/spark into scalatest\n046540d [witgo] fix RDDSuite.scala\n2c543b9 [witgo] fix ReplSuite.scala\nc458928 [witgo] update scalatest to version 2.1.5","witgo",true],["a6c72ab16e7a3027739ab419819f5222e270838e","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SPARK-1994][SQL] Weird data corruption bug when running Spark SQL on data in HDFS\n\nBasically there is a race condition (possibly a scala bug?) when these values are recomputed on all of the slaves that results in an incorrect projection being generated (possibly because the GUID uniqueness contract is broken?).\n\nIn general we should probably enforce that all expression planing occurs on the driver, as is now occurring here.\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #1004 from marmbrus/fixAggBug and squashes the following commits:\n\ne0c116c [Michael Armbrust] Compute aggregate expression during planning instead of lazily on workers.","Michael Armbrust",true],["3ace10dc91e72ebe5013d5106eb0968a77c99d8d","8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","HOTFIX: Support empty body in merge script\n\nDiscovered in #992\n\nAuthor: Patrick Wendell <pwendell@gmail.com>\n\nCloses #1007 from pwendell/hotfix and squashes the following commits:\n\naf90aa0 [Patrick Wendell] HOTFIX: Support empty body in merge script","Patrick Wendell",true],["7b877b27053bfb7092e250e01a3b887e1b50a109","6504ab7c5083d5070fa8749cea625821ca8a6c66@spotify.com","SPARK-2056 Set RDD name to input path\n\nAuthor: Neville Li <neville@spotify.com>\n\nCloses #992 from nevillelyh/master and squashes the following commits:\n\n3011739 [Neville Li] [SPARK-2056] Set RDD name to input path","Neville Li",true],["a338834f90556d78119b37985b603ecee85f97ed","9e5910599b22a2ff8c9ccd67d61bc3265a983719@gmail.com","SPARK-2026: Maven Hadoop Profiles Should Set The Hadoop Version\n\nThe Maven Profiles that refer to hadoopX, e.g. `hadoop2.4`, should set the expected\n`hadoop.version` and `yarn.version`.\n\ne.g.\n\n```\n<profile>\n      <id>hadoop-2.4</id>\n      <properties>\n        <hadoop.version>2.4.0</hadoop.version>\n         <yarn.version>${hadoop.version}</yarn.version>\n        <protobuf.version>2.5.0</protobuf.version>\n        <jets3t.version>0.9.0</jets3t.version>\n      </properties>\n</profile>\n```\n\nBuilds can still define the `-Dhadoop.version` option but this will correctly default the\nHadoop Version to the one that is expected according the profile that is selected.\n\ne.g.\n\n```$ mvn -P hadoop-2.4,yarn clean install```\nor\n\n```$ mvn -P hadoop-0.23,yarn clean install```\n\n[ticket] : https://issues.apache.org/jira/browse/SPARK-2026\n\nAuthor      : berngp\nReviewer    : ?\n\nAuthor: Bernardo Gomez Palacio <bernardo.gomezpalacio@gmail.com>\n\nCloses #998 from berngp/feature/SPARK-2026 and squashes the following commits:\n\n07ba4f7 [Bernardo Gomez Palacio] SPARK-2026: Maven Hadoop Profiles Should Set The Hadoop Version","Bernardo Gomez Palacio",true],["ee96e9406613e621837360b15c34ea7c7220a7a3","3142ce8a0bdba4a38073365893e6b74f72d9c512@cloudera.com","SPARK-1898: In deploy.yarn.Client, use YarnClient not YarnClientImpl\n\nhttps://issues.apache.org/jira/browse/SPARK-1898\n\nAuthor: Colin Patrick McCabe <cmccabe@cloudera.com>\n\nCloses #850 from cmccabe/master and squashes the following commits:\n\nd66eddc [Colin Patrick McCabe] SPARK-1898: In deploy.yarn.Client, use YarnClient rather than YarnClientImpl","Colin Patrick McCabe",true],["a71c6d1cf0bbc027f633a65042191cf2201330d6","5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","SPARK-1628: Add missing hashCode methods in Partitioner subclasses\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-1628\n\nAdded `hashCode` in HashPartitioner, RangePartitioner, PythonPartitioner and PageRankUtils.CustomPartitioner.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #549 from zsxwing/SPARK-1628 and squashes the following commits:\n\n2620936 [zsxwing] SPARK-1628: Add missing hashCode methods in Partitioner subclasses","zsxwing",true],["e9261d0866a610eab29fa332726186b534d1018f","5cc7173406045a3242d3595588aa610a46f4e47d@asiainfo-linkage.com","Update run-example\n\nOld code can only be ran under spark_home and use \"bin/run-example\".\n Error \"./run-example: line 55: ./bin/spark-submit: No such file or directory\" appears when running in other place. So change this\n\nAuthor: maji2014 <maji3@asiainfo-linkage.com>\n\nCloses #1011 from maji2014/master and squashes the following commits:\n\n2cc1af6 [maji2014] Update run-example\n\nCloses #988.","maji2014",true],["219dc00b30c8d9c4c0a6ce5d566497a93f21cb57","c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@apache.org","SPARK-1628 follow up: Improve RangePartitioner's documentation.\n\nAdding a paragraph clarifying a weird behavior in RangePartitioner.\n\nSee also #549.\n\nAuthor: Reynold Xin <rxin@apache.org>\n\nCloses #1012 from rxin/partitioner-doc and squashes the following commits:\n\n6f0109e [Reynold Xin] SPARK-1628 follow up: Improve RangePartitioner's documentation.","Reynold Xin",true],["15ddbef414d5fd6d4672936ba3c747b5fb7ab52b","6504ab7c5083d5070fa8749cea625821ca8a6c66@spotify.com","[SPARK-2067] use relative path for Spark logo in UI\n\nAuthor: Neville Li <neville@spotify.com>\n\nCloses #1006 from nevillelyh/gh/SPARK-2067 and squashes the following commits:\n\n9ee64cf [Neville Li] [SPARK-2067] use relative path for Spark logo in UI","Neville Li",true],["32ee9f0668e50083e415b0662b18f5d2581413f0","02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@andrewash.com","Grammar: read -> reads\n\nAuthor: Andrew Ash <andrew@andrewash.com>\n\nCloses #1016 from ash211/patch-6 and squashes the following commits:\n\ne3865c8 [Andrew Ash] Grammar: read -> reads","Andrew Ash",true],["6113ac1559d62c828dfbf08ef0f7f172c24cf7f5","5617ee1acc0052186605e19fc1926f15c864b625@cloudera.com","[SPARK-1308] Add getNumPartitions to pyspark RDD\n\nAdd getNumPartitions to pyspark RDD to provide an intuitive way to get number of partitions in RDD like we can do in scala today.\n\nAuthor: Syed Hashmi <shashmi@cloudera.com>\n\nCloses #995 from syedhashmi/master and squashes the following commits:\n\nde0ed5e [Syed Hashmi] [SPARK-1308] Add getNumPartitions to pyspark RDD","Syed Hashmi",true]],"size":19,"ref":"refs/heads/master","head":"6113ac1559d62c828dfbf08ef0f7f172c24cf7f5"}