GitHub PushEvent at 2014-10-17 13:58:56
Repository: spark
Actor: nabeelmukhtar
Payload: {"shas":[["044583a241203e7fe759366b273ad32fd9bf7c05","c4ef0ddac529825d297b4ece01d1517a7f691739@gmail.com","[Core] Upgrading ScalaStyle version to 0.5 and removing SparkSpaceAfterCommentStartChecker.\n\nAuthor: prudhvi <prudhvi953@gmail.com>\n\nCloses #2799 from prudhvije/ScalaStyle/space-after-comment-start and squashes the following commits:\n\nfc263a1 [prudhvi] [Core] Using scalastyle to check the space after comment start","prudhvi",true],["4c589cac4496c6a4bb8485a340bd0641dca13847","c1e8a5c1b546352227f2ce5d2e678223fda8f219@gmail.com","[SPARK-3944][Core] Code re-factored as suggested\n\nAuthor: Shiti <ssaxena.ece@gmail.com>\n\nCloses #2810 from Shiti/master and squashes the following commits:\n\n051d82f [Shiti] setting the default value of uri scheme to \"file\"  where matching \"file\" or None yields the same result","Shiti",true],["091d32c52e9d73da95896016c1d920e89858abfa","399a71a219a2bbf15a9ad52c797d50eb921e7e28@gmail.com","[SPARK-3971] [MLLib] [PySpark] hotfix: Customized pickler should work in cluster mode\n\nCustomized pickler should be registered before unpickling, but in executor, there is no way to register the picklers before run the tasks.\n\nSo, we need to register the picklers in the tasks itself, duplicate the javaToPython() and pythonToJava() in MLlib, call SerDe.initialize() before pickling or unpickling.\n\nAuthor: Davies Liu <davies.liu@gmail.com>\n\nCloses #2830 from davies/fix_pickle and squashes the following commits:\n\n0c85fb9 [Davies Liu] revert the privacy change\n6b94e15 [Davies Liu] use JavaConverters instead of JavaConversions\n0f02050 [Davies Liu] hotfix: Customized pickler does not work in cluster","Davies Liu",true],["99e416b6d64402a5432a265797a1c155a38f4e6f","a50d2185142f97721f64187b05c8deeec99604b1@gmail.com","[SQL] Fixes the race condition that may cause test failure\n\nThe removed `Future` was used to end the test case as soon as the Spark SQL CLI process exits. When the process exits prematurely, this mechanism prevents the test case to wait until timeout. But it also creates a race condition: when `foundAllExpectedAnswers.tryFailure` is called, there are chances that the last expected output line of the CLI process hasn't been caught by the main logics of the test code, thus fails the test case.\n\nRemoving this `Future` doesn't affect correctness.\n\nAuthor: Cheng Lian <lian.cs.zju@gmail.com>\n\nCloses #2823 from liancheng/clean-clisuite and squashes the following commits:\n\n489a97c [Cheng Lian] Fixes the race condition that may cause test failure","Cheng Lian",true],["2fe0ba95616bb3860736b6b426635a5d2a0e9bd9","38ba40da102c3b82607690471c403766355c3e07@imaginea.com","SPARK-3874: Provide stable TaskContext API\n\nThis is a small number of clean-up changes on top of #2782. Closes #2782.\n\nAuthor: Prashant Sharma <prashant.s@imaginea.com>\nAuthor: Patrick Wendell <pwendell@gmail.com>\n\nCloses #2803 from pwendell/pr-2782 and squashes the following commits:\n\n56d5b7a [Patrick Wendell] Minor clean-up\n44089ec [Patrick Wendell] Clean-up the TaskContext API.\ned551ce [Prashant Sharma] Fixed a typo\ndf261d0 [Prashant Sharma] Josh's suggestion\nfacf3b1 [Prashant Sharma] Fixed the mima issue.\n7ecc2fe [Prashant Sharma] CR, Moved implementations to TaskContextImpl\nbbd9e05 [Prashant Sharma] adding missed out files to git.\nef633f5 [Prashant Sharma] SPARK-3874, Provide stable TaskContext API","Prashant Sharma",true],["7f7b50ed9d4ffdd6b23e0faa56b068a049da67f7","5a7a6a189b47cd31d24667a751e4003cd04eced5@databricks.com","[SPARK-3923] Increase Akka heartbeat pause above heartbeat interval\n\nSomething about the 2.3.4 upgrade seems to have made the issue manifest where all the services disconnect from each other after exactly 1000 seconds (which is the heartbeat interval). [This post](https://groups.google.com/forum/#!topic/akka-user/X3xzpTCbEFs) suggests that heartbeat pause should be greater than heartbeat interval, and increasing the pause from 600s to 6000s seems to have rectified the issue. My current cluster has now exceeded 1400s of uptime without failure!\n\nI do not know why this fixed it, because the threshold we have set for the failure detector is the exponent of a timeout, and 300 is extremely large. Perhaps the default failure detector changed in 2.3.4 and now ignores threshold.\n\nAuthor: Aaron Davidson <aaron@databricks.com>\n\nCloses #2784 from aarondav/fix-timeout and squashes the following commits:\n\nbd1151a [Aaron Davidson] Increase pause, don't decrease interval\n9cb0372 [Aaron Davidson] [SPARK-3923] Decrease Akka heartbeat interval below heartbeat pause","Aaron Davidson",true],["be2ec4a91d14f48e6323989fb0e0226a9d65bf7e","5d1ed0070fb6b6d5f66c1cf6a3d0bb37cf6f15d1@gmail.com","[SQL]typo in HiveFromSpark\n\nAuthor: Kun Li <jacky.likun@gmail.com>\n\nCloses #2809 from jackylk/patch-1 and squashes the following commits:\n\n46c926b [Kun Li] typo in HiveFromSpark","Kun Li",true],["642b246beb7879978d31f2e6e97de7e06c74dcb7","7632d88ebdd73175d662531eefd9a84b8ade5e24@intel.com","[SPARK-3941][CORE] _remainingmem should not increase twice when updateBlockInfo\n\nIn BlockManagermasterActor, _remainingMem would increase memSize for twice when updateBlockInfo if new storageLevel is invalid and old storageLevel is \"useMemory\". Also, _remainingMem should increase with original memory size instead of new memSize.\n\nAuthor: Zhang, Liye <liye.zhang@intel.com>\n\nCloses #2792 from liyezhang556520/spark-3941-remainMem and squashes the following commits:\n\n3d487cc [Zhang, Liye] make the code concise\n0380a32 [Zhang, Liye] [SPARK-3941][CORE] _remainingmem should not increase twice when updateBlockInfo","Zhang, Liye",true],["e7f4ea8a52f0d3d56684b4f9caadce978eac4816","ba60166985ef87d02b3f95e141f5ecd3cb447aa2@aliyun.com","[SPARK-3890][Docs]remove redundant spark.executor.memory in doc\n\nIntroduced in https://github.com/pwendell/spark/commit/f7e79bc42c1635686c3af01eef147dae92de2529, I'm not sure why we need two spark.executor.memory here.\n\nAuthor: WangTaoTheTonic <barneystinson@aliyun.com>\nAuthor: WangTao <barneystinson@aliyun.com>\n\nCloses #2745 from WangTaoTheTonic/redundantconfig and squashes the following commits:\n\ne7564dc [WangTao] too long line\nfdbdb1f [WangTaoTheTonic] trivial workaround\nd06b6e5 [WangTaoTheTonic] remove redundant spark.executor.memory in doc","WangTaoTheTonic",true],["56fd34af52a18230bf3ea7b041f2a184eddc1103","5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","[SPARK-3741] Add afterExecute for handleConnectExecutor\n\nSorry. I found that I forgot to add `afterExecute` for `handleConnectExecutor` in #2593.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #2794 from zsxwing/SPARK-3741 and squashes the following commits:\n\na0bc4dd [zsxwing] Add afterExecute for handleConnectExecutor","zsxwing",true],["dedace83f35cba0f833d962acbd75572318948c4","a65aabad0bc0c7b99f75bd4351db13c2d716aa79@163.com","[SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section sometimes\n\nJobProgressPage could not show Fair Scheduler Pools section sometimes.\nSparkContext starts webui and then postEnvironmentUpdate. Sometimes JobProgressPage is accessed between webui starting and postEnvironmentUpdate, then the lazy val isFairScheduler will be false. The Fair Scheduler Pools section will not display any more.\n\nAuthor: yantangzhai <tyz0303@163.com>\nAuthor: YanTangZhai <hakeemzhai@tencent.com>\n\nCloses #1966 from YanTangZhai/SPARK-3067 and squashes the following commits:\n\nd4323f8 [yantangzhai] update [SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section sometimes\n8a00106 [YanTangZhai] Merge pull request #6 from apache/master\nb6391cc [yantangzhai] revert [SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section sometimes\nd2226cd [yantangzhai] [SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section sometimes\ncbcba66 [YanTangZhai] Merge pull request #3 from apache/master\naac7f7b [yantangzhai] [SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section sometimes\ncdef539 [YanTangZhai] Merge pull request #1 from apache/master","yantangzhai",true],["e678b9f02a2936b35c95e91a5f0ff388b5720261","ad63b47dfac1951bde2b661ba9fbbe123f686bee@cs.berkeley.edu","[SPARK-3973] Print call site information for broadcasts\n\nIts hard to debug which broadcast variables refer to what in a big codebase. Printing call site information helps in debugging.\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #2829 from shivaram/spark-broadcast-print and squashes the following commits:\n\ncd6dbdf [Shivaram Venkataraman] Print call site information for broadcasts","Shivaram Venkataraman",true]],"size":12,"ref":"refs/heads/master","head":"e678b9f02a2936b35c95e91a5f0ff388b5720261"}