GitHub PushEvent at 2014-09-27 03:05:35
Repository: spark
Actor: rxin
Payload: {"shas":[["78d4220fa0bf2f9ee663e34bbf3544a5313b02f0","c303fda042c807757c14303e8b8efcad59173751@databricks.com","SPARK-3608 Break if the instance tag naming succeeds\n\nAuthor: Vida Ha <vida@databricks.com>\n\nCloses #2466 from vidaha/vida/spark-3608 and squashes the following commits:\n\n9509776 [Vida Ha] Break if the instance tag naming succeeds","Vida Ha",true],["c32c8538efca2124924920614e4dbe7ce90938f4","8da22c45572ff7532871766932ae66e63da3e927@mola.io","Fix Java example in Streaming Programming Guide\n\n\"val conf\" was used instead of \"SparkConf conf\" in Java snippet.\n\nAuthor: Santiago M. Mola <santi@mola.io>\n\nCloses #2472 from smola/patch-1 and squashes the following commits:\n\n5bfeb9b [Santiago M. Mola] Fix Java example in Streaming Programming Guide","Santiago M. Mola",true],["5f8833c672ab64aa5886a8239ae2ff2a8ea42363","1fa2ef4755a9226cb9a0a4840bd89b158ac71391@redhat.com","[PySpark] remove unnecessary use of numSlices from pyspark tests\n\nAuthor: Matthew Farrellee <matt@redhat.com>\n\nCloses #2467 from mattf/master-pyspark-remove-numslices-from-tests and squashes the following commits:\n\nc49a87b [Matthew Farrellee] [PySpark] remove unnecessary use of numSlices from pyspark tests","Matthew Farrellee",true],["7c8ad1c0838762f5b632f683834c88a711aef4dd","5673041ad99da73cfefa170a7081c0daed5d97b6@cloudera.com","SPARK-3574. Shuffle finish time always reported as -1\n\nThe included test waits 100 ms after job completion for task completion events to come in so it can verify they have reasonable finish times.  Does anyone know a better way to wait on listener events that are expected to come in?\n\nAuthor: Sandy Ryza <sandy@cloudera.com>\n\nCloses #2440 from sryza/sandy-spark-3574 and squashes the following commits:\n\nc81439b [Sandy Ryza] Fix test failure\nb340956 [Sandy Ryza] SPARK-3574. Remove shuffleFinishTime metric","Sandy Ryza",true],["7f54580c4503d8b6bfcf7d4cbc83b83458140926","a50d2185142f97721f64187b05c8deeec99604b1@gmail.com","[SPARK-3609][SQL] Adds sizeInBytes statistics for Limit operator when all output attributes are of native data types\n\nThis helps to replace shuffled hash joins with broadcast hash joins in some cases.\n\nAuthor: Cheng Lian <lian.cs.zju@gmail.com>\n\nCloses #2468 from liancheng/more-stats and squashes the following commits:\n\n32687dc [Cheng Lian] Moved the test case to PlannerSuite\n5595a91 [Cheng Lian] Removes debugging code\n73faf69 [Cheng Lian] Test case for auto choosing broadcast hash join\nf30fe1d [Cheng Lian] Adds sizeInBytes estimation for Limit when all output types are native types","Cheng Lian",true],["293ce85145d7a37f7cb329831cbf921be571c2f5","17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","[SPARK-3414][SQL] Replace LowerCaseSchema with Resolver\n\n**This PR introduces a subtle change in semantics for HiveContext when using the results in Python or Scala.  Specifically, while resolution remains case insensitive, it is now case preserving.**\n\n_This PR is a follow up to #2293 (and to a lesser extent #2262 #2334)._\n\nIn #2293 the catalog was changed to store analyzed logical plans instead of unresolved ones.  While this change fixed the reported bug (which was caused by yet another instance of us forgetting to put in a `LowerCaseSchema` operator) it had the consequence of breaking assumptions made by `MultiInstanceRelation`.  Specifically, we can't replace swap out leaf operators in a tree without rewriting changed expression ids (which happens when you self join the same RDD that has been registered as a temp table).\n\nIn this PR, I instead remove the need to insert `LowerCaseSchema` operators at all, by moving the concern of matching up identifiers completely into analysis.  Doing so allows the test cases from both #2293 and #2262 to pass at the same time (and likely fixes a slew of other \"unknown unknown\" bugs).\n\nWhile it is rolled back in this PR, storing the analyzed plan might actually be a good idea.  For instance, it is kind of confusing if you register a temporary table, change the case sensitivity of resolution and now you can't query that table anymore.  This can be addressed in a follow up PR.\n\nFollow-ups:\n - Configurable case sensitivity\n - Consider storing analyzed plans for temp tables\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #2382 from marmbrus/lowercase and squashes the following commits:\n\nc21171e [Michael Armbrust] Ensure the resolver is used for field lookups and ensure that case insensitive resolution is still case preserving.\nd4320f1 [Michael Armbrust] Merge remote-tracking branch 'origin/master' into lowercase\n2de881e [Michael Armbrust] Address comments.\n219805a [Michael Armbrust] style\n5b93711 [Michael Armbrust] Replace LowerCaseSchema with Resolver.","Michael Armbrust",true],["8e875d2aff5f30a5f7a4bf694fc89a8b852fdcdc","ba60166985ef87d02b3f95e141f5ecd3cb447aa2@aliyun.com","[SPARK-3599]Avoid loading properties file frequently\n\nhttps://issues.apache.org/jira/browse/SPARK-3599\n\nAuthor: WangTao <barneystinson@aliyun.com>\nAuthor: WangTaoTheTonic <barneystinson@aliyun.com>\n\nCloses #2454 from WangTaoTheTonic/avoidLoadingFrequently and squashes the following commits:\n\n3681182 [WangTao] do not use clone\n7dca036 [WangTao] use lazy val instead\n2a79f26 [WangTaoTheTonic] Avoid loaing properties file frequently","WangTao",true],["d112a6c79dee7b5d8459696f97d329190e8d09a5","8bcd50e7b1eea79f965dde19d326001805380997@gmail.com","MAINTENANCE: Automated closing of pull requests.\n\nThis commit exists to close the following pull requests on Github:\n\nCloses #1328 (close requested by 'pwendell')\nCloses #2314 (close requested by 'pwendell')\nCloses #997 (close requested by 'pwendell')\nCloses #550 (close requested by 'pwendell')\nCloses #1506 (close requested by 'pwendell')\nCloses #2423 (close requested by 'mengxr')\nCloses #554 (close requested by 'joshrosen')","Patrick Wendell",true],["a0454efe21e5c7ffe1b9bb7b18021a5580952e69","57a33a5496950fec8433e4dd83347673459dcdfc@themodernlife.net","[SPARK-3595] Respect configured OutputCommitters when calling saveAsHadoopFile\n\nAddresses the issue in https://issues.apache.org/jira/browse/SPARK-3595, namely saveAsHadoopFile hardcoding the OutputCommitter.  This is not ideal when running Spark jobs that write to S3, especially when running them from an EMR cluster where the default OutputCommitter is a DirectOutputCommitter.\n\nAuthor: Ian Hummel <ian@themodernlife.net>\n\nCloses #2450 from themodernlife/spark-3595 and squashes the following commits:\n\nf37a0e5 [Ian Hummel] Update based on comments from pwendell\na11d9f3 [Ian Hummel] Fix formatting\n4359664 [Ian Hummel] Add an example showing usage\n8b6be94 [Ian Hummel] Add ability to specify OutputCommitter, espcially useful when writing to an S3 bucket from an EMR cluster","Ian Hummel",true],["fd0b32c520e3d1088b2fe9228be114932e6c3a0c","f98b0bead8ca0151df080a83631a6c2556256922@huawei.com","[Minor]ignore .idea_modules\n\nignore .idea_modules ,  ```sbt/sbt gen-idea``` generate this dir.\n\nAuthor: wangfei <wangfei1@huawei.com>\n\nCloses #2476 from scwf/patch-4 and squashes the following commits:\n\ne6ab88a [wangfei] ignore .idea_modules","wangfei",true],["fec921552ffccc36937214406b3e4a050eb0d8e0","3270cab15a4ae4af5d46c40db43f5b35eac294d2@gmail.com","[MLLib] Fix example code variable name misspelling in MLLib Feature Extraction guide\n\nAuthor: RJ Nowling <rnowling@gmail.com>\n\nCloses #2459 from rnowling/tfidf-fix and squashes the following commits:\n\nb370a91 [RJ Nowling] Fix variable name misspelling in MLLib Feature Extraction guide","RJ Nowling",true],["56dae30ca70489a62686cb245728b09b2179bb5a","8d6a149a60244da40a78dc7cfec3efb2fee99fe2@gmail.com","Update docs to use jsonRDD instead of wrong jsonRdd.\n\nAuthor: Grega Kespret <grega.kespret@gmail.com>\n\nCloses #2479 from gregakespret/patch-1 and squashes the following commits:\n\ndd6b90a [Grega Kespret] Update docs to use jsonRDD instead of wrong jsonRdd.","Grega Kespret",true],["f9d6220c792b779be385f3022d146911a22c2130","f264844b57730cf91a9068607b05f0fbe2cff97a@gmail.com","[SPARK-3578] Fix upper bound in GraphGenerators.sampleLogNormal\n\nGraphGenerators.sampleLogNormal is supposed to return an integer strictly less than maxVal. However, it violates this guarantee. It generates its return value as follows:\n\n```scala\nvar X: Double = maxVal\n\nwhile (X >= maxVal) {\n  val Z = rand.nextGaussian()\n  X = math.exp(mu + sigma*Z)\n}\nmath.round(X.toFloat)\n```\n\nWhen X is sampled to be close to (but less than) maxVal, then it will pass the while loop condition, but the rounded result will be equal to maxVal, which will violate the guarantee. For example, if maxVal is 5 and X is 4.9, then X < maxVal, but `math.round(X.toFloat)` is 5.\n\nThis PR instead rounds X before checking the loop condition, guaranteeing that the condition will hold for the return value.\n\nAuthor: Ankur Dave <ankurdave@gmail.com>\n\nCloses #2439 from ankurdave/SPARK-3578 and squashes the following commits:\n\nf6655e5 [Ankur Dave] Go back to math.floor\n5900c22 [Ankur Dave] Round X in loop condition\n6fd5fb1 [Ankur Dave] Run sampleLogNormal bounds check 1000 times\n1638598 [Ankur Dave] Round down in sampleLogNormal to guarantee upper bound","Ankur Dave",true],["14f8c340402366cb998c563b3f7d9ff7d9940271","7515399f38852828862eccfc240e12b11a53a4bb@xiaomi.com","[YARN] SPARK-2668: Add variable of yarn log directory for reference from the log4j configuration\n\nAssign value of yarn container log directory to java opts \"spark.yarn.app.container.log.dir\", So user defined log4j.properties can reference this value and write log to YARN container's log directory.\nOtherwise, user defined file appender will only write to container's CWD, and log files in CWD will not be displayed on YARN UI，and either cannot be aggregated to HDFS log directory after job finished.\n\nUser defined log4j.properties reference example:\nlog4j.appender.rolling_file.File = ${spark.yarn.app.container.log.dir}/spark.log\n\nAuthor: peng.zhang <peng.zhang@xiaomi.com>\n\nCloses #1573 from renozhang/yarn-log-dir and squashes the following commits:\n\n16c5cb8 [peng.zhang] Update doc\nf2b5e2a [peng.zhang] Change variable's name, and update running-on-yarn.md\n503ea2d [peng.zhang] Support log4j log to yarn container dir","peng.zhang",true],["c4022dd52b4827323ff956632dc7623f546da937","a5c478d606458f6c868a5126471748acd4730a6f@gmail.com","[SPARK-3477] Clean up code in Yarn Client / ClientBase\n\nThis is part of a broader effort to clean up the Yarn integration code after #2020.\n\nThe high-level changes in this PR include:\n- Removing duplicate code, especially across the alpha and stable APIs\n- Simplify unnecessarily complex method signatures and hierarchies\n- Rename unclear variable and method names\n- Organize logging output produced when the user runs Spark on Yarn\n- Extensively add documentation\n- Privatize classes where possible\n\nI have tested the stable API on a Hadoop 2.4 cluster. I tested submitting a jar that references classes in other jars in both client and cluster mode. I also made changes in the alpha API, though I do not have access to an alpha cluster. I have verified that it compiles, but it would be ideal if others can help test it.\n\nFor those interested in some examples in detail, please read on.\n\n--------------------------------------------------------------------------------------------------------\n\n***Appendix***\n\n- The loop to `getApplicationReport` from the RM is duplicated in 4 places: in the stable `Client`, alpha `Client`, and twice in `YarnClientSchedulerBackend`. We should not have different loops for client and cluster deploy modes.\n- There are many fragmented small helper methods that are only used once and should just be inlined. For instance, `ClientBase#getLocalPath` returns `null` on certain conditions, and its only caller `ClientBase#addFileToClasspath` checks whether the value returned is `null`. We could just have the caller check on that same condition to avoid passing `null`s around.\n- In `YarnSparkHadoopUtil#addToEnvironment`, we take in an argument `classpathSeparator` that always has the same value upstream (i.e. `File.pathSeparator`). This argument is now removed from the signature and all callers of this method upstream.\n- `ClientBase#copyRemoteFile` is now renamed to `copyFileToRemote`. It was unclear whether we are copying a remote file to our local file system, or copying a locally visible file to a remote file system. Also, even the content of the method has inaccurately named variables. We use `val remoteFs` to signify the file system of the locally visible file and `val fs` to signify the remote, destination file system. These are now renamed `srcFs` and `destFs` respectively.\n- We currently log the AM container's environment and resource mappings directly as Scala collections. This is incredibly hard to read and probably too verbose for the average Spark user. In other modes (e.g. standalone), we also don't log the launch commands by default, so the logging level of these information is now set to `DEBUG`.\n- None of these classes (`Client`, `ClientBase`, `YarnSparkHadoopUtil` etc.) is intended to be used by a Spark application (the user should go through Spark submit instead). At the very least they should be `private[spark]`.\n\nAuthor: Andrew Or <andrewor14@gmail.com>\n\nCloses #2350 from andrewor14/yarn-cleanup and squashes the following commits:\n\n39e8c7b [Andrew Or] Address review comments\n6619f9b [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-cleanup\n2ca6d64 [Andrew Or] Improve logging in application monitor\na3b9693 [Andrew Or] Minor changes\n7dd6298 [Andrew Or] Simplify ClientBase#monitorApplication\n547487c [Andrew Or] Provide default values for null application report entries\na0ad1e9 [Andrew Or] Fix class not found error\n1590141 [Andrew Or] Address review comments\n45ccdea [Andrew Or] Remove usages of getAMMemory\nd8e33b6 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-cleanup\ned0b42d [Andrew Or] Fix alpha compilation error\nc0587b4 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-cleanup\n6d74888 [Andrew Or] Minor comment changes\n6573c1d [Andrew Or] Clean up, simplify and document code for setting classpaths\ne4779b6 [Andrew Or] Clean up log messages + variable naming in ClientBase\n8766d37 [Andrew Or] Heavily add documentation to Client* classes + various clean-ups\n6c94d79 [Andrew Or] Various cleanups in ClientBase and ClientArguments\nef7069a [Andrew Or] Clean up YarnClientSchedulerBackend more\n6de9072 [Andrew Or] Guard against potential NPE in debug logging mode\nfabe4c4 [Andrew Or] Reuse more code in YarnClientSchedulerBackend\n3f941dc [Andrew Or] First cut at simplifying the Client (stable and alpha)","Andrew Or",true],["11c10df825419372df61a8d23c51e8c3cc78047f","7d45abe39fd242ee2f6575da28f13d66568418b3@oss.nttdata.co.jp","[SPARK-3304] [YARN] ApplicationMaster's Finish status is wrong when uncaught exception is thrown from ReporterThread\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #2198 from sarutak/SPARK-3304 and squashes the following commits:\n\n2696237 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3304\n5b80363 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3304\n4eb0a3e [Kousuke Saruta] Remoed the description about spark.yarn.scheduler.reporterThread.maxFailure\n9741597 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3304\nf7538d4 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3304\n358ef8d [Kousuke Saruta] Merge branch 'SPARK-3304' of github.com:sarutak/spark into SPARK-3304\n0d138c6 [Kousuke Saruta] Revert \"tmp\"\nf8da10a [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3304\nb6e9879 [Kousuke Saruta] tmp\n8d256ed [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3304\n13b2652 [Kousuke Saruta] Merge branch 'SPARK-3304' of github.com:sarutak/spark into SPARK-3304\n2711e15 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3304\nc081f8e [Kousuke Saruta] Modified ApplicationMaster to handle exception in ReporterThread itself\n0bbd3a6 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-3304\na6982ad [Kousuke Saruta] Added ability handling uncaught exception thrown from Reporter thread","Kousuke Saruta",true],["66bc0f2d675d06cdd48638f124a1ff32be2bf456","c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","[SPARK-3598][SQL]cast to timestamp should be the same as hive\n\nthis patch fixes timestamp smaller than 0 and cast int as timestamp\n\nselect cast(1000 as timestamp) from src limit 1;\n\nshould return 1970-01-01 00:00:01, but we now take it as 1000 seconds.\nalso, current implementation has bug when the time is before 1970-01-01 00:00:00.\nrxin marmbrus chenghao-intel\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #2458 from adrian-wang/timestamp and squashes the following commits:\n\n4274b1d [Daoyuan Wang] set test not related to timezone\n1234f66 [Daoyuan Wang] fix timestamp smaller than 0 and cast int as timestamp","Daoyuan Wang",true],["116016b481cecbd8ad6e9717d92f977a164a6653","c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","[SPARK-3582][SQL] not limit argument type for hive simple udf\n\nSince we have moved to `ConventionHelper`, it is quite easy to avoid call `javaClassToDataType` in hive simple udf. This will solve SPARK-3582.\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #2506 from adrian-wang/spark3582 and squashes the following commits:\n\n450c28e [Daoyuan Wang] not limit argument type for hive simple udf","Daoyuan Wang",true],["3b8eefa9b843c7f1e0e8dda6023272bc9f011c5c","f4572b456463079b9d9dee14a4b6e053ed064605@huawei.com","[SPARK-3536][SQL] SELECT on empty parquet table throws exception\n\nIt returns null metadata from parquet if querying on empty parquet file while calculating splits.So added null check and returns the empty splits.\n\nAuthor : ravipesala ravindra.pesalahuawei.com\n\nAuthor: ravipesala <ravindra.pesala@huawei.com>\n\nCloses #2456 from ravipesala/SPARK-3536 and squashes the following commits:\n\n1e81a50 [ravipesala] Fixed the issue when querying on empty parquet file.","ravipesala",true],["e73b48ace0a7e0f249221240140235d33eeac36b","291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","SPARK-2745 [STREAMING] Add Java friendly methods to Duration class\n\ntdas is this what you had in mind for this JIRA? I saw this one and thought it would be easy to take care of, and helpful as I use streaming from Java.\n\nI could do the same for `Time`? Happy to do so.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #2403 from srowen/SPARK-2745 and squashes the following commits:\n\n5a9e706 [Sean Owen] Change \"Duration\" to \"Durations\" to avoid changing Duration case class API\nbda301c [Sean Owen] Just delegate to Scala binary operator syntax to avoid scalastyle warning\n7dde949 [Sean Owen] Disable scalastyle for false positives. Add Java static factory methods seconds(), minutes() to Duration. Add Java-friendly methods to Time too, and unit tests. Remove unnecessary math.floor from Time.floor()\n4dee32e [Sean Owen] Add named methods to Duration in parallel to symbolic methods for Java-friendliness. Also add unit tests for Duration, in Scala and Java.","Sean Owen",true]],"size":50,"ref":"refs/heads/mapstatus","head":"11f531961a3f37c91563fae93999e3f7baa98788"}