GitHub PushEvent at 2014-11-05 11:00:40
Repository: apache-spark
Actor: ueshin
Payload: {"shas":[["f225b3cc18698b2ee8a94c8ffa0b6aca2fce7cf9","c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","[SPARK-3964] [MLlib] [PySpark] add Hypothesis test Python API\n\n```\npyspark.mllib.stat.StatisticschiSqTest(observed, expected=None)\n    :: Experimental ::\n\n    If `observed` is Vector, conduct Pearson's chi-squared goodness\n    of fit test of the observed data against the expected distribution,\n    or againt the uniform distribution (by default), with each category\n    having an expected frequency of `1 / len(observed)`.\n    (Note: `observed` cannot contain negative values)\n\n    If `observed` is matrix, conduct Pearson's independence test on the\n    input contingency matrix, which cannot contain negative entries or\n    columns or rows that sum up to 0.\n\n    If `observed` is an RDD of LabeledPoint, conduct Pearson's independence\n    test for every feature against the label across the input RDD.\n    For each feature, the (feature, label) pairs are converted into a\n    contingency matrix for which the chi-squared statistic is computed.\n    All label and feature values must be categorical.\n\n    :param observed: it could be a vector containing the observed categorical\n                     counts/relative frequencies, or the contingency matrix\n                     (containing either counts or relative frequencies),\n                     or an RDD of LabeledPoint containing the labeled dataset\n                     with categorical features. Real-valued features will be\n                     treated as categorical for each distinct value.\n    :param expected: Vector containing the expected categorical counts/relative\n                     frequencies. `expected` is rescaled if the `expected` sum\n                     differs from the `observed` sum.\n    :return: ChiSquaredTest object containing the test statistic, degrees\n             of freedom, p-value, the method used, and the null hypothesis.\n```\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #3091 from davies/his and squashes the following commits:\n\n145d16c [Davies Liu] address comments\n0ab0764 [Davies Liu] fix float\n5097d54 [Davies Liu] add Hypothesis test Python API\n\n(cherry picked from commit c8abddc5164d8cf11cdede6ab3d5d1ea08028708)\nSigned-off-by: Xiangrui Meng <meng@databricks.com>","Davies Liu",true],["46654b0661257f432932c6efc09c4c0983521834","4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","[SPARK-4029][Streaming] Update streaming driver to reliably save and recover received block metadata on driver failures\n\nAs part of the initiative of preventing data loss on driver failure, this JIRA tracks the sub task of modifying the streaming driver to reliably save received block metadata, and recover them on driver restart.\n\nThis was solved by introducing a `ReceivedBlockTracker` that takes all the responsibility of managing the metadata of received blocks (i.e. `ReceivedBlockInfo`, and any actions on them (e.g, allocating blocks to batches, etc.). All actions to block info get written out to a write ahead log (using `WriteAheadLogManager`). On recovery, all the actions are replaying to recreate the pre-failure state of the `ReceivedBlockTracker`, which include the batch-to-block allocations and the unallocated blocks.\n\nFurthermore, the `ReceiverInputDStream` was modified to create `WriteAheadLogBackedBlockRDD`s when file segment info is present in the `ReceivedBlockInfo`. After recovery of all the block info (through recovery `ReceivedBlockTracker`), the `WriteAheadLogBackedBlockRDD`s gets recreated with the recovered info, and jobs submitted. The data of the blocks gets pulled from the write ahead logs, thanks to the segment info present in the `ReceivedBlockInfo`.\n\nThis is still a WIP. Things that are missing here are.\n\n- *End-to-end integration tests:* Unit tests that tests the driver recovery, by killing and restarting the streaming context, and verifying all the input data gets processed. This has been implemented but not included in this PR yet. A sneak peek of that DriverFailureSuite can be found in this PR (on my personal repo): https://github.com/tdas/spark/pull/25 I can either include it in this PR, or submit that as a separate PR after this gets in.\n\n- *WAL cleanup:* Cleaning up the received data write ahead log, by calling `ReceivedBlockHandler.cleanupOldBlocks`. This is being worked on.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #3026 from tdas/driver-ha-rbt and squashes the following commits:\n\na8009ed [Tathagata Das] Added comment\n1d704bb [Tathagata Das] Enabled storing recovered WAL-backed blocks to BM\n2ee2484 [Tathagata Das] More minor changes based on PR\n47fc1e3 [Tathagata Das] Addressed PR comments.\n9a7e3e4 [Tathagata Das] Refactored ReceivedBlockTracker API a bit to make things a little cleaner for users of the tracker.\naf63655 [Tathagata Das] Minor changes.\nfce2b21 [Tathagata Das] Removed commented lines\n59496d3 [Tathagata Das] Changed class names, made allocation more explicit and added cleanup\n19aec7d [Tathagata Das] Fixed casting bug.\nf66d277 [Tathagata Das] Fix line lengths.\ncda62ee [Tathagata Das] Added license\n25611d6 [Tathagata Das] Minor changes before submitting PR\n7ae0a7fb [Tathagata Das] Transferred changes from driver-ha-working branch\n\n(cherry picked from commit 5f13759d3642ea5b58c12a756e7125ac19aff10e)\nSigned-off-by: Tathagata Das <tathagata.das1565@gmail.com>","Tathagata Das",true]],"size":2,"ref":"refs/heads/branch-1.2","head":"46654b0661257f432932c6efc09c4c0983521834"}