fixed


On Mon, Feb 25, 2013 at 3:25 PM, Tathagata Das <notifications@github.com>wrote:

> In docs/streaming-programming-guide.md:
>
> >
> > -Checkpointing prevents extra computation during both failure recovery and normal operations by transparently truncating this lineage. Note that checkpointing also incurs the cost of saving to HDFS which may cause the corresponding batch to take longer to process. Hence, the interval of checkpointing needs to be set carefully. At small batch sizes (say 1 second), checkpointing every batch may significantly reduce operation throughput. Conversely, checkpointing too slowly causes the lineage and task sizes to grow which may have detrimental effects. Typically, a checkpoint interval of 5 - 10 times of sliding interval of a DStream is good setting to try.
> > +Because stateful operations have a dependency on previous batches of data, they continuously accumulate metadata over time. To clear this meta-data, streaming supports periodic _checkpointing_ by saving intermediate data to HDFS. Note that checkpointing also incurs the cost of saving to HDFS which may cause the corresponding batch to take longer to process. Hence, the interval of checkpointing needs to be set carefully. At small batch sizes (say 1 second), checkpointing every batch may significantly reduce operation throughput. Conversely, checkpointing too slowly causes the lineage and task sizes to grow which may have detrimental effects. Typically, a checkpoint interval of 5 - 10 times of sliding interval of a DStream is good setting to try.
>
> damn. meta-data !?!? :(
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/mesos/spark/pull/500/files#r3145034>
> .
>