[SPARK-1341] [Streaming] Throttle BlockGenerator to limit rate of data consumption.

Maybe. The Jenkins being often too slow at doing things, it maybe causing
the data rate in this test to be lower than expected.




On Fri, Jul 18, 2014 at 5:29 PM, Kay Ousterhout <notifications@github.com>
wrote:

> In
> streaming/src/test/scala/org/apache/spark/streaming/NetworkReceiverSuite.scala:
>
> > +      blockGenerator += count
> > +      generatedData += count
> > +      count += 1
> > +      Thread.sleep(1)
> > +    }
> > +    blockGenerator.stop()
> > +
> > +    val recordedData = blockGeneratorListener.arrayBuffers
> > +    assert(blockGeneratorListener.arrayBuffers.size > 0)
> > +    assert(recordedData.flatten.toSet === generatedData.toSet)
> > +    // recordedData size should be close to the expected rate
> > +    assert(recordedData.flatten.size >= expectedMessages * 0.9 &&
> > +      recordedData.flatten.size <= expectedMessages * 1.1 )
> > +    // the first and last block may be incomplete, so we slice them out
> > +    recordedData.slice(1, recordedData.size - 1).foreach { block =>
> > +      assert(block.size >= expectedMessagesPerBlock * 0.8 &&
>
> This line is failing on Jenkins for an (I think) unrelated change (
> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16838/consoleFull)
> -- any change this is flaky?
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/945/files#r15140315>.
>