[Spark] RDD take() method: overestimate too much

In the comment (Line 1083), it says: "Otherwise, interpolate the number of partitions we need to try, but overestimate it by 50%."

`(1.5 * num * partsScanned / buf.size).toInt` is the guess of "num of total partitions needed". In every iteration, we should consider the increment `(1.5 * num * partsScanned / buf.size).toInt - partsScanned`
Existing implementation 'exponentially' grows `partsScanned ` ( roughly: `x_{n+1} >= (1.5 + 1) x_n`)

This could be a performance problem. (unless this is the intended behavior)


If we assume some 'uniformness' in the distribution of items over partitions, then in your example, we probably have to try 1000 partitions. (uniformness --> linear prediction).  However, since I am really new to Spark, I have little idea about such distribution.

That being said, having a upper bound also makes sense. I am wondering how other people think about it.