[SPARK-4384] [PySpark] improve sort spilling

If there some big broadcasts (or other object) in Python worker, the free memory could be used for sorting will be too small, then it will keep spilling small files into disks, finally failed with too many open files.

This PR try to delay the spilling until the used memory goes over limit and start to increase since last spilling, it will increase the size of spilling files, improve the stability and performance in this cases. (We also do this in ExternalAggregator).

Author: Davies Liu <davies@databricks.com>

Closes #3252 from davies/sort and squashes the following commits:

711fb6c [Davies Liu] improve sort spilling

(cherry picked from commit 73c8ea84a668f443eb18ce15ba97023da041d808)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
[DOC][PySpark][Streaming] Fix docstring for sphinx

This commit should be merged for 1.2 release.
cc tdas

Author: Ken Takagiwa <ugw.gi.world@gmail.com>

Closes #3311 from giwa/patch-3 and squashes the following commits:

ab474a8 [Ken Takagiwa] [DOC][PySpark][Streaming] Fix docstring for sphinx

(cherry picked from commit 9b7bbcef8863ecd69e7511825ef9c93d8632dac2)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
[SPARK-4294][Streaming] UnionDStream stream should express the requirements in the same way as TransformedDStream

In class TransformedDStream:
```scala
require(parents.length > 0, "List of DStreams to transform is empty")
require(parents.map(.ssc).distinct.size == 1, "Some of the DStreams have different contexts")
require(parents.map(.slideDuration).distinct.size == 1,
"Some of the DStreams have different slide durations")
```

In class UnionDStream:
```scala
if (parents.length == 0)
{ throw new IllegalArgumentException("Empty array of parents") }
if (parents.map(.ssc).distinct.size > 1)
{ throw new IllegalArgumentException("Array of parents have different StreamingContexts") }
if (parents.map(.slideDuration).distinct.size > 1)
{ throw new IllegalArgumentException("Array of parents have different slide times") }
```

The function is the same, but the realization is not. I think they shoule be the same.

Author: Yadong Qi <qiyadong2010@gmail.com>

Closes #3152 from watermen/bug-fix1 and squashes the following commits:

ed66db6 [Yadong Qi] Change transform to union
b6b3b8b [Yadong Qi] The same function should have the same realization.

(cherry picked from commit c3002c4a61c4fc5b966aa384c41c3cba33de0aa6)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>