[SPARK-3114] [PySpark] Fix Python UDFs in Spark SQL.

This fixes SPARK-3114, an issue where we inadvertently broke Python UDFs in Spark SQL.

This PR modifiers the test runner script to always run the PySpark SQL tests, irrespective of whether SparkSQL itself has been modified.  It also includes Davies' fix for the bug.

Closes #2026.

Author: Josh Rosen <joshrosen@apache.org>
Author: Davies Liu <davies.liu@gmail.com>

Closes #2027 from JoshRosen/pyspark-sql-fix and squashes the following commits:

9af2708 [Davies Liu] bugfix: disable compression of command
0d8d3a4 [Josh Rosen] Always run Python Spark SQL tests.
[SPARK-3116] Remove the excessive lockings in TorrentBroadcast

Author: Reynold Xin <rxin@apache.org>

Closes #2028 from rxin/torrentBroadcast and squashes the following commits:

92c62a5 [Reynold Xin] Revert the MEMORY_AND_DISK_SER changes.
03a5221 [Reynold Xin] [SPARK-3116] Remove the excessive lockings in TorrentBroadcast
Fix typo in decision tree docs

Candidate splits were inconsistent with the example.

Author: Matt Forbes <matt@tellapart.com>

Closes #1837 from emef/tree-doc and squashes the following commits:

3be14a1 [Matt Forbes] Fix typo in decision tree docs
Remove "client mode" boolean argument

... and then make SparkClassLauncher private[spark].
Fix precedence of library paths, classpath, java opts and memory

This was previously broken because of the way we pass command line
arguments. As of this commit, the ordering becomes:

SPARK_SUBMIT_DRIVER_MEMORY > spark.driver.memory > SPARK_DRIVER_MEMORY
SPARK_SUBMIT_CLASSPATH > spark.driver.extraClassPath
SPARK_SUBMIT_LIBRARY_PATH > spark.driver.extraLibraryPath
SPARK_SUBMIT_JAVA_OPTS > spark.driver.extraJavaOpts

We achieve this by passing existing environment variables to
SparkClassLauncher directly.
SparkClassLauncher -> SparkSubmitDriverBootstrapper
Merge branch 'master' of github.com:apache/spark into handle-configs-bash