[SPARK-3216] [SPARK-3232] Spark-shell is broken in branch-1.0 / Backport SPARK-3006 into branch-1.0

Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Author: Andrew Or <andrewor14@gmail.com>

Closes #2136 from sarutak/SPARK-3216 and squashes the following commits:

5c20d2d [Kousuke Saruta] Merge branch 'branch-1.0' of git://git.apache.org/spark into SPARK-3216
af0517f [Kousuke Saruta] Modified spark-shell.cmd, backporting SPARK-3006
bbc7221 [Kousuke Saruta] Replace 4-space into 2-space
15cd980 [Andrew Or] Fix spark-shell in branch-1.0