[SPARK-3216] Spark-shell is broken in branch-1.0

This is an exact copy of #1825 for branch-1.0, which was originally posted by @sarutak. Thanks @brkyvz for reporting this issue.

This is caused by #1831, which is a fix that broke spark-shell. The follow-up fix in #1825, however, was only merged into master and branch-1.1, but not branch-1.0.

Tested locally.