[SPARK-2098] All Spark processes should support spark-defaults.conf, config file

This is another implementation about #1256
cc @andrewor14 @vanzin
I believe we deprecated being able to specify the log directory through command line. Maybe we shouldn't document this. Can you confirm @vanzin