[SPARK-3661] Respect spark.*.memory in cluster mode

This also includes minor re-organization of the code. Tested locally in both client and deploy modes.
the javadoc of `mergeSparkProperties` seems wrong now. It should be called `mergeDefaultProperties` now.