[SPARK-3696]Do not override the user-difined conf_dir

https://issues.apache.org/jira/browse/SPARK-3696

We see if SPARK_CONF_DIR is already defined before assignment.