[SPARK-3399][PySpark] Test for PySpark should ignore HADOOP_CONF_DIR and YARN_CONF_DIR

If this problem only happen during testing, could we put these in python/run-tests? pyspark will be often used as python shell.