Repository: intel/ipex-llm
Pull request: change default hadoop user name from root to os.env[HADOOP_USER_NAME] or current user
## Description

change default hadoop user name in init_nncontext_on_yarn

### 1. Why the change?

fixed #7905 

### 2. User API changes

None

### 3. Summary of the change 

change default hadoop user name in init_nncontext_on_yarn

### 4. How to test?
- [ ] N/A
- [ ] Unit test
- [ ] Application test
- [ ] Document test
- [ ] ...



Problem statement:
default to user 'root' when init_nncontext with yarn-cluster and yarn-local modes
I just noticed that bigdl seems to default to 'root' when init_nncontext is called with yarn-cluster and yarn-local modes. That leads to be permission errors if the actual user is not root. This does not happen when using 'local' or 'spark-submit' for init_nncontext(). 

Along with the same context, what is the proper configuration place to change application settings, such as the username, the working path, the number of executors, executor memory, driver memory, and the number of slaves used in Spark/Yarn.

Hints from issue discussion:
Yes, it's really not good to use `root` as default. We will fix it.
Hi @yuwgit Thanks for pointing out, we will fix it.
As a current workaround, you can specify the hadoop_user_name argument in init_orca_context to be your current user (which is default to be root...)
```
init_orca_context(..., hadoop_user_name="...")
```
You may also see [Orca Yarn Tutorial](https://bigdl.readthedocs.io/en/latest/doc/Orca/Tutorial/yarn.html#) to learn how to use `init_orca_context` as a workaround currently.

Commit message: update hadoop user name

Patch to review:
diff --git a/python/dllib/src/bigdl/dllib/nncontext.py b/python/dllib/src/bigdl/dllib/nncontext.py
index 426b72922d1..98c0b0b48a5 100644
--- a/python/dllib/src/bigdl/dllib/nncontext.py
+++ b/python/dllib/src/bigdl/dllib/nncontext.py
@@ -65,7 +65,7 @@ def init_spark_on_yarn(hadoop_conf,
                        extra_python_lib=None,
                        penv_archive=None,
                        additional_archive=None,
-                       hadoop_user_name="root",
+                       hadoop_user_name=None,
                        spark_yarn_archive=None,
                        spark_log_level="WARN",
                        redirect_spark_log=True,
@@ -93,7 +93,8 @@ def init_spark_on_yarn(hadoop_conf,
            Default to be None.
     :param additional_archive: Comma-separated list of additional archives to be uploaded and
            unpacked on executors. Default to be None.
-    :param hadoop_user_name: The user name for running the yarn cluster. Default to be 'root'.
+    :param hadoop_user_name: The user name for running the yarn cluster. Default to be None.
+           None means os.environ["HADOOP_USER_NAME"], or current user if HADOOP_USER_NAME is unset.
     :param spark_yarn_archive: Conf value for setting spark.yarn.archive. Default to be None.
     :param spark_log_level: The log level for Spark. Default to be 'WARN'.
     :param redirect_spark_log: Whether to redirect the Spark log to local file. Default to be True.
@@ -140,7 +141,7 @@ def init_spark_on_yarn_cluster(hadoop_conf,
                                extra_python_lib=None,
                                penv_archive=None,
                                additional_archive=None,
-                               hadoop_user_name="root",
+                               hadoop_user_name=None,
                                spark_yarn_archive=None,
                                spark_log_level="WARN",
                                redirect_spark_log=True,
@@ -168,7 +169,8 @@ def init_spark_on_yarn_cluster(hadoop_conf,
            Default to be None.
     :param additional_archive: Comma-separated list of additional archives to be uploaded and
            unpacked on executors. Default to be None.
-    :param hadoop_user_name: The user name for running the yarn cluster. Default to be 'root'.
+    :param hadoop_user_name: The user name for running the yarn cluster. Default to be None.
+           None means os.environ["HADOOP_USER_NAME"], or current user if HADOOP_USER_NAME is unset.
     :param spark_yarn_archive: Conf value for setting spark.yarn.archive. Default to be None.
     :param spark_log_level: The log level for Spark. Default to be 'WARN'.
     :param redirect_spark_log: Whether to redirect the Spark log to local file. Default to be True.
diff --git a/python/dllib/src/bigdl/dllib/utils/spark.py b/python/dllib/src/bigdl/dllib/utils/spark.py
index 7295d26fefe..c6f5b4d8af0 100644
--- a/python/dllib/src/bigdl/dllib/utils/spark.py
+++ b/python/dllib/src/bigdl/dllib/utils/spark.py
@@ -77,7 +77,7 @@ def init_spark_on_yarn(self,
                            extra_python_lib=None,
                            penv_archive=None,
                            additional_archive=None,
-                           hadoop_user_name="root",
+                           hadoop_user_name=None,
                            spark_yarn_archive=None,
                            conf=None,
                            jars=None,
@@ -85,7 +85,10 @@ def init_spark_on_yarn(self,
         print("Initializing SparkContext for yarn-client mode")
         executor_python_env = "python_env"
         os.environ["HADOOP_CONF_DIR"] = hadoop_conf
-        os.environ["HADOOP_USER_NAME"] = hadoop_user_name
+        if hadoop_user_name:
+            os.environ["HADOOP_USER_NAME"] = hadoop_user_name
+        elif not os.environ.get("HADOOP_USER_NAME"):
+            os.environ["HADOOP_USER_NAME"] = os.getlogin()
         os.environ["PYSPARK_PYTHON"] = "{}/bin/python".format(executor_python_env)
 
         pack_env = False
@@ -149,7 +152,7 @@ def init_spark_on_yarn_cluster(self,
                                    extra_python_lib=None,
                                    penv_archive=None,
                                    additional_archive=None,
-                                   hadoop_user_name="root",
+                                   hadoop_user_name=None,
                                    spark_yarn_archive=None,
                                    conf=None,
                                    jars=None,
@@ -157,7 +160,10 @@ def init_spark_on_yarn_cluster(self,
         print("Initializing job for yarn-cluster mode")
         executor_python_env = "python_env"
         os.environ["HADOOP_CONF_DIR"] = hadoop_conf
-        os.environ["HADOOP_USER_NAME"] = hadoop_user_name
+        if hadoop_user_name:
+            os.environ["HADOOP_USER_NAME"] = hadoop_user_name
+        elif not os.environ.get("HADOOP_USER_NAME"):
+            os.environ["HADOOP_USER_NAME"] = os.getlogin()
 
         pack_env = False
         invalidInputError(penv_archive or conda_name,


Code review comments:
diff_hunk: @@ -93,7 +93,8 @@ def init_spark_on_yarn(hadoop_conf,
            Default to be None.
     :param additional_archive: Comma-separated list of additional archives to be uploaded and
            unpacked on executors. Default to be None.
-    :param hadoop_user_name: The user name for running the yarn cluster. Default to be 'root'.
+    :param hadoop_user_name: The user name for running the yarn cluster. Default to be None.
+           None means os.environ["HADOOP_USER_NAME"], or current user if HADOOP_USER_NAME is unset.
path: python/dllib/src/bigdl/dllib/nncontext.py
text: @user1:
Change to `Default to be None, which will use the system configured user.`?