Repository: wandb/wandb

Issue:
[Q] When I try to use transformer. I get a problem "PermissionError: [Errno 13] Permission denied: '/var/run/secrets/kubernetes.io/serviceaccount/token'",How can I fix it？
settings["docker"] = wandb.env.get_docker(wandb.util.image_id_from_k8s())
  File "/lib/python3.9/site-packages/wandb/util.py", line 1250, in image_id_from_k8s
    headers={"Authorization": f"Bearer {open(token_path).read()}"},
PermissionError: [Errno 13] Permission denied: '/var/run/secrets/kubernetes.io/serviceaccount/token'


Discussion:
Hi @gddange thanks for reporting this issue. 

Would it be possible that you provide some additional context, where are you running the transformer? Could you share with us a minimal code example, so that we could debug further?

From the error that you've mentioned, it appears you get a `PermissionError`. Are you a root user in this container? Are you mounting a k8s secret, and what are the token permissions? 
> Hi @gddange thanks for reporting this issue.
> 
> Would it be possible that you provide some additional context, where are you running the transformer? Could you share with us a minimal code example, so that we could debug further?
> 
> From the error that you've mentioned, it appears you get a `PermissionError`. Are you a root user in this container? Are you mounting a k8s secret, and what are the token permissions?

HI,  I am trying to finetune a llama model using the toolbox mentioned in this Github repository: https://github.com/OptimalScale/LMFlow. As a non-root user, I'm unable to obtain root user permissions. The code is being run on a x86_64 GNU/Linux system.
The trace are:
Traceback (most recent call last):
  File "LMFlow/examples/finetune.py", line 70, in <module>
    main()
  File "LMFlow/examples/finetune.py", line 66, in main
    tuned_model = finetuner.tune(model=model, lm_dataset=lm_dataset)
  File "LMFlow/src/lmflow/pipeline/finetuner.py", line 232, in tune
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/transfomers/src/transformers/trainer.py", line 1639, in train
    return inner_training_loop(
  File "/transfomers/src/transformers/trainer.py", line 1822, in _inner_training_loop
    self.control = self.callback_handler.on_train_begin(args, self.state, self.control)
  File "/transfomers/src/transformers/trainer_callback.py", line 353, in on_train_begin
    return self.call_event("on_train_begin", args, state, control)
  File "/transfomers/src/transformers/trainer_callback.py", line 397, in call_event
    result = getattr(callback, event)(
  File "//transfomers/src/transformers/integrations.py", line 764, in on_train_begin
    self.setup(args, state, model, **kwargs)
  File "/transfomers/src/transformers/integrations.py", line 738, in setup
    self._wandb.init(
  File "/lib/python3.9/site-packages/wandb/sdk/wandb_init.py", line 1181, in init
    raise Exception("problem") from error_seen
Exception: problem
Hi @gddange thanks for the additional information. Could you please first try to login in your CLI with `wand login` and then `wandb init` commands? do these work for you?

Resolving patch:
diff --git a/wandb/util.py b/wandb/util.py
--- a/wandb/util.py
+++ b/wandb/util.py
@@ -1050,7 +1050,21 @@ def image_id_from_k8s() -> Optional[str]:
           fieldPath: metadata.namespace
     """
     token_path = "/var/run/secrets/kubernetes.io/serviceaccount/token"
-    if os.path.exists(token_path):
+    try:
+        with open(token_path) as token_file:
+            token = token_file.read()
+    except FileNotFoundError:
+        wandb.termwarn(f"Token file not found at {token_path}.")
+        return None
+    except PermissionError as e:
+        current_uid = os.getuid()
+        wandb.termwarn(
+            f"Unable to read the token file at {token_path} due to permission error ({e})."
+            + f"The current user id is {current_uid}. Consider changing the securityContext to run the container as the current user."
+        )
+        return None
+
+    if token:
         k8s_server = "https://{}:{}/api/v1/namespaces/{}/pods/{}".format(
             os.getenv("KUBERNETES_SERVICE_HOST"),
             os.getenv("KUBERNETES_PORT_443_TCP_PORT"),
@@ -1062,7 +1076,7 @@ def image_id_from_k8s() -> Optional[str]:
                 k8s_server,
                 verify="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
                 timeout=3,
-                headers={"Authorization": f"Bearer {open(token_path).read()}"},
+                headers={"Authorization": f"Bearer {token}"},
             )
             res.raise_for_status()
         except requests.RequestException:


Test patch: