Repository: ray-project/ray
Pull request: fix: WandbLogger crashing silently on a FileNotFoundError
Fixes: #50307
  - #50307

Problem statement:
[Air] WandB logger / _WandbLoggingActor crashes silently when logging a video with relative path or if it cannot find the given file.
### What happened + What you expected to happen

I want to log a video to wandb that is stored as a file, what I did was roughly.

```
python
from wandb import Video

result["video"] = Video("a_file", format="mp4")
train.report(result)
```

I just did this in a custom `WandbLoggerCallback.log_trial_result` function.

As a result the WandB logger **did not only not log my video but no further metrics at all**. I then realized that no final output of the `_WandbLoggingActor` was shown and I assume something has crashed it without logging an error.

Finally, I could solve the problem with using an absolute path for the video:
```python
result["video"] = Video(os.path.abspath("a_file"), format="mp4")
```

I strongly assume that there is a silent `FileNotFoundError` underneath that is not logged.

### Versions / Dependencies

ray, version 2.38.0
python3.10
Ubuntu
wandb, version 0.19.5

### Reproduction script

Setup a Tuner with a WandbLoggerCallback.

In the trainable

```python
from wandb import Video

def trainable(params):
    for i in range(19):
         result = {}
         result["video"] = Video("a_file", format="mp4")  # for the silent error id does not matter that the file does not exist
         train.report(result)
```

### Issue Severity

Medium: It is a significant difficulty but I can work around it.

Hints from issue discussion:

Commit message: Clarify effect of WandB error

Signed-off-by: Daraan <github.blurry@9ox.net>

Patch to review:
diff --git a/python/ray/air/integrations/wandb.py b/python/ray/air/integrations/wandb.py
index dceeca29f46b2..2a4f177263c92 100644
--- a/python/ray/air/integrations/wandb.py
+++ b/python/ray/air/integrations/wandb.py
@@ -420,6 +420,12 @@ def run(self):
                 # Ignore HTTPError. Missing a few data points is not a
                 # big issue, as long as things eventually recover.
                 logger.warning("Failed to log result to w&b: {}".format(str(e)))
+            except FileNotFoundError as e:
+                logger.error(
+                    "FileNotFoundError: This result will not be logged to WandB. "
+                    "Possible reason: relative instead of absolute path for file %s",
+                    e,
+                )
         self._wandb.finish()
 
     def _handle_checkpoint(self, checkpoint_path: str):


Code review comments:
diff_hunk: @@ -420,6 +420,12 @@ def run(self):
                 # Ignore HTTPError. Missing a few data points is not a
                 # big issue, as long as things eventually recover.
                 logger.warning("Failed to log result to w&b: {}".format(str(e)))
+            except FileNotFoundError as e:
+                logger.error(
+                    "FileNotFoundError: This result will not be logged to WandB. "
+                    "Possible reason: relative instead of absolute path for file %s",
+                    e,
+                )
path: python/ray/air/integrations/wandb.py
text: @user1:
```suggestion
            except FileNotFoundError as e:
                logger.error(
                    f"FileNotFoundError: Did not log result to Weights & Biases. "
                    f"Possible cause: relative file path used instead of an absolute path. "
                    f"Error: {e}"
                )
```

@author:
Thank you. I incorporated you wording your wording. Concerning the f-string I am bit reluctant to using them as they go against [flake8 G004](https://docs.astral.sh/ruff/rules/logging-f-string/) so I left them out for now.