Repository: ray-project/ray

Issue:
[rllib] Weights & Biases logger cannot handle objects in configuration
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->

### What is the problem?

The Weights & Biases logger cannot handle object references in RLlib configurations, for example in the callback API.

```
Process _WandbLoggingProcess-1:
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "[...]/ray/tune/integration/wandb.py", line 127, in run
    wandb.init(*self.args, **self.kwargs)
  File "[...]/wandb/__init__.py", line 1303, in init
    as_defaults=not allow_val_change)
  File "[...]/wandb/wandb_config.py", line 333, in _update
    self.persist()
  File "[...]/wandb/wandb_config.py", line 238, in persist
    conf_file.write(str(self))
  File "[...]/wandb/wandb_config.py", line 374, in __str__
    allow_unicode=True, encoding='utf-8')
  File "[...]/yaml/__init__.py", line 290, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
  File "[...]/yaml/__init__.py", line 278, in dump_all
    dumper.represent(data)
  File "[...]/yaml/representer.py", line 27, in represent
    node = self.represent_data(data)
  File "[...]/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "[...]/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "[...]/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "[...]/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "[...]/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "[...]/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "[...]/yaml/representer.py", line 58, in represent_data
    node = self.yaml_representers[None](self, data)
  File "[...]/yaml/representer.py", line 231, in represent_undefined
    raise RepresenterError("cannot represent an object", data)
yaml.representer.RepresenterError: ('cannot represent an object', <class '__main__.MyCallbacks'>)
```
*Ray version and other system information (Python version, TensorFlow version, OS):*

- Ray 0.8.7
- Ubuntu 18.04
- Python 3.7

### Reproduction (REQUIRED)
Please provide a script that can be run to reproduce the issue. The script should have **no external library dependencies** (i.e., use fake or mock data / environments):

```
from ray import tune
from ray.rllib.agents.ppo import PPOTrainer
from ray.rllib.agents.callbacks import DefaultCallbacks
from ray.tune.integration.wandb import WandbLogger

class MyCallbacks(DefaultCallbacks):
    def on_episode_end(self, worker, base_env, policies, episode, **kwargs):
        print("Episode ended")

tune.run(
    PPOTrainer,
    checkpoint_freq=1,
    config={
        "framework": "torch",
        "num_workers": 8,
        "num_gpus": 1,
        "env": "CartPole-v0",
        "callbacks": MyCallbacks,
        "logger_config": {
          "wandb": {
              "project": "test",
              "api_key_file": "./wandb_api_key_file",
          }
        }
    },
    stop={
        "training_iteration":10
    },
    loggers=[WandbLogger]
)
```

If we cannot run your script, we cannot fix your issue.

- [X] I have verified my script runs in a clean environment and reproduces the issue.
- [X] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).



Discussion:
I noticed that the callback is already [explicitly filtered out in the result handling](https://github.com/ray-project/ray/blob/0aec4cbccb5862a7808f98b90696236da9e2a03e/python/ray/tune/integration/wandb.py#L159), but of course that doesn't help during initialization. Is there any reason why objects shouldn't be generally filtered out? Probably only numbers and strings make sense to log anyway?
cc @krfricke 
Thanks. For the moment I submitted a [tiny PR](https://github.com/ray-project/ray/pull/10441) to fix the callback API specifically. I agree it makes sense to filter out invalid values, I'll get back to this soon.
@krfricke  I am having a similar problem trying to use APPO (PPO works fine). See https://pastebin.com/SdpNp0Pq 
Hey @janblumenkamp and @rhamnett this should be fixed via #10654. If this issue prevails, feel free to re-open!
@krfricke @richardliaw Afraid that I am still seeing the same error when running an APPO tune job
Hi @rhamnett, did you install the latest nightly ray release? Do you have a script to reproduce the error?
@krfricke yes I am familiar with re-installing the nightly and can confirm that I have done that correctly. My script is lengthy and contains private intellectual property, so I am wondering if you can just simply adapt a PPO test you might have to APPO (should be trivial) and then let me know if you can re-produce?

If after trying that, you can't reproduce, I will seek to dedicate some time to give you an example script.

Resolving patch:
diff --git a/python/ray/tune/integration/wandb.py b/python/ray/tune/integration/wandb.py
--- a/python/ray/tune/integration/wandb.py
+++ b/python/ray/tune/integration/wandb.py
@@ -9,6 +9,8 @@
 from ray.tune.logger import Logger
 from ray.tune.utils import flatten_dict
 
+import yaml
+
 try:
     import wandb
 except ImportError:
@@ -29,6 +31,12 @@ def _clean_log(obj):
     # Else
     try:
         pickle.dumps(obj)
+        yaml.dump(
+            obj,
+            Dumper=yaml.SafeDumper,
+            default_flow_style=False,
+            allow_unicode=True,
+            encoding="utf-8")
         return obj
     except Exception:
         # give up, similar to _SafeFallBackEncoder


Test patch: