Repository: ray-project/ray

Issue:
MLFlow parameter error using rllib agents
<!--
General questions should be asked on the mailing list ray-dev@googlegroups.com.
Questions about how to use Ray should be asked on
[StackOverflow](https://stackoverflow.com/questions/tagged/ray).

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: MacOS 10.14.6
- **Ray installed from (source or binary)**: source
- **Ray version**: 0.7.5
- **Python version**: 3.7.3
- **Exact command to reproduce**:

<!--
You can obtain the Ray version with

python -c "import ray; print(ray.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

When I tried to integrate MLFlow to my experiment that uses a DQN agent, I get the following error:
```
    Trainable.__init__(self, config, logger_creator)
  File "/python3.7/site-packages/ray/tune/trainable.py", line 96, in __init__
    self._setup(copy.deepcopy(self.config))
  File "/python3.7/site-packages/ray/rllib/agents/trainer.py", line 478, in _setup
    self._allow_unknown_subkeys)
  File "/python3.7/site-packages/ray/tune/util.py", line 160, in deep_update
    raise Exception("Unknown config parameter `{}` ".format(k))
Exception: Unknown config parameter mlflow_experiment_id 
```

It seems like the agent's config is quite frozen, and there doesn't seem to be an explicit way to allow new keys to that config.

When I digged a bit in the code, I found, in the `rllib/agents/trainer.py` that there's an argument to the deep_update function that merges configs together called `allow_unknown_configs` and that is by default set to False. When I set it to True, for testing purposes, it worked.
So maybe we should add an abstraction layer to that so that it can easily be set from the experiment itself.

### Source code / logs

This is part of the source code of when I ran the experiment on a DQN agent. I got the error above

```
   tune.run(
        ...
        config={
            "mlflow_experiment_id": experiment_id,
            ...
    )
```



Discussion:
I have a similar, but not MLflow related, problem. The basic question is the same: How do you pass arguments via the config if the agent does not allow additional config values.

Maybe an abstraction layer to allow additional parameters makes sense.
Or maybe allowing a field like `"custom_config"` in the config dict should be allowed all the time.


I have the same issue, I have to hack the code in trainer.py 
put `_allow_unknown_configs=True`. Only in this way I am able to use MLflow 
@fatenghriss  

Resolving patch:
diff --git a/python/ray/tune/examples/mlflow_example.py b/python/ray/tune/examples/mlflow_example.py
--- a/python/ray/tune/examples/mlflow_example.py
+++ b/python/ray/tune/examples/mlflow_example.py
@@ -41,7 +41,9 @@ def easy_objective(config):
         num_samples=5,
         loggers=DEFAULT_LOGGERS + (MLFLowLogger, ),
         config={
-            "mlflow_experiment_id": experiment_id,
+            "logger_config": {
+                "mlflow_experiment_id": experiment_id,
+            },
             "width": tune.sample_from(
                 lambda spec: 10 + int(90 * random.random())),
             "height": tune.sample_from(lambda spec: int(100 * random.random()))
diff --git a/python/ray/tune/logger.py b/python/ray/tune/logger.py
--- a/python/ray/tune/logger.py
+++ b/python/ray/tune/logger.py
@@ -78,9 +78,12 @@ class MLFLowLogger(Logger):
     """
 
     def _init(self):
+        logger_config = self.config.get("logger_config", {})
         from mlflow.tracking import MlflowClient
-        client = MlflowClient()
-        run = client.create_run(self.config.get("mlflow_experiment_id"))
+        client = MlflowClient(
+            tracking_uri=logger_config.get("mlflow_tracking_uri"),
+            registry_uri=logger_config.get("mlflow_registry_uri"))
+        run = client.create_run(logger_config.get("mlflow_experiment_id"))
         self._run_id = run.info.run_id
         for key, value in self.config.items():
             client.log_param(self._run_id, key, value)
diff --git a/rllib/agents/trainer.py b/rllib/agents/trainer.py
--- a/rllib/agents/trainer.py
+++ b/rllib/agents/trainer.py
@@ -370,6 +370,10 @@
         "replay_mode": "independent",
     },
 
+    # === Logger ===
+    # Define logger-specific configuration to be used inside Logger
+    "logger_config": {},
+
     # === Replay Settings ===
     # The number of contiguous environment steps to replay at once. This may
     # be set to greater than 1 to support recurrent models.


Test patch: