Repository: ray-project/ray

Issue:
[rllib] Python 3.5 compaltibility
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->

### What is the problem?
In the fresh install of deep learning VM on GCE, rllib could not start training PPO with CartPole-v0.
There are some f-strings in the code and such code do not run on python 3.5.

Tested environment:
 + Debian9 on Google Compute Engine with Deep learning VM (Tensorflow 1.15.2)
 + Python 3.5.3
 + tensorflow-gpu 1.15.2  (Installed with Deep learning VM)
 + some python modules
> pip3 install dm-tree tabulate lz4 tensorboardX torch torchvision gym
 + Ray 0.8.4
> pip3 install ray
 + and Ray 0.9.0.dev0 (installed from snapshot wheel)
> pip3 install -U https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.9.0.dev0-cp35-cp35m-manylinux1_x86_64.whl

### Reproduction (REQUIRED)

From the command line on the document "rllib in 60 seconds"

> rllib train --run=PPO --env=CartPole-v0

Results

```
Traceback (most recent call last):
  File "/home/yos/bin/rllib", line 5, in <module>
    from ray.rllib.scripts import cli
  File "/home/yos/.local/lib/python3.5/site-packages/ray/rllib/__init__.py", line 60, in <module>
    _register_all()
  File "/home/yos/.local/lib/python3.5/site-packages/ray/rllib/__init__.py", line 37, in _register_all
    register_trainable(key, get_agent_class(key))
  File "/home/yos/.local/lib/python3.5/site-packages/ray/rllib/agents/registry.py", line 130, in get_agent_class
    return _get_agent_class(alg)
  File "/home/yos/.local/lib/python3.5/site-packages/ray/rllib/agents/registry.py", line 140, in _get_agent_class
    return CONTRIBUTED_ALGORITHMS[alg]()
  File "/home/yos/.local/lib/python3.5/site-packages/ray/rllib/contrib/registry.py", line 26, in _import_bandit_linucb
    from ray.rllib.contrib.bandits.agents.lin_ucb import LinUCBTrainer
  File "/home/yos/.local/lib/python3.5/site-packages/ray/rllib/contrib/bandits/agents/__init__.py", line 1, in <module>
    from ray.rllib.contrib.bandits.agents.lin_ts import LinTSTrainer
  File "/home/yos/.local/lib/python3.5/site-packages/ray/rllib/contrib/bandits/agents/lin_ts.py", line 5, in <module>
    from ray.rllib.contrib.bandits.agents.policy import BanditPolicy
  File "/home/yos/.local/lib/python3.5/site-packages/ray/rllib/contrib/bandits/agents/policy.py", line 6, in <module>
    from ray.rllib.contrib.bandits.models.linear_regression import \
  File "/home/yos/.local/lib/python3.5/site-packages/ray/rllib/contrib/bandits/models/linear_regression.py", line 87
    1] == self.d, f"Feature dimensions of weights ({self.d}) and " \
                                                                 ^
SyntaxError: invalid syntax
```

I've searched f-string in the source tree and patched like this commit(73ecb585b9f43a021e74f802ae83c9ed9889b4bf), and syntax error is gone.

However, even with this patch, training of CartPole-v0 could not be started because of another error:

```
2020-04-07 14:13:30,625	ERROR trial_runner.py:521 -- Trial PPO_CartPole-v0_00000: Error processing event.
Traceback (most recent call last):
  File "/home/yos/.local/lib/python3.5/site-packages/ray/tune/trial_runner.py", line 467, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/home/yos/.local/lib/python3.5/site-packages/ray/tune/ray_trial_executor.py", line 381, in fetch_result
    result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT)
  File "/home/yos/.local/lib/python3.5/site-packages/ray/worker.py", line 1486, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(NameError): ray::PPO.__init__() (pid=1684, ip=10.140.0.3)
  File "/home/yos/.local/lib/python3.5/site-packages/ray/rllib/utils/from_config.py", line 157, in from_config
    constructor = getattr(module, type_)
AttributeError: module 'ray.rllib.utils.exploration.exploration' has no attribute 'StochasticSampling'
```

I have no idea with this error. Any help please?

- [V ] I have verified my script runs in a clean environment and reproduces the issue.
- [V ] I have verified the issue also occurs with the [latest wheels]




Discussion:

Resolving patch:
diff --git a/rllib/contrib/bandits/examples/LinTS_train_wheel_env.py b/rllib/contrib/bandits/examples/LinTS_train_wheel_env.py
--- a/rllib/contrib/bandits/examples/LinTS_train_wheel_env.py
+++ b/rllib/contrib/bandits/examples/LinTS_train_wheel_env.py
@@ -10,7 +10,7 @@
 
 def plot_model_weights(means, covs):
     fmts = ["bo", "ro", "yx", "k+", "gx"]
-    labels = [f"arm{i}" for i in range(5)]
+    labels = ["arm{}".format(i) for i in range(5)]
 
     fig, ax = plt.subplots(figsize=(6, 4))
 
diff --git a/rllib/contrib/bandits/examples/tune_LinTS_train_wheel_env.py b/rllib/contrib/bandits/examples/tune_LinTS_train_wheel_env.py
--- a/rllib/contrib/bandits/examples/tune_LinTS_train_wheel_env.py
+++ b/rllib/contrib/bandits/examples/tune_LinTS_train_wheel_env.py
@@ -15,7 +15,7 @@
 
 def plot_model_weights(means, covs, ax):
     fmts = ["bo", "ro", "yx", "k+", "gx"]
-    labels = [f"arm{i}" for i in range(5)]
+    labels = ["arm{}".format(i) for i in range(5)]
 
     ax.set_title("Weights distributions of arms")
 
diff --git a/rllib/contrib/bandits/models/linear_regression.py b/rllib/contrib/bandits/models/linear_regression.py
--- a/rllib/contrib/bandits/models/linear_regression.py
+++ b/rllib/contrib/bandits/models/linear_regression.py
@@ -85,9 +85,9 @@ def _check_inputs(self, x, y=None):
         assert x.ndim in [2, 3], \
             "Input context tensor must be 2 or 3 dimensional, where the" \
             " first dimension is batch size"
-        assert x.shape[
-            1] == self.d, f"Feature dimensions of weights ({self.d}) and " \
-                          f"context ({x.shape[1]}) do not match!"
+        assert x.shape[1] == self.d, \
+            "Feature dimensions of weights ({}) and context ({}) do not " \
+            "match!".format(self.d, x.shape[1])
         if y:
             assert torch.is_tensor(y) and y.numel() == 1,\
                 "Target should be a tensor;" \
@@ -134,9 +134,9 @@ def predict(self, x, sample_theta=False, use_ucb=False):
             return scores
 
     def partial_fit(self, x, y, arm):
-        assert 0 <= arm.item() < len(self.arms),\
-            f"Invalid arm: {arm.item()}." \
-            f"It should be 0 <= arm < {len(self.arms)}"
+        assert 0 <= arm.item() < len(self.arms), \
+            "Invalid arm: {}. It should be 0 <= arm < {}".format(
+                arm.item(), len(self.arms))
         self.arms[arm].partial_fit(x, y)
 
     @override(ModelV2)


Test patch: