Repository: ray-project/ray

Issue:
[rllib] rllib train ... --checkpoint-at-end flag is ignored
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->

### What is the problem?

Ray 0.8.5
Python 3.7
MacOS Catalina
PyTorch 1.4.0

No checkpoint is produced when the `--checkpoint-at-end` flag is used. Adding `--checkpoint-freq 10` does cause checkpoints to be saved.

### Reproduction (REQUIRED)

```shell
rllib train --run PPO --env CartPole-v0 --stop='{"training_iteration": 25}' --ray-address auto --checkpoint-at-end
```

No checkpoint directory exists under `~/ray_results/default/PPO...`

Add the `--checkpoint-freq 10` flag:

```shell
rllib train --run PPO --env CartPole-v0 --stop='{"training_iteration": 25}' --ray-address auto --checkpoint-at-end --checkpoint-freq 10
```

Now there are `checkpoint_10` and `checkpoint_20` directories, but not a `checkpoint_25` at the end. 

Could the choice of stop criteria, `training_iteration`, have something to do with it?

> **Note:** Being persnickety, it bugs me that the final directory path for the checkpoints is `.../checkpoint_20/checkpoint-20` (underscore vs. dash). How about one or the other?

- [x] I have verified my script runs in a clean environment and reproduces the issue.
- [ ] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).

[rllib] rllib train ... --checkpoint-at-end flag is ignored
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->

### What is the problem?

Ray 0.8.5
Python 3.7
MacOS Catalina
PyTorch 1.4.0

No checkpoint is produced when the `--checkpoint-at-end` flag is used. Adding `--checkpoint-freq 10` does cause checkpoints to be saved.

### Reproduction (REQUIRED)

```shell
rllib train --run PPO --env CartPole-v0 --stop='{"training_iteration": 25}' --ray-address auto --checkpoint-at-end
```

No checkpoint directory exists under `~/ray_results/default/PPO...`

Add the `--checkpoint-freq 10` flag:

```shell
rllib train --run PPO --env CartPole-v0 --stop='{"training_iteration": 25}' --ray-address auto --checkpoint-at-end --checkpoint-freq 10
```

Now there are `checkpoint_10` and `checkpoint_20` directories, but not a `checkpoint_25` at the end. 

Could the choice of stop criteria, `training_iteration`, have something to do with it?

> **Note:** Being persnickety, it bugs me that the final directory path for the checkpoints is `.../checkpoint_20/checkpoint-20` (underscore vs. dash). How about one or the other?

- [x] I have verified my script runs in a clean environment and reproduces the issue.
- [ ] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).



Discussion:

Resolving patch:
diff --git a/rllib/train.py b/rllib/train.py
--- a/rllib/train.py
+++ b/rllib/train.py
@@ -149,6 +149,7 @@ def run(args, parser):
             args.experiment_name: {  # i.e. log to ~/ray_results/default
                 "run": args.run,
                 "checkpoint_freq": args.checkpoint_freq,
+                "checkpoint_at_end": args.checkpoint_at_end,
                 "keep_checkpoints_num": args.keep_checkpoints_num,
                 "checkpoint_score_attr": args.checkpoint_score_attr,
                 "local_dir": args.local_dir,


Test patch:
diff --git a/rllib/tests/test_checkpoint_restore.py b/rllib/tests/test_checkpoint_restore.py
--- a/rllib/tests/test_checkpoint_restore.py
+++ b/rllib/tests/test_checkpoint_restore.py
@@ -124,6 +124,9 @@ def ckpt_restore_test(alg_name, tfe=False):
                 if abs(a1 - a2) > .1:
                     raise AssertionError("algo={} [a1={} a2={}]".format(
                         alg_name, a1, a2))
+            # Stop both Trainers.
+            alg1.stop()
+            alg2.stop()
 
 
 class TestCheckpointRestore(unittest.TestCase):