Repository: ray-project/ray

Issue:
[tune] Population-based training: broken when using keep_checkpoint_num
When using **population-based** training TUNE stops after some times throwing the following error:

`There are paused trials, but no more pending trials with sufficient resources.`

This is caused by not finding the latest checkpoint:

```
Failure # 1 (occurred at 2020-06-19_11-26-36)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/ray_trial_executor.py", line 294, in start_trial
    self._start_trial(trial, checkpoint, train=train)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/ray_trial_executor.py", line 235, in _start_trial
    self.restore(trial, checkpoint)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/ray_trial_executor.py", line 673, in restore
    data_dict = TrainableUtil.pickle_checkpoint(value)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trainable.py", line 62, in pickle_checkpoint
    checkpoint_dir = TrainableUtil.find_checkpoint_dir(checkpoint_path)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trainable.py", line 87, in find_checkpoint_dir
    raise FileNotFoundError("Path does not exist", checkpoint_path)
FileNotFoundError: [Errno Path does not exist] /content/TRASH_TUNE_PBT_oversampling_mimic_densenet121/TUNE_Model_0_2020-06-19_11-24-215xncry9c/checkpoint_6/
```

The error appears to be somewhat random since it only appears after quite some iterations

The error can be reproduced in this [colab notebook](https://colab.research.google.com/drive/1-o896bEUm7DTvS24Do0btlqbSHre49MH?usp=sharing). **It is not a COLAB related issue since the same problem arises on our own server.**

@richardliaw Is this related to #8772 ?


Discussion:
Yeah this seems like an issue. We'll take a look into this.
This is the issue (just for reference):
```
from ray.tune.schedulers import  PopulationBasedTraining

scheduler = PopulationBasedTraining(
            time_attr='training_iteration',
            metric='mean_accuracy',
            mode='max',
            perturbation_interval=1,
            log_config=True,
            hyperparam_mutations={      
                "lr": lambda: 10 ** np.random.uniform(-2, -5)
            })

train_config = {'lr': 0.01}


analysis = tune.run(
        TUNE_Model,         # Reference to the model class. 
        config=train_config, 
        fail_fast=True,         # Stop after first error
        num_samples=4,          # Num of different hyperparameter configurations
        search_alg=None,     # Search algotihm like Hyperopt, Nevergrad...
        resources_per_trial={"gpu": 1}, # Requested resources per TRIAL. If set to fraction, total_GPUS/res_p_trial can be run in parallel
        global_checkpoint_period=np.inf,   # Do not save checkpoints based on time interval
        checkpoint_freq = 1,        # Save checkpoint every time the checkpoint_score_attr improves
        checkpoint_at_end = True,   
        keep_checkpoints_num = 2,   # Keep only the best checkpoint
        checkpoint_score_attr = 'mean_accuracy', # Metric used to compare checkpoints
        verbose=1,
        scheduler=scheduler,             # Stop bad trials early
        name="TRASH_TUNE_PBT_oversampling_mimic_densenet121",
        local_dir="/content",
        stop={                          # Stop a single trial if one of the conditions are met
            "mean_accuracy": 0.85,
            "training_iteration": 15})
```
@Cryptiex for a temporary workaround, I think you can just do:

```python
  def _save(self, tmp_checkpoint_dir):
      """ Saves the model checkpoint depending of checkpoint_freq and checkpoint_at_end set in tune.run.

      Arguments:
          tmp_checkpoint_dir {str} -- [Folder where the checkpoints get saved.]

      Returns:
          [str] -- [tmp_checkpoint_dir]
      """
      checkpoint_path = os.path.join(tmp_checkpoint_dir, "model.pth")
      torch.save(self.model.state_dict(), checkpoint_path)
      return checkpoint_path # CHANGE THIS

  def _restore(self, checkpoint_path): # CHANGE THIS
      """ Restore model.
      """
      self.model.load_state_dict(torch.load(checkpoint_path))
I think the temporary fix actually is to comment this out: 
```
        keep_checkpoints_num = 2,   # Keep only the best checkpoint
```
Thanks for the fast reply! Commenting out **keep_checkpoints_num** worked for me as a temporary fix. Any idea where this is coming from? Does the PBT scheduler reference an already deleted checkpoint?
Yeah, I think it's something about not properly bookkeeping the checkpoints on disk. 
Any updates on this? The temporary fix seems not to work as indicated by #8441 and my own experiences

Resolving patch:
diff --git a/python/ray/tune/checkpoint_manager.py b/python/ray/tune/checkpoint_manager.py
--- a/python/ray/tune/checkpoint_manager.py
+++ b/python/ray/tune/checkpoint_manager.py
@@ -109,6 +109,10 @@ def on_checkpoint(self, checkpoint):
             return
 
         old_checkpoint = self.newest_persistent_checkpoint
+
+        if old_checkpoint.value == checkpoint.value:
+            return
+
         self.newest_persistent_checkpoint = checkpoint
 
         # Remove the old checkpoint if it isn't one of the best ones.


Test patch:
diff --git a/python/ray/tune/tests/test_trial_scheduler_pbt.py b/python/ray/tune/tests/test_trial_scheduler_pbt.py
new file mode 100644
--- /dev/null
+++ b/python/ray/tune/tests/test_trial_scheduler_pbt.py
@@ -0,0 +1,90 @@
+import numpy as np
+import os
+import pickle
+import random
+import unittest
+import sys
+
+from ray import tune
+from ray.tune.schedulers import PopulationBasedTraining
+
+
+class MockTrainable(tune.Trainable):
+    def setup(self, config):
+        self.iter = 0
+        self.a = config["a"]
+        self.b = config["b"]
+        self.c = config["c"]
+
+    def step(self):
+        self.iter += 1
+        return {"mean_accuracy": (self.a - self.iter) * self.b}
+
+    def save_checkpoint(self, tmp_checkpoint_dir):
+        checkpoint_path = os.path.join(tmp_checkpoint_dir, "model.mock")
+        with open(checkpoint_path, "wb") as fp:
+            pickle.dump((self.a, self.b), fp)
+        return tmp_checkpoint_dir
+
+    def load_checkpoint(self, tmp_checkpoint_dir):
+        checkpoint_path = os.path.join(tmp_checkpoint_dir, "model.mock")
+        with open(checkpoint_path, "rb") as fp:
+            self.a, self.b = pickle.load(fp)
+
+
+class MockParam(object):
+    def __init__(self, params):
+        self._params = params
+        self._index = 0
+
+    def __call__(self, *args, **kwargs):
+        val = self._params[self._index % len(self._params)]
+        self._index += 1
+        return val
+
+
+class PopulationBasedTrainingResumeTest(unittest.TestCase):
+    def testPermutationContinuation(self):
+        """
+        Tests continuation of runs after permutation.
+        Sometimes, runs were continued from deleted checkpoints.
+        This deterministic initialisation would fail when the
+        fix was not applied.
+        See issues #9036, #9036
+        """
+        scheduler = PopulationBasedTraining(
+            time_attr="training_iteration",
+            metric="mean_accuracy",
+            mode="max",
+            perturbation_interval=1,
+            log_config=True,
+            hyperparam_mutations={"c": lambda: 1})
+
+        param_a = MockParam([10, 20, 30, 40])
+        param_b = MockParam([1.2, 0.9, 1.1, 0.8])
+
+        random.seed(100)
+        np.random.seed(1000)
+        tune.run(
+            MockTrainable,
+            config={
+                "a": tune.sample_from(lambda _: param_a()),
+                "b": tune.sample_from(lambda _: param_b()),
+                "c": 1
+            },
+            fail_fast=True,
+            num_samples=20,
+            global_checkpoint_period=1,
+            checkpoint_freq=1,
+            checkpoint_at_end=True,
+            keep_checkpoints_num=1,
+            checkpoint_score_attr="min-training_iteration",
+            scheduler=scheduler,
+            name="testPermutationContinuation",
+            stop={"training_iteration": 5})
+
+
+if __name__ == "__main__":
+    import pytest
+
+    sys.exit(pytest.main(["-v", __file__]))