internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: Lightning-AI/pytorch-lightning

Issue:
Add resuming from specific checkpoint
**Is your feature request related to a problem? Please describe.**
In current version, there is no way to resume training from a specific checkpoint (not the last checkpoint).
Sometimes (very often in my case), one needs experiment training with different hyperparameters (e.g. dropout rate, augmentation) from a specific checkpoint.

**Describe the solution you'd like**
Add `resume_from_checkpoint`argument to Trainer class.

**Describe alternatives you've considered**
I tried to use `restore` from Trainer class, but was not successful because it is meant to be used only when .fit() is called.

**Additional context**
FYI, I made a PR to demonstrate my idea: https://github.com/williamFalcon/pytorch-lightning/pull/516


Discussion:

Resolving patch:
diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py
--- a/pytorch_lightning/trainer/trainer.py
+++ b/pytorch_lightning/trainer/trainer.py
@@ -83,7 +83,8 @@ def __init__(self,
                  weights_save_path=None,
                  amp_level='O1',
                  nb_sanity_val_steps=5,
-                 truncated_bptt_steps=None):
+                 truncated_bptt_steps=None,
+                 resume_from_checkpoint=None):
         """
 
         :param logger: Logger for experiment tracking
@@ -140,6 +141,7 @@ def __init__(self,
         self.nb_sanity_val_steps = nb_sanity_val_steps
         self.print_nan_grads = print_nan_grads
         self.truncated_bptt_steps = truncated_bptt_steps
+        self.resume_from_checkpoint = resume_from_checkpoint
         self.shown_warnings = set()
 
         self.fast_dev_run = fast_dev_run
diff --git a/pytorch_lightning/trainer/trainer_io.py b/pytorch_lightning/trainer/trainer_io.py
--- a/pytorch_lightning/trainer/trainer_io.py
+++ b/pytorch_lightning/trainer/trainer_io.py
@@ -138,8 +138,11 @@ def restore_weights(self, model):
             torch.cuda.empty_cache()
 
         if not did_restore_hpc_weights:
-            # restore weights if same exp version
-            self.restore_state_if_checkpoint_exists(model)
+            if self.resume_from_checkpoint is not None:
+                self.restore(self.resume_from_checkpoint, on_gpu=self.on_gpu)
+            else:
+                # restore weights if same exp version
+                self.restore_state_if_checkpoint_exists(model)
 
         # wait for all models to restore weights
         if self.use_ddp or self.use_ddp2:


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Training with DDP could fail at startup due to FileExistsError
In multi-GPU mode with DDP, starting the training can fail with the following error:

```
Traceback (most recent call last):
  File "train.py", line 109, in <module>
    main(hyperparams)
  File "train.py", line 62, in main
    trainer.fit(model)
  File "/home/tanel/miniconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 343, in fit
    mp.spawn(self.ddp_train, nprocs=self.num_gpus, args=(model,))
  File "/home/tanel/miniconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "/home/tanel/miniconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 118, in join
    raise Exception(msg)
Exception: 

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/home/tanel/miniconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/tanel/miniconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/ddp_mixin.py", line 181, in ddp_train
    self.run_pretrain_routine(model)
  File "/home/tanel/miniconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 415, in run_pretrain_routine
    self.configure_checkpoint_callback()
  File "/home/tanel/miniconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/callback_config_mixin.py", line 28, in configure_checkpoint_callback
    filepath=ckpt_path
  File "/home/tanel/miniconda3/lib/python3.7/site-packages/pytorch_lightning/callbacks/pt_callbacks.py", line 202, in __init__
    os.makedirs(filepath)
  File "/home/tanel/miniconda3/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/tanel/devel/torch-xvectors/lightning_logs/version_151/checkpoints'

```
It happens quite rarely.


**Desktop (please complete the following information):**
 - OS: Linux
 - Version 0.5.3.2 (git as of 2019-11-20)

Fix is easy, I'll create a PR.


Discussion:

Resolving patch:
diff --git a/pytorch_lightning/callbacks/pt_callbacks.py b/pytorch_lightning/callbacks/pt_callbacks.py
--- a/pytorch_lightning/callbacks/pt_callbacks.py
+++ b/pytorch_lightning/callbacks/pt_callbacks.py
@@ -198,8 +198,7 @@ def __init__(self, filepath, monitor='val_loss', verbose=0,
         self.monitor = monitor
         self.verbose = verbose
         self.filepath = filepath
-        if not os.path.exists(filepath):
-            os.makedirs(filepath)
+        os.makedirs(filepath, exist_ok=True)
         self.save_top_k = save_top_k
         self.save_weights_only = save_weights_only
         self.period = period


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
`trainer.on_gpu == True` for `gpus = 0` or `gpus=[]`
**Describe the bug**
`trainer.on_gpu == True` for `gpus = 0` or `gpus=[]` because of [this line](https://github.com/williamFalcon/pytorch-lightning/blob/master/pytorch_lightning/trainer/trainer.py#L135).
When resuming from a checkpoint, the model is transfered to gpu at [this line](https://github.com/williamFalcon/pytorch-lightning/blob/9785a3e78e6c5b5ff2bde45d36f5e1165dbe16ab/pytorch_lightning/trainer/trainer_io.py#L171), even if `gpus=0`

**Expected behavior**
`gpus=0` should make training happen on CPU.

I can make a PR for that, the fix is pretty simple, I think we can just replace
`self.on_gpu = gpus is not None and torch.cuda.is_available()`
by 
`self.on_gpu = True if (gpus and torch.cuda.is_available()) else False`

Am I missing something? 



Discussion:
Thanks for bringing this up. To use no gpus just set 
```
gpus=None
```
Or don't pass any arguments to trainer about gpu

If you think that gpus=0 or gpus=[] should also run on no GPUs feel free to submit a PR

Resolving patch:
diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py
--- a/pytorch_lightning/trainer/trainer.py
+++ b/pytorch_lightning/trainer/trainer.py
@@ -132,7 +132,7 @@ def __init__(self,
         self.gradient_clip_val = gradient_clip_val
         self.check_val_every_n_epoch = check_val_every_n_epoch
         self.track_grad_norm = track_grad_norm
-        self.on_gpu = gpus is not None and torch.cuda.is_available()
+        self.on_gpu = True if (gpus and torch.cuda.is_available()) else False
         self.process_position = process_position
         self.weights_summary = weights_summary
         self.max_nb_epochs = max_nb_epochs


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Using print_nan_grads in the Trainer results in an error
**Describe the bug**
When using 
```
print_nan_grads=True
```
in the Trainer, I am getting the error below.

trainer.fit(lstm_model)
  File "/Users/anaconda3/envs/snorkel/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 364, in fit
    self.run_pretrain_routine(model)
  File "/Users/anaconda3/envs/snorkel/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 471, in run_pretrain_routine
    self.train()
  File "/Users/anaconda3/envs/snorkel/lib/python3.6/site-packages/pytorch_lightning/trainer/train_loop_mixin.py", line 60, in train
    self.run_training_epoch()
  File "/Users/anaconda3/envs/snorkel/lib/python3.6/site-packages/pytorch_lightning/trainer/train_loop_mixin.py", line 99, in run_training_epoch
    output = self.run_training_batch(batch, batch_nb)
  File "/Users/anaconda3/envs/snorkel/lib/python3.6/site-packages/pytorch_lightning/trainer/train_loop_mixin.py", line 219, in run_training_batch
    self.print_nan_gradients()
  File "/Users/anaconda3/envs/snorkel/lib/python3.6/site-packages/pytorch_lightning/trainer/training_tricks_mixin.py", line 16, in print_nan_gradients
    if torch.isnan(param.grad.float()).any():
AttributeError: 'NoneType' object has no attribute 'float'

**To Reproduce**
Steps to reproduce the behavior:
If some param object, does not have **.grad**, then that object should not be checked for nans



Discussion:
sounds like an easy fix: iterate only over params where grad is not None. Would that solve the issue?
exactly. Anyone want to submit the PR?

Resolving patch:
diff --git a/pytorch_lightning/trainer/training_tricks_mixin.py b/pytorch_lightning/trainer/training_tricks_mixin.py
--- a/pytorch_lightning/trainer/training_tricks_mixin.py
+++ b/pytorch_lightning/trainer/training_tricks_mixin.py
@@ -13,7 +13,7 @@ def clip_gradients(self):
     def print_nan_gradients(self):
         model = self.get_model()
         for param in model.parameters():
-            if torch.isnan(param.grad.float()).any():
+            if (param.grad is not None) and torch.isnan(param.grad.float()).any():
                 logging.info(param, param.grad)
 
     def configure_accumulated_gradients(self, accumulate_grad_batches):


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Inadequate total steps shown in progress bar during sanity validation check when number of validation dataloaders >= 2
**Describe the bug**
The number of total steps shown in progress bar during validation sanity check is underestimated when multiple validation dataloaders are provided.

**To Reproduce**
Steps to reproduce the behavior:
1. At def val_dataloader(self), return a list of >= 2 validation dataloaders.
e.g. return [val_dataloader1, val_dataloader2]
2. Set nb_sanity_val_steps for trainer to n  (default n = 5)
2. Run experiment  

**Expected behavior**
1. If two validation dataloaders are provided, the total steps shown in progress bar during validation sanity check are n * 2. That is, 10 steps shown progress bar.

In current master - trainer.py
```    
if self.get_val_dataloaders() is not None and self.num_sanity_val_steps > 0:
            # init progress bars for validation sanity check
            pbar = tqdm.tqdm(desc='Validation sanity check', total=self.num_sanity_val_steps,
                             leave=False, position=2 * self.process_position,
                             disable=not self.show_progress_bar, dynamic_ncols=True, unit='batch')
```

Should be modified to
```
if self.get_val_dataloaders() is not None and self.num_sanity_val_steps > 0:
            # init progress bars for validation sanity check
            pbar = tqdm.tqdm(desc='Validation sanity check', total=self.num_sanity_val_steps*len(self.get_val_dataloaders()),
                             leave=False, position=2 * self.process_position,
                             disable=not self.show_progress_bar, dynamic_ncols=True, unit='batch')
```


Discussion:

Resolving patch:
diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py
--- a/pytorch_lightning/trainer/trainer.py
+++ b/pytorch_lightning/trainer/trainer.py
@@ -491,7 +491,8 @@ def run_pretrain_routine(self, model):
         ref_model.on_sanity_check_start()
         if self.get_val_dataloaders() is not None and self.num_sanity_val_steps > 0:
             # init progress bars for validation sanity check
-            pbar = tqdm.tqdm(desc='Validation sanity check', total=self.num_sanity_val_steps,
+            pbar = tqdm.tqdm(desc='Validation sanity check',
+                             total=self.num_sanity_val_steps * len(self.get_val_dataloaders()),
                              leave=False, position=2 * self.process_position,
                              disable=not self.show_progress_bar, dynamic_ncols=True, unit='batch')
             self.main_progress_bar = pbar


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Early Stopping kicks in at min_epochs + 2 instead of min_epochs
### Describe the bug
I was working on a fix for #524 and found that early stopping starts to kick in at epoch 3 despite min_epochs = 1. 

#### To Reproduce
run basic_examples/gpu_template.py and log the callback calls every epoch.

#### Expected behavior
When setting min_epochs=n (counting from 1), we should evaluate early stopping at the end of epoch n. 

#### Proposed fix:
I propose to change [this](https://github.com/williamFalcon/pytorch-lightning/blob/58cc6e13b9999161a526b83063c983750e6553b1/pytorch_lightning/trainer/training_loop.py#L322) line in the training loop:
`met_min_epochs = epoch > self.min_epochs`
to 
`met_min_epochs = epoch >= self.min_epochs - 1`

- Why the "-1"? The `epoch` variable in the training loop starts at 0, but the Trainer argument `min_epochs` starts counting at 1. 

- Why the ">="? The early stop check is done at the end of each epoch, hence the `epoch` counter will be = to `min_epochs` after `min_epochs` have passed.

Desktop (please complete the following information):
 - OS: Linux
 - Version: master


Discussion:
submit the PR for this?

Resolving patch:
diff --git a/pytorch_lightning/trainer/training_loop.py b/pytorch_lightning/trainer/training_loop.py
--- a/pytorch_lightning/trainer/training_loop.py
+++ b/pytorch_lightning/trainer/training_loop.py
@@ -50,6 +50,9 @@
     # pass in your own to override the default callback
     trainer = Trainer(early_stop_callback=early_stop_callback)
 
+    # pass in min_epochs to enable the callback after min_epochs have run
+    trainer = Trainer(early_stop_callback=early_stop_callback, min_epochs=5)
+
     # pass in None to disable it
     trainer = Trainer(early_stop_callback=None)
 
@@ -339,7 +342,7 @@ def train(self):
                 self.reduce_lr_on_plateau_scheduler.step(val_loss, epoch=self.current_epoch)
 
             # early stopping
-            met_min_epochs = epoch > self.min_epochs
+            met_min_epochs = epoch >= self.min_epochs - 1
             if self.enable_early_stop and (met_min_epochs or self.fast_dev_run):
                 should_stop = self.early_stop_callback.on_epoch_end(epoch=epoch,
                                                                     logs=self.callback_metrics)


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Comet PAPI Depreciated
Use of the Comet API logger reports an unecessary depreciation warning relating to the use of comet_ml.papi, rather than the newer comet_ml.api.

Example:
`COMET WARNING: You have imported comet_ml.papi; this interface is deprecated. Please use comet_ml.api instead. For more information, see: https://www.comet.ml/docs/python-sdk/releases/#release-300`


Discussion:

Resolving patch:
diff --git a/pytorch_lightning/logging/comet.py b/pytorch_lightning/logging/comet.py
--- a/pytorch_lightning/logging/comet.py
+++ b/pytorch_lightning/logging/comet.py
@@ -52,7 +52,11 @@ def any_lightning_module_function_or_hook(...):
 try:
     from comet_ml import Experiment as CometExperiment
     from comet_ml import OfflineExperiment as CometOfflineExperiment
-    from comet_ml.papi import API
+    try:
+        from comet_ml.api import API
+    except ImportError:
+        # For more information, see: https://www.comet.ml/docs/python-sdk/releases/#release-300
+        from comet_ml.papi import API
 except ImportError:
     raise ImportError('Missing comet_ml package.')
 


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
map_location hasn't been implemented in load_from_checkpoint
**Describe the bug**
Current `load_from_checkpoint` function does not support to load a gpu checkpoint on cpu devices, i.e. map the location. 

**To Reproduce**
```
# on a cpu only device:
Model.load_from_checkpoint("GPU_CHECKPOINT_FILE")
```

**Expected behavior**
`AssertionError: Torch not compiled with CUDA enabled`

**Desktop (please complete the following information):**
 - OS: macOS Catalina 10.15.1
 - Browser: chrome
 - Version: 0.5.3.1



Discussion:
weird, i thought we did. mind submitting a PR?
Sure will do. 

Resolving patch:
diff --git a/pytorch_lightning/core/lightning.py b/pytorch_lightning/core/lightning.py
--- a/pytorch_lightning/core/lightning.py
+++ b/pytorch_lightning/core/lightning.py
@@ -930,7 +930,7 @@ def val_dataloader(self):
         return None
 
     @classmethod
-    def load_from_metrics(cls, weights_path, tags_csv):
+    def load_from_metrics(cls, weights_path, tags_csv, map_location=None):
         """Primary way of loading model from csv weights path.
 
         :param str weights_path: Path to a PyTorch checkpoint
@@ -975,9 +975,10 @@ def load_from_metrics(cls, weights_path, tags_csv):
         hparams = load_hparams_from_tags_csv(tags_csv)
         hparams.__setattr__('on_gpu', False)
 
-        # load on CPU only to avoid OOM issues
-        # then its up to user to put back on GPUs
-        checkpoint = torch.load(weights_path, map_location=lambda storage, loc: storage)
+        if map_location is not None:
+            checkpoint = torch.load(weights_path, map_location=map_location)
+        else:
+            checkpoint = torch.load(weights_path, map_location=lambda storage, loc: storage)
 
         # load the state_dict on the model automatically
         model = cls(hparams)
@@ -989,7 +990,7 @@ def load_from_metrics(cls, weights_path, tags_csv):
         return model
 
     @classmethod
-    def load_from_checkpoint(cls, checkpoint_path):
+    def load_from_checkpoint(cls, checkpoint_path, map_location=None):
         """
         Primary way of loading model from a checkpoint
         :param checkpoint_path:
@@ -997,9 +998,11 @@ def load_from_checkpoint(cls, checkpoint_path):
         :return:
         """
 
-        # load on CPU only to avoid OOM issues
-        # then its up to user to put back on GPUs
-        checkpoint = torch.load(checkpoint_path, map_location=lambda storage, loc: storage)
+        if map_location is not None:
+            checkpoint = torch.load(checkpoint_path, map_location=map_location)
+        else:
+            checkpoint = torch.load(checkpoint_path, map_location=lambda storage, loc: storage)
+
         try:
             ckpt_hparams = checkpoint['hparams']
         except KeyError:


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Checkpoint saving isn't atomic
## 🐛 Bug

Saving checkpoints happens non-atomically. In some cases, this causes an incomplete write of a checkpoint (for example when receiving a SIGKILL during writing), causing any subsequent loading to fail with 
`RuntimeError: unexpected EOF, expected 8 more bytes. The file might be corrupted.`

### To Reproduce

This is difficult to reproduce, since it relies on timing outside of code. For me, it happens with fast-running models that run at ~1-4 seconds per epoch.

### Expected behavior

Checkpointing should be resistant to such issues, and instead simply continue as-is.


Discussion:

Resolving patch:
diff --git a/pytorch_lightning/trainer/training_io.py b/pytorch_lightning/trainer/training_io.py
--- a/pytorch_lightning/trainer/training_io.py
+++ b/pytorch_lightning/trainer/training_io.py
@@ -255,17 +255,34 @@ def term_handler(self, signum, frame):
     # --------------------
     # MODEL SAVE CHECKPOINT
     # --------------------
+    def _atomic_save(self, checkpoint, filepath):
+        """Saves a checkpoint atomically, avoiding the creation of incomplete checkpoints.
+
+        This will create a temporary checkpoint with a suffix of ``.part``, then copy it to the final location once
+        saving is finished.
+
+        Args:
+            checkpoint (object): The object to save.
+                Built to be used with the ``dump_checkpoint`` method, but can deal with anything which ``torch.save``
+                accepts.
+            filepath (str|pathlib.Path): The path to which the checkpoint will be saved.
+                This points to the file that the checkpoint will be stored in.
+        """
+        tmp_path = str(filepath) + ".part"
+        torch.save(checkpoint, tmp_path)
+        os.replace(tmp_path, filepath)
+
     def save_checkpoint(self, filepath):
         checkpoint = self.dump_checkpoint()
 
         # do the actual save
         try:
-            torch.save(checkpoint, filepath)
+            self._atomic_save(checkpoint, filepath)
         except AttributeError:
             if 'hparams' in checkpoint:
                 del checkpoint['hparams']
 
-            torch.save(checkpoint, filepath)
+            self._atomic_save(checkpoint, filepath)
 
     def restore(self, checkpoint_path, on_gpu):
         # if on_gpu:
@@ -412,12 +429,12 @@ def hpc_save(self, folderpath, logger):
         # do the actual save
         # TODO: fix for anything with multiprocess DP, DDP, DDP2
         try:
-            torch.save(checkpoint, filepath)
+            self._atomic_save(checkpoint, filepath)
         except AttributeError:
             if 'hparams' in checkpoint:
                 del checkpoint['hparams']
 
-            torch.save(checkpoint, filepath)
+            self._atomic_save(checkpoint, filepath)
 
         return filepath
 


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
batch may not have the copy method
## 🐛 Bug
In this commit: https://github.com/PyTorchLightning/pytorch-lightning/commit/48b797fdb046bab73fc04ef6d6780f05d3623485 

The training `batch` is copied before `transfer_batch_to_gpu `, but a batch may not have the `copy` method. Thus, the following error will be raised in some cases (e.g., the batch is a tuple ):
```
Traceback (most recent call last):
  File "scripts/msmacro.py", line 113, in <module>
    main()
  File "scripts/msmacro.py", line 109, in main
    trainer.fit(model)
  File "/home/zhaohao/Documents/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 405, in fit
    self.single_gpu_train(model)
  File "/home/zhaohao/Documents/pytorch-lightning/pytorch_lightning/trainer/distrib_parts.py", line 441, in single_gpu_train
    self.run_pretrain_routine(model)
  File "/home/zhaohao/Documents/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 539, in run_pretrain_routine
    self.train()
  File "/home/zhaohao/Documents/pytorch-lightning/pytorch_lightning/trainer/training_loop.py", line 332, in train
    self.run_training_epoch()
  File "/home/zhaohao/Documents/pytorch-lightning/pytorch_lightning/trainer/training_loop.py", line 385, in run_training_epoch
    output = self.run_training_batch(batch, batch_idx)
  File "/home/zhaohao/Documents/pytorch-lightning/pytorch_lightning/trainer/training_loop.py", line 504, in run_training_batch
    loss = optimizer_closure()
  File "/home/zhaohao/Documents/pytorch-lightning/pytorch_lightning/trainer/training_loop.py", line 473, in optimizer_closure
    split_batch, batch_idx, opt_idx, self.hiddens)
  File "/home/zhaohao/Documents/pytorch-lightning/pytorch_lightning/trainer/training_loop.py", line 589, in training_forward
    batch = self.transfer_batch_to_gpu(batch.copy(), gpu_id)
AttributeError: 'tuple' object has no attribute 'copy'
```



Discussion:

Resolving patch:
diff --git a/pytorch_lightning/trainer/training_loop.py b/pytorch_lightning/trainer/training_loop.py
--- a/pytorch_lightning/trainer/training_loop.py
+++ b/pytorch_lightning/trainer/training_loop.py
@@ -151,7 +151,7 @@ def training_step(self, batch, batch_idx):
 
 
 """
-
+import copy
 import inspect
 from abc import ABC, abstractmethod
 import warnings
@@ -586,7 +586,7 @@ def training_forward(self, batch, batch_idx, opt_idx, hiddens):
             gpu_id = 0
             if isinstance(self.data_parallel_device_ids, list):
                 gpu_id = self.data_parallel_device_ids[0]
-            batch = self.transfer_batch_to_gpu(batch.copy(), gpu_id)
+            batch = self.transfer_batch_to_gpu(copy.copy(batch), gpu_id)
             args[0] = batch
             output = self.model.training_step(*args)
 


Test patch: