internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: Lightning-AI/pytorch-lightning

Issue:
Update CHANGELOG for 0.7.x
## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

Updated CHANGELOG according to the reset changes (about last two weeks) especially deprecated items like `data_loader` or `xxxxx_end`

### Additional context

<!-- Add any other context about the problem here. -->

https://github.com/PyTorchLightning/pytorch-lightning/milestone/4


Discussion:

Resolving patch:
diff --git a/pytorch_lightning/core/decorators.py b/pytorch_lightning/core/decorators.py
--- a/pytorch_lightning/core/decorators.py
+++ b/pytorch_lightning/core/decorators.py
@@ -6,11 +6,10 @@
 def data_loader(fn):
     """Decorator to make any fx with this use the lazy property.
 
-    :param fn:
-    :return:
+    Warnings:
+        This decorator deprecated in v0.7.0 and it will be removed v0.9.0.
     """
-    w = 'data_loader decorator deprecated in 0.7.0. Will remove 0.9.0'
-    warnings.warn(w)
+    warnings.warn('`data_loader` decorator deprecated in v0.7.0. Will be removed v0.9.0', DeprecationWarning)
 
     def inner_fx(self):
         return fn(self)


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Lower default progress_bar_refresh_rate
## 🚀 Feature
From a conversation in Slack.
In v0.7.1, the default value of  `progress_bar_refresh_rate` is 50, which I think too high.
It is a measure to prevent notebook freezing, however, I believe most users run PL on CLI.
In addition, the high default value confuses the users in `fast_dev_run` because it has only 2 iterations.

So, I suggest the following:
- Set the default value as `1` or `2`. (I think `1` is better.)
- Add doc for notebook users like "you should raise `progress_bar_refresh_rate` to prevent freezing!"

for participants of the conversation: @williamFalcon @Borda @jeremyjordan 


Discussion:
Thanks @S-aiueo32! Do you mind submitting a PR with the change?
@jeremyjordan yeah, just a moment 👍 

Resolving patch:
diff --git a/pytorch_lightning/trainer/__init__.py b/pytorch_lightning/trainer/__init__.py
--- a/pytorch_lightning/trainer/__init__.py
+++ b/pytorch_lightning/trainer/__init__.py
@@ -619,13 +619,13 @@ def on_train_end(self):
 progress_bar_refresh_rate
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 How often to refresh progress bar (in steps).
-Faster refresh rates (lower number), in notebooks is known to crash them
-because of their screen refresh rates. 50 is optimal for those cases.
+In notebooks, faster refresh rates (lower number) is known to crash them
+because of their screen refresh rates, so raise it to 50 or more.
 
 Example::
 
     # default used by the Trainer
-    trainer = Trainer(progress_bar_refresh_rate=50)
+    trainer = Trainer(progress_bar_refresh_rate=1)
 
 
 reload_dataloaders_every_epoch
diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py
--- a/pytorch_lightning/trainer/trainer.py
+++ b/pytorch_lightning/trainer/trainer.py
@@ -88,7 +88,7 @@ def __init__(
             num_tpu_cores: Optional[int] = None,
             log_gpu_memory: Optional[str] = None,
             show_progress_bar: bool = True,
-            progress_bar_refresh_rate: int = 50,
+            progress_bar_refresh_rate: int = 1,
             overfit_pct: float = 0.0,
             track_grad_norm: int = -1,
             check_val_every_n_epoch: int = 1,


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
clarify 3 things in docs
**clarify 3 things in docs**

1. DP, DDP don't matter with TPUs. TPUs work in DDP by default (at least rn).

2. Explain how the different steps are called (i thought we did).
(here: https://pytorch-lightning.readthedocs.io/en/0.7.1/lightning-module.html#training-loop-structure)
```
# how val loop happens


all_outs = []

for batch in val_data:
    batches = split_batch(batch)
    outs = []
    for batch in batches:
         
         # call 1
         out = validation_step()
         outs.append(out)
    
    # call 2
    batch_out = validation_end(outs)
    all_outs.append(batch_out)

# call 3
validation_epoch_end(all_outs)
```

3. Explain that checkpoint + early stopping don't get overridden by things passed in as callbacks.
(here: https://pytorch-lightning.readthedocs.io/en/0.7.1/callbacks.html)


Discussion:

Resolving patch:
diff --git a/pytorch_lightning/core/__init__.py b/pytorch_lightning/core/__init__.py
--- a/pytorch_lightning/core/__init__.py
+++ b/pytorch_lightning/core/__init__.py
@@ -107,9 +107,10 @@ def configure_optimizers(self):
 -----------------------
 
 The general pattern is that each loop (training, validation, test loop)
-has 2 methods:
+has 3 methods:
 
 - ``` ___step ```
+- ``` ___step_end ```
 - ``` ___epoch_end```
 
 To show how lightning calls these, let's use the validation loop as an example
@@ -126,6 +127,28 @@ def configure_optimizers(self):
     # like calculate validation set accuracy or loss
     validation_epoch_end(val_outs)
 
+if we use dp or ddp2 mode, we can also define the ```XXX_step_end``` method to operate
+on all parts of the batch
+
+.. code-block:: python
+
+    val_outs = []
+    for val_batch in val_data:
+        batches = split_batch(val_batch)
+        dp_outs = []
+        for sub_batch in batches:
+            dp_out = validation_step(sub_batch)
+            dp_outs.append(dp_out)
+
+        out = validation_step_end(dp_outs)
+        val_outs.append(out)
+
+    # do something with the outputs for all batches
+    # like calculate validation set accuracy or loss
+    validation_epoch_end(val_outs)
+
+.. note:: ```training_step_end``` is not available yet but coming in the next release.
+
 Add validation loop
 ^^^^^^^^^^^^^^^^^^^
 
diff --git a/pytorch_lightning/trainer/__init__.py b/pytorch_lightning/trainer/__init__.py
--- a/pytorch_lightning/trainer/__init__.py
+++ b/pytorch_lightning/trainer/__init__.py
@@ -146,7 +146,8 @@ def forward(self, x):
 callbacks
 ^^^^^^^^^
 
-Add a list of user defined callbacks.
+Add a list of user defined callbacks. These callbacks DO NOT replace the explicit callbacks
+(loggers, EarlyStopping or ModelCheckpoint).
 
 .. note:: Only user defined callbacks (ie: Not EarlyStopping or ModelCheckpoint)
 
@@ -239,6 +240,8 @@ def on_train_end(self):
     # ddp2 = DistributedDataParallel + dp
     trainer = Trainer(gpus=2, num_nodes=2, distributed_backend='ddp2')
 
+.. note:: this option does not apply to TPU. TPUs use ```ddp``` by default (over each core)
+
 early_stop_callback
 ^^^^^^^^^^^^^^^^^^^
 


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Pretty test result with pprint
## 🚀 Feature
### Motivation
Test results are shown at the end of test loops since v0.7.0.
The content of the indication is `prog_bar_metrics`, which is basically inputted as a `dict`.
By default, this part uses built-in `print` function:
![image](https://user-images.githubusercontent.com/14917234/76911857-44df9100-68f5-11ea-8719-744acfc05855.jpeg)
This way cannot display the pretty results so I suggest using `pprint`:
![image](https://user-images.githubusercontent.com/14917234/76911882-5c1e7e80-68f5-11ea-81c5-54226c3262e8.jpeg)

### Pitch
- Substitute `print` at the end of the test loop with `pprint` .

### Alternatives
Modify [here ](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/trainer/evaluation_loop.py#L375)like:
```python
        if test_mode:
            if self.proc_rank == 0:
                from pprint import pprint

                print('-' * 100)
                print('TEST RESULTS')
                pprint(prog_bar_metrics)
                print('-' * 100)
```



Discussion:

Resolving patch:
diff --git a/pytorch_lightning/trainer/evaluation_loop.py b/pytorch_lightning/trainer/evaluation_loop.py
--- a/pytorch_lightning/trainer/evaluation_loop.py
+++ b/pytorch_lightning/trainer/evaluation_loop.py
@@ -125,6 +125,7 @@
 
 import sys
 from abc import ABC, abstractmethod
+from pprint import pprint
 from typing import Callable
 
 import torch
@@ -376,7 +377,7 @@ def run_evaluation(self, test_mode: bool = False):
             if self.proc_rank == 0:
                 print('-' * 100)
                 print('TEST RESULTS')
-                print(prog_bar_metrics)
+                pprint(prog_bar_metrics)
                 print('-' * 100)
 
         # log metrics


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
multi-gpu ddp calls validation and testing loops too many times
When using ddp with multiple gpus, each validation and test loop is called with the entire validation dataset for each gpu.

Expected behavior is that the dataset is divided appropriately across the gpus.

I am using current master (cloned Mar 14), Ubuntu 19.10, Cuda 10.1, python 3.7.5, pytorch 1.4, venv environment.

The problem appears to be in `auto_add_sampler()` in data_loading.py. It does not create a `DistributedSampler` for validation or test datasets.


Discussion:
Latest pull - 1 hour ago, no longer this behavior. Closing.
Sorry - this issue still exists in some configurations. My proposed fix is not the total picture. Still investigating - will provide reproducible example.
Testing underway. Will make PR tomorrow.

Resolving patch:
diff --git a/pytorch_lightning/trainer/data_loading.py b/pytorch_lightning/trainer/data_loading.py
--- a/pytorch_lightning/trainer/data_loading.py
+++ b/pytorch_lightning/trainer/data_loading.py
@@ -91,15 +91,12 @@ def auto_add_sampler(self, dataloader: DataLoader, train: bool) -> DataLoader:
                 )
                 dl_args['shuffle'] = False
             else:
-                if train:
-                    sampler = DistributedSampler(dataloader.dataset)
-                    dl_args['shuffle'] = False
-                else:
-                    sampler = SequentialSampler(dataloader.dataset)
+                sampler = DistributedSampler(dataloader.dataset)
+                dl_args['shuffle'] = False
 
             dl_args['sampler'] = sampler
-
             dataloader = DataLoader(**dl_args)
+
         return dataloader
 
     def reset_train_dataloader(self, model: LightningModule) -> None:


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Restoring training session docs
## 📚 Documentation

["Restoring training session"](https://pytorch-lightning.readthedocs.io/en/latest/pytorch_lightning.trainer.training_io.html#restoring-training-session) documentation shows example, which does not work correctly - [Google Colab](https://colab.research.google.com/drive/14C4G2kKcCS6CvEHv6b1ikQoymsiDwjlA).

Furthermore, when I was looking how to resume sessions I was confused by `load_from_checkpoint`  (Model) and `resume_from_checkpoint` (Trainer) descriptions. I suppose clarifying that the former does only load model state (weights etc.) and the latter load model and training state would be helpful.

The most user-friendly solution would be the one currently stated in docs - passing version id to any logger would cause `fit()` to try to load the latest checkpoint from the version's folder. I think that would require every logger to implement `version` and `get_last_checkpoint` methods, which constitutes a source code change PR.



Discussion:
Hi! thanks for your contribution!, great first issue!
Thx, mind drafting a PR? :robot:
the last checkpoint would work only if your training saves all because for example with saving the N best there is no guarantee that the saved checkpoints are even from last trining stage...
cc: @PyTorchLightning/core-contributors we have discussed it earlier...

Resolving patch:
diff --git a/pytorch_lightning/trainer/training_io.py b/pytorch_lightning/trainer/training_io.py
--- a/pytorch_lightning/trainer/training_io.py
+++ b/pytorch_lightning/trainer/training_io.py
@@ -39,15 +39,9 @@
 .. code-block:: python
 
     from pytorch_lightning import Trainer
-    from pytorch_lightning.loggers import TestTubeLogger
 
-    logger = TestTubeLogger(
-        save_dir='./savepath',
-        version=1  # An existing version with a saved checkpoint
-    )
     trainer = Trainer(
-        logger=logger,
-        default_save_path='./savepath'
+        resume_from_checkpoint=PATH
     )
 
     # this fit call loads model weights and trainer state


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Not auto add DistributedSampler for DDP training
## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->
in 0.72, even if we don't set sampler, pytorch_lightning will not add  DistributedSampler for us.
### To Reproduce
the reason is in pytorch, if we don't set sampler,  pytorch will add a sampler for us.
in pytorch's dataloader.py:
```
         if sampler is None:  # give default samplers
            if self._dataset_kind == _DatasetKind.Iterable:
                # See NOTE [ Custom Samplers and IterableDataset ]
                sampler = _InfiniteConstantSampler()
            else:  # map-style
                if shuffle:
                    sampler = RandomSampler(dataset)
                else:
                    sampler = SequentialSampler(dataset)
```

but in pytorch_lightning we check whether sampler is None to  decide to add sampler 
in  data_loading.py funciton auto_add_sampler:
```
        no_sampler_added = dataloader.sampler is None
```

because pytorch have default sampler for us, which is not None, pytorch_lighting will not automatically add sampler.






Discussion:

Resolving patch:
diff --git a/pytorch_lightning/trainer/data_loading.py b/pytorch_lightning/trainer/data_loading.py
--- a/pytorch_lightning/trainer/data_loading.py
+++ b/pytorch_lightning/trainer/data_loading.py
@@ -87,9 +87,8 @@ def auto_add_sampler(self, dataloader: DataLoader, train: bool) -> DataLoader:
             return dataloader
 
         need_dist_sampler = self.use_ddp or self.use_ddp2 or self.use_tpu
-        no_sampler_added = dataloader.sampler is None
 
-        if need_dist_sampler and no_sampler_added:
+        if need_dist_sampler:
 
             skip_keys = ['sampler', 'batch_sampler', 'dataset_kind']
 


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
run_training_batch breaks on None batch or -1 response from on_batch_start (in new 0.7.2 release)
## 🐛 Bug

run_training_batch now is supposed to return a 4-tuple in 0.7.2 

however, there are two places where it still returns a 3-tuple, which will cause the program to crash, saying "ValueError: not enough values to unpack (expected 4, got 3)"

(training_loop.py:533)
```
if batch is None:
    return 0, grad_norm_dic, {}
``` 

(training_loop.py:543)
```
if response == -1:
    return -1, grad_norm_dic, {}
```

vs. the standard return
`return 0, grad_norm_dic, all_log_metrics, batch_output`

### To reproduce
just return -1 from on_batch_start



Discussion:
good catch. mind submitting a PR?

Resolving patch:
diff --git a/pytorch_lightning/trainer/training_io.py b/pytorch_lightning/trainer/training_io.py
--- a/pytorch_lightning/trainer/training_io.py
+++ b/pytorch_lightning/trainer/training_io.py
@@ -322,7 +322,7 @@ def dump_checkpoint(self):
             checkpoint['hparams_type'] = 'namespace' if is_namespace else 'dict'
         else:
             rank_zero_warn(
-                "Did not find hyperparameters at model.hparams. Saving checkpoint without hyperparameters."
+                "Did not find hyperparameters at model hparams. Saving checkpoint without hyperparameters."
             )
 
         # give the model a chance to add a few things
diff --git a/pytorch_lightning/trainer/training_loop.py b/pytorch_lightning/trainer/training_loop.py
--- a/pytorch_lightning/trainer/training_loop.py
+++ b/pytorch_lightning/trainer/training_loop.py
@@ -531,7 +531,7 @@ def run_training_batch(self, batch, batch_idx):
         all_log_metrics = []
 
         if batch is None:
-            return 0, grad_norm_dic, {}
+            return 0, grad_norm_dic, {}, {}
 
         # Batch start events
         with self.profiler.profile('on_batch_start'):
@@ -541,7 +541,7 @@ def run_training_batch(self, batch, batch_idx):
             if self.is_function_implemented('on_batch_start'):
                 response = self.get_model().on_batch_start(batch)
                 if response == -1:
-                    return -1, grad_norm_dic, {}
+                    return -1, grad_norm_dic, {}, {}
 
         splits = [batch]
         if self.truncated_bptt_steps is not None:


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Test metrics is not being reported to TensorBoard since 0.7.2
<!-- 
### Common bugs:
1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/PyTorchLightning/pytorch-lightning/issues/79).    
2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/PyTorchLightning/pytorch-lightning#faq)    
-->

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

### To Reproduce

Steps to reproduce the behavior:

https://colab.research.google.com/drive/1fM6xL140u9pU0vcmJf6qKzHwczjcMpcF

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->


#### Code sample
<!-- Ideally attach a minimal code sample to reproduce the decried issue. 
Minimal means having the shortest code but still preserving the bug. -->

Please see the colab above.

### Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
The test metrics should be reported.

### Environment

The Colab environment:
```
cuda:
	GPU:
	available:           False
	version:             10.1
packages:
	numpy:               1.18.2
	pyTorch_debug:       False
	pyTorch_version:     1.4.0
	pytorch-lightning:   0.7.2
	tensorboard:         2.2.0
	tqdm:                4.38.0
system:
	OS:                  Linux
	architecture:
		64bit
		
	processor:           x86_64
	python:              3.6.9
	version:             #1 SMP Wed Feb 19 05:26:34 PST 2020
```

### Additional context
Regression from 0.7.1


Discussion:
Hi! thanks for your contribution!, great first issue!
@williamFalcon ^^ 0.7.3?

Resolving patch:
diff --git a/pytorch_lightning/__init__.py b/pytorch_lightning/__init__.py
--- a/pytorch_lightning/__init__.py
+++ b/pytorch_lightning/__init__.py
@@ -1,6 +1,6 @@
 """Root package info."""
 
-__version__ = '0.7.3rc1'
+__version__ = '0.7.3rc2'
 __author__ = 'William Falcon et al.'
 __author_email__ = 'waf2107@columbia.edu'
 __license__ = 'Apache-2.0'
diff --git a/pytorch_lightning/trainer/evaluation_loop.py b/pytorch_lightning/trainer/evaluation_loop.py
--- a/pytorch_lightning/trainer/evaluation_loop.py
+++ b/pytorch_lightning/trainer/evaluation_loop.py
@@ -377,10 +377,10 @@ def run_evaluation(self, test_mode: bool = False):
         self.add_tqdm_metrics(prog_bar_metrics)
 
         # log results of test
-        if test_mode and self.proc_rank == 0 and prog_bar_metrics:
+        if test_mode and self.proc_rank == 0 and len(callback_metrics) > 0:
             print('-' * 80)
             print('TEST RESULTS')
-            pprint(prog_bar_metrics)
+            pprint(callback_metrics)
             print('-' * 80)
 
         # log metrics


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Propose to save new model before deleting previous ones in ModelCheckpointing
## 🚀 Feature
<!-- A clear and concise description of the feature proposal -->
In an edge case, the trainer deleted previous model and then was killed because of system error before successfully saving new model. Thus all the models were lost.
I understand specifying save_top_k > 1 helps, and saving before deleting leads to larger disk consumption. But it might be good to provide an option for this?
### Motivation

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->

### Pitch

<!-- A clear and concise description of what you want to happen. -->
in the worst case, you have two but never none...

### Alternatives

<!-- A clear and concise description of any alternative solutions or features you've considered, if any. -->

### Additional context

<!-- Add any other context or screenshots about the feature request here. -->



Discussion:
That is good point, I like it... @PyTorchLightning/core-contributors ^^
@Ir1d sounds great. submit a PR?
I can do it this friday

Resolving patch:
diff --git a/pytorch_lightning/callbacks/model_checkpoint.py b/pytorch_lightning/callbacks/model_checkpoint.py
--- a/pytorch_lightning/callbacks/model_checkpoint.py
+++ b/pytorch_lightning/callbacks/model_checkpoint.py
@@ -216,10 +216,12 @@ def on_validation_end(self, trainer, pl_module):
 
     def _do_check_save(self, filepath, current, epoch):
         # remove kth
+
+        del_list = []
         if len(self.best_k_models) == self.save_top_k and self.save_top_k > 0:
             delpath = self.kth_best_model
             self.best_k_models.pop(self.kth_best_model)
-            self._del_model(delpath)
+            del_list.append(delpath)
 
         self.best_k_models[filepath] = current
         if len(self.best_k_models) == self.save_top_k:
@@ -238,3 +240,7 @@ def _do_check_save(self, filepath, current, epoch):
                 f' {current:0.5f} (best {self.best:0.5f}), saving model to'
                 f' {filepath} as top {self.save_top_k}')
         self._save_model(filepath)
+
+        for cur_path in del_list:
+            if cur_path != filepath:
+                self._del_model(cur_path)


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Learning rate scheduler should step after each optimizer step
## 🐛 Bug

I'm not sure that this is a bug or if it is a deliberate design decision, but right now the learning rate schedule gets updated at every "step" which actually corresponds to every forward pass. I think a more standard implementation would have the learning rate scheduler "step" interval correspond to being updated every backwards pass. This has caused me a lot of problems with instability as I did not realize that using standard learning rate warmups of say 16000 steps would actually only warm up for 1000 steps if I set `accumulate_grad_batches=16`.



Discussion:
good point. it should be every backward pass as you mention.
mind submitting  PR?
Sure, will do.

Resolving patch:
diff --git a/pytorch_lightning/trainer/training_loop.py b/pytorch_lightning/trainer/training_loop.py
--- a/pytorch_lightning/trainer/training_loop.py
+++ b/pytorch_lightning/trainer/training_loop.py
@@ -454,8 +454,11 @@ def run_training_epoch(self):
             # when returning -1 from train_step, we end epoch early
             early_stop_epoch = batch_result == -1
 
-            # update lr
-            self.update_learning_rates(interval='step')
+            # TODO: consolidate all actions that need to take place only after
+            # self.accumulate_grad_batches steps (optimizer step, lr update, global step increment)
+            if (self.batch_idx + 1) % self.accumulate_grad_batches == 0:
+                # update lr
+                self.update_learning_rates(interval='step')
 
             # ---------------
             # RUN VAL STEP


Test patch: