internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: Lightning-AI/pytorch-lightning

Issue:
on_before_zero_grad hook
## 📚 Documentation

The documentation report the method `on_before_zero_grad`. Strangely, this method is not shown in the lifecycle for hooks documentation. Moreover, when it is defined in a lightning module it is not called.

Hence the question : is it a discontinued hook ? If so we could erase its mention in the docs.

Thanks.



Discussion:
Hi! thanks for your contribution!, great first issue!
Hi, thanks for catching that!
I added it in #1483 . Do you see any other hooks missing from the lifecycle?
@awaelchli not that I know ! 
It seems however that this particular hook is not called. Tried a few experiments and never reached it. [The codebase](https://github.com/PyTorchLightning/pytorch-lightning/blob/8544b334e4af9caa060a280146e7d3bb10648332/pytorch_lightning/trainer/training_loop.py#L636) doesn't seem to call it either. 

Resolving patch:
diff --git a/pytorch_lightning/core/lightning.py b/pytorch_lightning/core/lightning.py
--- a/pytorch_lightning/core/lightning.py
+++ b/pytorch_lightning/core/lightning.py
@@ -1157,6 +1157,10 @@ def optimizer_step(self, current_epoch, batch_idx, optimizer,
                     optimizer.step()
                     optimizer.zero_grad()
 
+        Note:
+            If you also override the :meth:`~pytorch_lightning.core.hooks.ModelHooks.on_before_zero_grad`
+            model hook don't forget to add the call to it before ``optimizer.zero_grad()`` yourself.
+
         """
         if self.trainer.use_tpu and XLA_AVAILABLE:
             xm.optimizer_step(optimizer)
@@ -1165,6 +1169,9 @@ def optimizer_step(self, current_epoch, batch_idx, optimizer,
         else:
             optimizer.step()
 
+        # model hook
+        self.on_before_zero_grad(optimizer)
+
         # clear gradients
         optimizer.zero_grad()
 


Test patch:
diff --git a/tests/models/test_hooks.py b/tests/models/test_hooks.py
new file mode 100644
--- /dev/null
+++ b/tests/models/test_hooks.py
@@ -0,0 +1,39 @@
+import pytest
+
+import tests.base.utils as tutils
+from pytorch_lightning import Trainer
+from tests.base import (
+    LightTrainDataloader,
+    LightValidationMixin,
+    TestModelBase,
+    LightTestMixin)
+
+
+@pytest.mark.parametrize('max_steps', [1, 2, 3])
+def test_on_before_zero_grad_called(max_steps):
+
+    class CurrentTestModel(
+        LightTrainDataloader,
+        LightValidationMixin,
+        LightTestMixin,
+        TestModelBase,
+    ):
+        on_before_zero_grad_called = 0
+
+        def on_before_zero_grad(self, optimizer):
+            self.on_before_zero_grad_called += 1
+
+    hparams = tutils.get_default_hparams()
+    model = CurrentTestModel(hparams)
+
+    trainer = Trainer(
+        max_steps=max_steps,
+        num_sanity_val_steps=5,
+    )
+    assert 0 == model.on_before_zero_grad_called
+    trainer.fit(model)
+    assert max_steps == model.on_before_zero_grad_called
+
+    model.on_before_zero_grad_called = 0
+    trainer.test(model)
+    assert 0 == model.on_before_zero_grad_called
Repository: Lightning-AI/pytorch-lightning

Issue:
Performance drop when activating gradient clipping
Hello all,

I experienced a substantial drop in computation time when activating gradient clipping (by passing a non-zero value to the keyword argument `gradient_clip_val` when initializing the Trainer).

I noticed that in the current implementation of the `clipping_gradient` method in pytorch-lightning/trainer/training_tricks.py redundant computations are made by first computing the 2-norm and second squaring this result, which could be shortened by computing the sum of squares directly. This saves one square root and squaring operation per parameter set.

Best,
Jonas

### Environment

```
cuda:
	GPU:
	available:           False
	version:             None
packages:
	numpy:               1.18.1
	pyTorch_debug:       False
	pyTorch_version:     1.4.0
	pytorch-lightning:   0.7.4-dev
	tensorboard:         2.2.1
	tqdm:                4.45.0
system:
	OS:                  Darwin
	architecture:
		64bit
		
	processor:           i386
	python:              3.8.2
	version:             Darwin Kernel Version 19.4.0: Wed Mar  4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64
```

### Additional context

I trained a relatively small (two-layered) MLP on MNIST; perhaps this performance drop does not become that apparent when training on larger network architectures.


Discussion:
Hi! thanks for your contribution!, great first issue!
Unfortunately, it seems that I do not have push access to this repository to push the proposed fix and create a pull request for this issue.
fork the repo
make the fix
submit a PR
:)

Resolving patch:
diff --git a/pytorch_lightning/trainer/training_tricks.py b/pytorch_lightning/trainer/training_tricks.py
--- a/pytorch_lightning/trainer/training_tricks.py
+++ b/pytorch_lightning/trainer/training_tricks.py
@@ -40,7 +40,7 @@ def clip_gradients(self):
                 device = parameters[0].device
                 total_norm = torch.zeros([], device=device if parameters else None)
                 for p in parameters:
-                    param_norm = p.grad.data.norm(norm_type) ** norm_type
+                    param_norm = p.grad.data.pow(norm_type).sum()
                     total_norm.add_(param_norm)
                 total_norm = (total_norm ** (1. / norm_type))
             eps = EPSILON_FP16 if self.precision == 16 else EPSILON


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Trainer.add_argparse_args bool type 
## 🐛 Bug

The boolean arguments added using `Trainer.add_argparse_args` always evaluate to True. This is caused by the following lines of the add_argparse_args fucntion:
```python
if isinstance(allowed_type, bool):
    def allowed_type(x):
         return bool(distutils.util.strtobool(x))
```
Because `allowed_type` is the actual data type and not an instance of bool. `isinstance(bool, bool)` is equal to `False`. `bool is bool` is equal True


Discussion:

Resolving patch:
diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py
--- a/pytorch_lightning/trainer/trainer.py
+++ b/pytorch_lightning/trainer/trainer.py
@@ -603,7 +603,7 @@ def add_argparse_args(cls, parent_parser: ArgumentParser) -> ArgumentParser:
         for arg, arg_types, arg_default in (at for at in cls.get_init_arguments_and_types()
                                             if at[0] not in depr_arg_names):
             for allowed_type in (at for at in allowed_types if at in arg_types):
-                if isinstance(allowed_type, bool):
+                if allowed_type is bool:
                     def allowed_type(x):
                         return bool(distutils.util.strtobool(x))
                 parser.add_argument(


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Batch being moved to gpu repeatedly with multiple optimizers and single gpu training
If you have multiple optimizers, then transfer_batch_to_gpu winds up getting called once per opt_idx, and the batch is copied each time via copy.copy(batch) in training_forward. Why copy the batch when there is only a single gpu? By removing the copy.copy() my GAN model moves from 8.53it/s to 9.25it/s. Pretty significant speedup.



Discussion:
amazing find! mind submitting a PR?
Sure, will do.

Resolving patch:
diff --git a/pytorch_lightning/trainer/training_loop.py b/pytorch_lightning/trainer/training_loop.py
--- a/pytorch_lightning/trainer/training_loop.py
+++ b/pytorch_lightning/trainer/training_loop.py
@@ -754,7 +754,11 @@ def training_forward(self, batch, batch_idx, opt_idx, hiddens):
             gpu_id = 0
             if isinstance(self.data_parallel_device_ids, list):
                 gpu_id = self.data_parallel_device_ids[0]
-            batch = self.transfer_batch_to_gpu(copy.copy(batch), gpu_id)
+
+            # Don't copy the batch since there is a single gpu that the batch could
+            # be referenced from and if there are multiple optimizers the batch will
+            # wind up copying it to the same device repeatedly.
+            batch = self.transfer_batch_to_gpu(batch, gpu_id)
             args[0] = batch
             output = self.model.training_step(*args)
 


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
hparams sometimes don't save in checkpoint
hparams sometimes don't save in checkpoint (maybe just a SLURM thing?)
checkpoints saved on slurm are missing hparams


Discussion:

Resolving patch:
diff --git a/pytorch_lightning/trainer/training_io.py b/pytorch_lightning/trainer/training_io.py
--- a/pytorch_lightning/trainer/training_io.py
+++ b/pytorch_lightning/trainer/training_io.py
@@ -325,6 +325,7 @@ def dump_checkpoint(self):
             checkpoint['native_amp_scaling_state'] = self.scaler.state_dict()
 
         if hasattr(model, "hparams"):
+            self.__clean_namespace(model.hparams)
             is_namespace = isinstance(model.hparams, Namespace)
             checkpoint['hparams'] = vars(model.hparams) if is_namespace else model.hparams
             checkpoint['hparams_type'] = 'namespace' if is_namespace else 'dict'
@@ -338,6 +339,31 @@ def dump_checkpoint(self):
 
         return checkpoint
 
+    def __clean_namespace(self, hparams):
+        """
+        Removes all functions from hparams so we can pickle
+        :param hparams:
+        :return:
+        """
+
+        if isinstance(hparams, Namespace):
+            del_attrs = []
+            for k in hparams.__dict__:
+                if callable(getattr(hparams, k)):
+                    del_attrs.append(k)
+
+            for k in del_attrs:
+                delattr(hparams, k)
+
+        elif isinstance(hparams, dict):
+            del_attrs = []
+            for k, v in hparams.items():
+                if callable(v):
+                    del_attrs.append(k)
+
+            for k in del_attrs:
+                del hparams[k]
+
     # --------------------
     # HPC IO
     # --------------------


Test patch:
diff --git a/tests/trainer/test_trainer.py b/tests/trainer/test_trainer.py
--- a/tests/trainer/test_trainer.py
+++ b/tests/trainer/test_trainer.py
@@ -27,7 +27,7 @@
 
 
 def test_hparams_save_load(tmpdir):
-    model = DictHparamsModel({'in_features': 28 * 28, 'out_features': 10})
+    model = DictHparamsModel({'in_features': 28 * 28, 'out_features': 10, 'failed_key': lambda x: x})
 
     # logger file to get meta
     trainer_options = dict(
@@ -79,12 +79,13 @@ class CurrentTestModel(LightTrainDataloader, TestModelBase):
     new_weights_path = os.path.join(tmpdir, 'save_test.ckpt')
     trainer.save_checkpoint(new_weights_path)
 
-    # load new model
-    tags_path = tutils.get_data_path(logger, path_dir=tmpdir)
-    tags_path = os.path.join(tags_path, 'meta_tags.csv')
+    # assert ckpt has hparams
+    ckpt = torch.load(new_weights_path)
+    assert 'hparams' in ckpt.keys(), 'hparams missing from checkpoints'
+
+    # won't load without hparams in the ckpt
     model_2 = LightningTestModel.load_from_checkpoint(
         checkpoint_path=new_weights_path,
-        tags_csv=tags_path
     )
     model_2.eval()
Repository: Lightning-AI/pytorch-lightning

Issue:
Wandb Group Argument
## 🚀 Feature
Wandb currently supports an argument group to its init function that is not currently exposed by the PL api. I was hoping that we could expose this argument to the user to pass into the wandb init call. 
### Motivation

This would allow users to group their experiments on the wandb platform and keep them ordered together. 

### Pitch

Add an argument to the wandb logger, pass this into the wandb init call. 

### Additional context

https://docs.wandb.com/library/init


Discussion:
Hi! thanks for your contribution!, great first issue!

Resolving patch:
diff --git a/pytorch_lightning/loggers/wandb.py b/pytorch_lightning/loggers/wandb.py
--- a/pytorch_lightning/loggers/wandb.py
+++ b/pytorch_lightning/loggers/wandb.py
@@ -39,6 +39,7 @@ class WandbLogger(LightningLoggerBase):
         log_model: Save checkpoints in wandb dir to upload on W&B servers.
         experiment: WandB experiment object
         entity: The team posting this run (default: your username or your default team)
+        group: A unique string shared by all runs in a given group
 
     Example:
         >>> from pytorch_lightning.loggers import WandbLogger
@@ -64,7 +65,8 @@ def __init__(self,
                  tags: Optional[List[str]] = None,
                  log_model: bool = False,
                  experiment=None,
-                 entity=None):
+                 entity=None,
+                 group: Optional[str] = None):
         super().__init__()
         self._name = name
         self._save_dir = save_dir
@@ -76,6 +78,7 @@ def __init__(self,
         self._offline = offline
         self._entity = entity
         self._log_model = log_model
+        self._group = group
 
     def __getstate__(self):
         state = self.__dict__.copy()
@@ -103,7 +106,8 @@ def experiment(self) -> Run:
                 os.environ['WANDB_MODE'] = 'dryrun'
             self._experiment = wandb.init(
                 name=self._name, dir=self._save_dir, project=self._project, anonymous=self._anonymous,
-                reinit=True, id=self._id, resume='allow', tags=self._tags, entity=self._entity)
+                reinit=True, id=self._id, resume='allow', tags=self._tags, entity=self._entity,
+                group=self._group)
             # save checkpoints in wandb dir to upload on W&B servers
             if self._log_model:
                 self.save_dir = self._experiment.dir


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
[Examples] The UNet model has some bugs
<!-- 
### Common bugs:
1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/PyTorchLightning/pytorch-lightning/issues/79).    
2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/PyTorchLightning/pytorch-lightning#faq)    
-->

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->
The UNet model definition has some bugs pertaining to bilinear interpolation.

#### Code sample
<!-- Ideally attach a minimal code sample to reproduce the decried issue. 
Minimal means having the shortest code but still preserving the bug. -->

1. 
https://github.com/PyTorchLightning/pytorch-lightning/blob/2950f669834506f8e5845b318b0f25d52d19e331/pl_examples/models/unet.py#L35-L37

In the code above, there seems to be a typo. The `bilinear` flag should be passed to the function `Up()`. It has instead been passed to the `.append()` method of the list.

2. 
https://github.com/PyTorchLightning/pytorch-lightning/blob/2950f669834506f8e5845b318b0f25d52d19e331/pl_examples/models/unet.py#L101-L104

The number of channels once the input passes through either one of these layers is different. For "bilinear", the number of channels remains the same, whereas they decrease to half if a `ConvTranspose2d` is used. This gives an error in the network's `.forward()` method.

I wanted to directly use the model for some other application, but not sure how issue 2 should be solved. Maybe use a 1x1 convolution to reduce the channels to half?


Discussion:
Hi! thanks for your contribution!, great first issue!
@MrinalJain17 mind draft a PR?

Resolving patch:
diff --git a/pl_examples/domain_templates/semantic_segmentation.py b/pl_examples/domain_templates/semantic_segmentation.py
--- a/pl_examples/domain_templates/semantic_segmentation.py
+++ b/pl_examples/domain_templates/semantic_segmentation.py
@@ -165,7 +165,7 @@ def validation_step(self, batch, batch_idx):
         return {'val_loss': loss_val}
 
     def validation_epoch_end(self, outputs):
-        loss_val = sum(output['val_loss'] for output in outputs) / len(outputs)
+        loss_val = torch.stack([x['val_loss'] for x in outputs]).mean()
         log_dict = {'val_loss': loss_val}
         return {'log': log_dict, 'val_loss': log_dict['val_loss'], 'progress_bar': log_dict}
 
diff --git a/pl_examples/models/unet.py b/pl_examples/models/unet.py
--- a/pl_examples/models/unet.py
+++ b/pl_examples/models/unet.py
@@ -33,7 +33,7 @@ def __init__(
             feats *= 2
 
         for _ in range(num_layers - 1):
-            layers.append(Up(feats, feats // 2), bilinear)
+            layers.append(Up(feats, feats // 2, bilinear))
             feats //= 2
 
         layers.append(nn.Conv2d(feats, num_classes, kernel_size=1))


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
diable val and test shuffling
Fixes part of #1551 


Discussion:
Need to request .shuffle in the dataloader
If shuffle is set to True, the DataLoader will set the RandomSampler, so as an alternative, we could check if `isinstance(dataloader.sampler, RandomSampler)`. 

https://github.com/pytorch/pytorch/blob/a13b5b0ae85ea6b9ba6038f99658a88039e23782/torch/utils/data/dataloader.py#L212
@awaelchli ahhh there it is haha. good catch.
Hello @williamFalcon! Thanks for updating this PR. 



* In the file [`pytorch_lightning/trainer/data_loading.py`](https://github.com/PyTorchLightning/pytorch-lightning/blob/455a6fb0bdd337553cebde647e6c7a42ae4b71a6/pytorch_lightning/trainer/data_loading.py):

> [Line 202:111](https://github.com/PyTorchLightning/pytorch-lightning/blob/455a6fb0bdd337553cebde647e6c7a42ae4b71a6/pytorch_lightning/trainer/data_loading.py#L202): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (116 > 110 characters)









##### Comment last updated at 2020-04-25 20:26:44 UTC
@williamFalcon check update

Resolving patch:
diff --git a/pytorch_lightning/trainer/data_loading.py b/pytorch_lightning/trainer/data_loading.py
--- a/pytorch_lightning/trainer/data_loading.py
+++ b/pytorch_lightning/trainer/data_loading.py
@@ -214,7 +214,7 @@ def _reset_eval_dataloader(self, model: LightningModule, mode: str) -> Tuple[int
         # shuffling in val and test set is bad practice
         for loader in dataloaders:
             if mode in ('val', 'test') and hasattr(loader, 'sampler') and isinstance(loader.sampler, RandomSampler):
-                raise MisconfigurationException(
+                rank_zero_warn(
                     f'Your {mode}_dataloader has shuffle=True, it is best practice to turn'
                     ' this off for validation and test dataloaders.')
 


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
`TrainerDDPMixin.resolve_root_node_address` fails if the host name contains a dash
## 🐛 Bug

When running under SLURM, if the nodes host names contains `-`, `MASTER_ADDR` omits the number part of the master host name (which of course makes everything else fail).

if `MASTER_ADDR` is not given, pl infers it from the node list using [`TrainerDDPMixin.resolve_root_node_address`](https://github.com/PyTorchLightning/pytorch-lightning/blob/8f6b7a2b4fea9b7bd0b873f5973e6364b3981412/pytorch_lightning/trainer/distrib_data_parallel.py#L427)

### To Reproduce

Steps to reproduce the behavior:

- Use pl in a SLURM cluster where nodes host names contains at least a dash (e.g. `jean-zay-ia810`)

The job should fail with something like this

```text
  File "<snip>/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 834, in fit
    self.ddp_train(task, model)
  File "<snip>/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 353, in ddp_train
    model.init_ddp_connection(self.proc_rank, self.world_size, self.is_slurm_managing_tasks)
  File "<snip>/lib/python3.7/site-packages/pytorch_lightning/core/lightning.py", line 957, in init_ddp_connection
    torch_distrib.init_process_group(torch_backend, rank=proc_rank, world_size=world_size)
  File "<snip>/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 393, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
  File "<snip>/lib/python3.7/site-packages/torch/distributed/rendezvous.py", line 172, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon, timeout)
ValueError: host not found: Name or service not known
```
In my case, looking into `os.environ` gives `'SLURM_NODELIST': 'jean-zay-ia[810,817-819]'` and `'MASTER_ADDR': 'jean-zay-ia'`.

The root cause being that

```python-console
>>> resolve_root_node_address("jean-zay-ia[810,817-819]")
'jean-zay-ia'
```

Fixing it should not be hard, should I submit a PR?



Discussion:
Hi! thanks for your contribution!, great first issue!

Resolving patch:
diff --git a/pytorch_lightning/trainer/distrib_data_parallel.py b/pytorch_lightning/trainer/distrib_data_parallel.py
--- a/pytorch_lightning/trainer/distrib_data_parallel.py
+++ b/pytorch_lightning/trainer/distrib_data_parallel.py
@@ -426,8 +426,8 @@ def load_spawn_weights(self, original_model):
 
     def resolve_root_node_address(self, root_node):
         if '[' in root_node:
-            name = root_node.split('[')[0]
-            number = root_node.split(',')[0]
+            name, numbers = root_node.split('[', maxsplit=1)
+            number = numbers.split(',', maxsplit=1)[0]
             if '-' in number:
                 number = number.split('-')[0]
 


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Set precision=16 (using apex) would cause early stopping break
## 🐛 Bug

The current early stopping monitor initilize by comparing if the function monitor_op is equal to torch.lt.

`self.best = torch_inf if self.monitor_op == torch.lt else -torch_inf`

https://github.com/PyTorchLightning/pytorch-lightning/blob/12138ced7c730b8b1a72f798ce74f1452ec8f4e0/pytorch_lightning/callbacks/early_stopping.py#L110

However when intializing with the apex, it seems that the torch.lt would change and this evaluation would be always false and thus the `self.best` is intialized to -inf instead of +inf.

### To Reproduce

```
import torch
from pytorch_lightning.callbacks.early_stopping import EarlyStopping
import apex.amp as amp

es = EarlyStopping()
es.monitor_op == torch.lt
Out[6]: True

model = torch.Linear(5, 5).to('cuda')
optimizers = torch.optim.Adam(model.parameters(), lr=1e-3)
amp.initialize(model, optimizers)

es.monitor_op == torch.lt
Out[22]: False
```

And this bug leads to the initialization of `self.best` to be -inf instead of inf

### Expected behavior

`self.best` should be initialized to inf instead of -inf.

### Environment

* CUDA:
        - GPU:
                - TITAN Xp
                - Quadro P400
        - available:         True
        - version:           10.1
* Packages:
        - numpy:             1.18.1
        - pyTorch_debug:     False
        - pyTorch_version:   1.4.0
        - pytorch-lightning: 0.7.5
        - tensorboard:       2.1.1
        - tqdm:              4.43.0
* System:
        - OS:                Linux
        - architecture:
                - 64bit
                -
        - processor:         x86_64
        - python:            3.6.10
        - version:           #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019
### Additional context

I bump into this bug after downloading from the master branch couple days ago. I would guess the old version is fine but did not test it.


Discussion:
Hi! thanks for your contribution!, great first issue!

Resolving patch:
diff --git a/pytorch_lightning/callbacks/early_stopping.py b/pytorch_lightning/callbacks/early_stopping.py
--- a/pytorch_lightning/callbacks/early_stopping.py
+++ b/pytorch_lightning/callbacks/early_stopping.py
@@ -107,7 +107,7 @@ def on_train_start(self, trainer, pl_module):
         # Allow instances to be re-used
         self.wait = 0
         self.stopped_epoch = 0
-        self.best = torch_inf if self.monitor_op == torch.lt else -torch_inf
+        self.best = -torch_inf if self.monitor_op(torch.Tensor(1), torch.Tensor(2))[0].item() else torch_inf
 
     def on_validation_end(self, trainer, pl_module):
         self._run_early_stopping_check(trainer, pl_module)


Test patch: