internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: Lightning-AI/pytorch-lightning

Issue:
TensorBoardLogger and ModelCheckpoint are not using the same folder by default
## 🐛 Bug
(master branch)
By default, the TensorBoardLogger writes logs into `lightning_logs/0` but ModelCheckpoint writes checkpoint into `lightning_logs/version_0`.


Discussion:

Resolving patch:
diff --git a/pytorch_lightning/logging/tensorboard.py b/pytorch_lightning/logging/tensorboard.py
--- a/pytorch_lightning/logging/tensorboard.py
+++ b/pytorch_lightning/logging/tensorboard.py
@@ -63,7 +63,7 @@ def experiment(self):
 
         root_dir = os.path.join(self.save_dir, self.name)
         os.makedirs(root_dir, exist_ok=True)
-        log_dir = os.path.join(root_dir, str(self.version))
+        log_dir = os.path.join(root_dir, "version_" + str(self.version))
         self._experiment = SummaryWriter(log_dir=log_dir, **self.kwargs)
         return self._experiment
 
@@ -131,9 +131,11 @@ def version(self):
 
     def _get_next_version(self):
         root_dir = os.path.join(self.save_dir, self.name)
-        existing_versions = [
-            int(d) for d in os.listdir(root_dir) if os.path.isdir(os.path.join(root_dir, d)) and d.isdigit()
-        ]
+        existing_versions = []
+        for d in os.listdir(root_dir):
+            if os.path.isdir(os.path.join(root_dir, d)) and d.startswith("version_"):
+                existing_versions.append(int(d.split("_")[1]))
+
         if len(existing_versions) == 0:
             return 0
         else:


Test patch:
diff --git a/tests/test_logging.py b/tests/test_logging.py
--- a/tests/test_logging.py
+++ b/tests/test_logging.py
@@ -296,8 +296,8 @@ def test_tensorboard_automatic_versioning(tmpdir):
     """Verify that automatic versioning works"""
 
     root_dir = tmpdir.mkdir("tb_versioning")
-    root_dir.mkdir("0")
-    root_dir.mkdir("1")
+    root_dir.mkdir("version_0")
+    root_dir.mkdir("version_1")
 
     logger = TensorBoardLogger(save_dir=tmpdir, name="tb_versioning")
 
@@ -308,9 +308,9 @@ def test_tensorboard_manual_versioning(tmpdir):
     """Verify that manual versioning works"""
 
     root_dir = tmpdir.mkdir("tb_versioning")
-    root_dir.mkdir("0")
-    root_dir.mkdir("1")
-    root_dir.mkdir("2")
+    root_dir.mkdir("version_0")
+    root_dir.mkdir("version_1")
+    root_dir.mkdir("version_2")
 
     logger = TensorBoardLogger(save_dir=tmpdir, name="tb_versioning", version=1)
Repository: Lightning-AI/pytorch-lightning

Issue:
Fitting with log_gpu_memory=True fails in python3.6.
## Bug

Fitting with `log_gpu_memory=True` in the Trainer fails in python3.6 version.

### To Reproduce

1. Use python3.6 version
2. Create any trainer with `log_gpu_memory=True` option.
3. Then fit it.
3. See error:

```
/a/pytorch-lightning/pytorch_lightning/core/memory.py in get_gpu_memory_map()
    237         encoding='utf-8',
    238         capture_output=True,
--> 239         check=True)
    240     # Convert lines into a dictionary
    241     gpu_memory = [int(x) for x in result.stdout.strip().split(os.linesep)]

/usr/lib/python3.6/subprocess.py in run(input, timeout, check, *popenargs, **kwargs)
    421         kwargs['stdin'] = PIPE
    422 
--> 423     with Popen(*popenargs, **kwargs) as process:
    424         try:
    425             stdout, stderr = process.communicate(input, timeout=timeout)

TypeError: __init__() got an unexpected keyword argument 'capture_output'

```


#### Code sample

```
trainer = Trainer(
    log_gpu_memory=True,
   # ....
)
trainer.fit()
```

### Expected behavior

For the same code there is no errors for python3.7

### Environment

pytorch:          1.2.0
Ubuntu 18.04
pytorch-lightning: 
    - installed to pip environment
    - commit 7a1df80f4e98fca
    - python setup.py develop
    - version 0.6.0 
python:         3.6.8
cuda: 10.0, V10.0.130 
cudnn:          7.6.2
GPU: RTX 2080 TI

### Additional context

In the `setup.py`
 python_requires='>=3.6',

But  `capture_output` is used in  `subprocess.run` calling, which is valid only for python3.7
See also workaround to maintain python3.6:
https://stackoverflow.com/questions/53209127/




Discussion:

Resolving patch:
diff --git a/pytorch_lightning/core/memory.py b/pytorch_lightning/core/memory.py
--- a/pytorch_lightning/core/memory.py
+++ b/pytorch_lightning/core/memory.py
@@ -5,6 +5,7 @@
 import gc
 import os
 import subprocess
+from subprocess import PIPE
 
 import numpy as np
 import pandas as pd
@@ -235,7 +236,8 @@ def get_gpu_memory_map():
             '--format=csv,nounits,noheader',
         ],
         encoding='utf-8',
-        capture_output=True,
+        # capture_output=True,          # valid for python version >=3.7
+        stdout=PIPE, stderr=PIPE,       # for backward compatibility with python version 3.6
         check=True)
     # Convert lines into a dictionary
     gpu_memory = [int(x) for x in result.stdout.strip().split(os.linesep)]


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
W&B: Allow for passing experiment into the WandbLogger (and logging semantics)
Currently, the WandbLogger will automatically create a new internal experiment (run) whenever you create a new WandbLogger.

# Issue 

If I instantiate a wandb experiment outside of the logger, then I will have two experiments when I train my model since there is no way to set the internal experiment of the WandbLogger to my current external experiment.

# Potential Solution
Allow for passing an experiment into the WandbLogger:

```
class WandbLogger(LightningLoggerBase):
    def __init__(self, name=None, save_dir=None, offline=False, id=None, anonymous=False,
                 version=None, project=None, tags=None, experiment=None):
              .
              .
              .
              self._experiment = experiment
```
Then I can do this:

```
experiment = wandb.init(.......)
wandb_logger = WandbLogger(experiment=experiment)
```
I made this change locally, however, I wasn't sure if this was something you also wanted to implement as well. It works for me.

# Another small note

In the `WandbLogger.log_metrics` function, I would change:

`self.experiment.history.add(metrics)` --> `self.experiment.log(metrics)`


Discussion:
@colehurwitz sounds good. feel free  to submit PR!
Will do!

Resolving patch:
diff --git a/pytorch_lightning/logging/wandb.py b/pytorch_lightning/logging/wandb.py
--- a/pytorch_lightning/logging/wandb.py
+++ b/pytorch_lightning/logging/wandb.py
@@ -33,7 +33,7 @@ class WandbLogger(LightningLoggerBase):
     """
 
     def __init__(self, name=None, save_dir=None, offline=False, id=None, anonymous=False,
-                 version=None, project=None, tags=None):
+                 version=None, project=None, tags=None, experiment=None):
         super().__init__()
         self._name = name
         self._save_dir = save_dir
@@ -41,7 +41,7 @@ def __init__(self, name=None, save_dir=None, offline=False, id=None, anonymous=F
         self._id = version or id
         self._tags = tags
         self._project = project
-        self._experiment = None
+        self._experiment = experiment
         self._offline = offline
 
     def __getstate__(self):
@@ -81,7 +81,7 @@ def log_hyperparams(self, params):
     @rank_zero_only
     def log_metrics(self, metrics, step=None):
         metrics["global_step"] = step
-        self.experiment.history.add(metrics)
+        self.experiment.log(metrics)
 
     def save(self):
         pass


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Better way to set retain_graph
Is there a better way to set `retain_graph`, especially when using two optimizers? 

I have read the issue  #356 and the corresponding fix, to set it by overriding the `backward` function.
However, this becomes messy especially when more than 1 optimizers are used, as the function doesn't have the `optimizer_idx` as an argument.

There are workarounds like naming the optimizer object and using that to set/reset the `retain graph`, but passing `optimizer_idx` would be great.

This shouldn't be difficult as the `opt_idx` variable is [available](https://github.com/PyTorchLightning/pytorch-lightning/blob/1030c4a79c2f106e22db02acc11ef2cc1bde068b/pytorch_lightning/trainer/train_loop_mixin.py#L179) inside the loop in which the `backward` function is called

 - Version [0.6.0]



Discussion:
good suggestion. a pr for this would be great!

Resolving patch:
diff --git a/pytorch_lightning/core/hooks.py b/pytorch_lightning/core/hooks.py
--- a/pytorch_lightning/core/hooks.py
+++ b/pytorch_lightning/core/hooks.py
@@ -124,12 +124,13 @@ def on_after_backward(self):
         """
         pass
 
-    def backward(self, use_amp, loss, optimizer):
+    def backward(self, use_amp, loss, optimizer, optimizer_idx):
         """Override backward with your own implementation if you need to
 
         :param use_amp: Whether amp was requested or not
         :param loss: Loss is already scaled by accumulated grads
         :param optimizer: Current optimizer being used
+        :param optimizer_idx: Index of the current optimizer being used
         :return:
 
         Called to perform backward step.
diff --git a/pytorch_lightning/trainer/training_loop.py b/pytorch_lightning/trainer/training_loop.py
--- a/pytorch_lightning/trainer/training_loop.py
+++ b/pytorch_lightning/trainer/training_loop.py
@@ -486,13 +486,14 @@ def optimizer_closure():
 
                     # backward pass
                     model_ref = self.get_model()
-                    model_ref.backward(self.use_amp, closure_loss, optimizer)
+                    model_ref.backward(self.use_amp, closure_loss, optimizer, opt_idx)
 
                     # track metrics for callbacks
                     all_callback_metrics.append(callback_metrics)
 
                     # track progress bar metrics
                     self.add_tqdm_metrics(progress_bar_metrics)
+                    self.add_tqdm_metrics(progress_bar_metrics)
                     all_log_metrics.append(log_metrics)
 
                     # insert after step hook


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
ModelCheckpoint Filepath Doesn't Use Logger Save Dir
# 🐛 Bug

Not sure if this is intended, but the model checkpoint isn't using the same directory as the logger, even if the logger exists. I would have expected this line [here](https://github.com/PyTorchLightning/pytorch-lightning/blob/588ad8377167c0b7d29cf4f362dbf42015f7568d/pytorch_lightning/trainer/callback_config.py#L28) to be `self.logger.save_dir` instead of `self.default_save_path`. 

Thank you,
-Collin


Discussion:
I am running into the same issue and feel like I must be doing something wrong. If I do not define a logger and checkpoint callback, the logger and checkpoint are saved in a clear structure under lightning_logs. But if I do provide them, it turns out to be difficult to get Lightning to follow that same structure. If I use lightning_logs as filepath I get the warning that that directory already exists and that my existing checkpoints will be overwritten. 
Were you able to find the checkpoints? I cant seem to find them anywhere!
The checkpoints are currently being saved to a subfolder of the current working directory. If you print the current working directory at the top of our file with `print(os.getcwd())` that should tell you where to find them.
FWIW, using the default checkpoint_callback, the checkpoint is not in default location. Still looking for it! Though no more default checkpoint_callback for me ...

Resolving patch:
diff --git a/pytorch_lightning/trainer/callback_config.py b/pytorch_lightning/trainer/callback_config.py
--- a/pytorch_lightning/trainer/callback_config.py
+++ b/pytorch_lightning/trainer/callback_config.py
@@ -23,8 +23,11 @@ def configure_checkpoint_callback(self):
         if self.checkpoint_callback is True:
             # init a default one
             if self.logger is not None:
+                save_dir = (getattr(self.logger, 'save_dir', None) or
+                            getattr(self.logger, '_save_dir', None) or
+                            self.default_save_path)
                 ckpt_path = os.path.join(
-                    self.default_save_path,
+                    save_dir,
                     self.logger.name,
                     f'version_{self.logger.version}',
                     "checkpoints"


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
TensorBoardLogger creates another tfevents file.
## 🐛 Bug
TensorBoardLogger creates another tfevents file when fit() is running.
It seems that no metrics are logged in the redundant file, but it will be shown in TensorBoard as a `run`.
I don't do anything about loggers in my LightningModules.


**Expected file structure:**
```
|
|- save_dir
|    |- name
|         |- version_0
|               |- events.out.tfevents.1579833025.ip-xxx-xxx-xxx-xxx.17584.0
|               |- meta_tags.csv
|- train.py
```

**Observed file structure:**
```
|
|- save_dir
|    |- name
|         |- version_0
|               |- 1579833032
|                     |- events.out.tfevents.1579833032.ip-xxx-xxx-xxx-xxx.17584.1
|               |- events.out.tfevents.1579833025.ip-xxx-xxx-xxx-xxx.17584.0
|               |- meta_tags.csv
|- train.py
```

### To Reproduce / Code sample
Basic training step of PyTorch  Lightning:
```python
# define a logger
logger = TensorBoardLogger(
    save_dir='runs',
    name=args.model
)

# define callbacks
ckpt_path = Path(logger.experiment.log_dir) / 'ckpts'
checkpoint_callback = ModelCheckpoint(filepath=ckpt_path)

# instantiate trainer
trainer = Trainer(
    logger=logger,
    checkpoint_callback=checkpoint_callback,
    gpus=args.gpus
)

# define a model
model = CoolModel(args)

# start training!
trainer.fit(model)
```

### Environment

 - PyTorch  Lightning Version (e.g., 1.0): `0.6.0`
 - PyTorch Version (e.g., 1.0): `1.3.1`
 - OS (e.g., Linux): `Ubuntu 16.04 LTS`
 - How you installed PyTorch (`conda`, `pip`, source): `pip`
 - Build command you used (if compiling from source):
 - Python version: `3.7.4`
 - CUDA/cuDNN version: `10.1`


Discussion:

Resolving patch:
diff --git a/pytorch_lightning/loggers/tensorboard.py b/pytorch_lightning/loggers/tensorboard.py
--- a/pytorch_lightning/loggers/tensorboard.py
+++ b/pytorch_lightning/loggers/tensorboard.py
@@ -84,8 +84,12 @@ def log_hyperparams(self, params):
                 " hyperparameter logging."
             )
         else:
-            # `add_hparams` requires both - hparams and metric
-            self.experiment.add_hparams(hparam_dict=params, metric_dict={})
+            from torch.utils.tensorboard.summary import hparams
+            exp, ssi, sei = hparams(params, {})
+            writer = self.experiment._get_file_writer()
+            writer.add_summary(exp)
+            writer.add_summary(ssi)
+            writer.add_summary(sei)
         # some alternative should be added
         self.tags.update(params)
 


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Optional dependencies are required for deprecated logging module
🐛 Bug

There is a backwards compatibility issues coming from PR #767. Notably, if a user doesn't have any of the extra logging dependencies then they'll be an import error.

### To Reproduce

1. Remove all logging dependencies from your environment (E.g. comet)
2. Depend on the deprecated pytorch_lightning.logging package and run

### Expected behavior

We expect to maintain backwards compatibility here so optional dependencies shouldn't be required.


Discussion:

Resolving patch:
diff --git a/pytorch_lightning/logging/__init__.py b/pytorch_lightning/logging/__init__.py
--- a/pytorch_lightning/logging/__init__.py
+++ b/pytorch_lightning/logging/__init__.py
@@ -8,6 +8,3 @@
               " and will be removed in v0.8.0", DeprecationWarning)
 
 from pytorch_lightning.loggers import *  # noqa: F403
-from pytorch_lightning.loggers import (  # noqa: E402
-    base, comet, mlflow, neptune, tensorboard, test_tube, wandb
-)


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Enable anchor links in docs
Our docs don't have these links. We need to enable them.

![image](https://user-images.githubusercontent.com/3640001/74248339-71303e00-4cb5-11ea-8657-eb8f847bf1ed.png)



Discussion:

Resolving patch:
diff --git a/docs/source/conf.py b/docs/source/conf.py
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -359,4 +359,15 @@ def find_source():
     'private-members': True,
 }
 
+# Sphinx will add “permalinks” for each heading and description environment as paragraph signs that
+#  become visible when the mouse hovers over them.
+# This value determines the text for the permalink; it defaults to "¶". Set it to None or the empty
+#  string to disable permalinks.
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-html_add_permalinks
 html_add_permalinks = True
+
+# True to prefix each section label with the name of the document it is in, followed by a colon.
+#  For example, index:Introduction for a section called Introduction that appears in document index.rst.
+#  Useful for avoiding ambiguity when the same section heading appears in different documents.
+# http://www.sphinx-doc.org/en/master/usage/extensions/autosectionlabel.html
+autosectionlabel_prefix_document = True


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
TPU Training Issues and Feature Requests
TPU training is really cool. A couple issues that came up in training. 

*  There is no longer a call to `optimizer_step` this is needed for setting learning rates that change over each step. Not sure how to fix this except by maybe having another "post_optimizer" call, which might be a better place for this style of stepping. 

* According to this tutorial https://github.com/pytorch/xla/blob/master/TROUBLESHOOTING.md `clip_grad_norm_` is very slow in XLA. The give a suggestion for a replacement function call. It would be nice if lightning just called this instead. 

* Nice to have: Point to that speed tutorial in the docs so people don't get stuck. 

* When using DDP, we have been using proc_rank to decide when to print debug statements. Might be nice to have a value that works with DDP and TPU to abstract away from using this term.



Discussion:
@srush 
can the .where also work when not on TPUs. The docs suggest that it's always a faster way of clipping grads?

Resolving patch:
diff --git a/pytorch_lightning/core/lightning.py b/pytorch_lightning/core/lightning.py
--- a/pytorch_lightning/core/lightning.py
+++ b/pytorch_lightning/core/lightning.py
@@ -65,6 +65,26 @@ def __init__(self, *args, **kwargs):
         #: True if using amp
         self.use_amp = False
 
+    def print(self, *args, **kwargs):
+        r"""
+        Prints only from process 0. Use this in any distributed mode to log only once
+
+        Args:
+            x (object): The thing to print
+
+        Example
+        -------
+
+        .. code-block:: python
+
+            # example if we were using this model as a feature extractor
+            def forward(self, x):
+                self.print(x, 'in loader')
+
+        """
+        if self.trainer.proc_rank == 0:
+            print(*args, **kwargs)
+
     @abstractmethod
     def forward(self, *args, **kwargs):
         r"""


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Add "epoch" options to basic templates
## 🚀 Feature
<!-- A clear and concise description of the feature proposal -->
Add "epochs" option to parser of 'basic_examples/lightning_module_template.py'

### Motivation
<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->
Thanks to 'basic_examples/lightning_module_template.py', I could build my deep learning model. Some beginners like me might build their model from this basic template. However, there are no options to manipulate epochs. I just thought that what people use often should be included in the basic template, so I uploaded my issue.

### Pitch
<!-- A clear and concise description of what you want to happen. -->
I suggest that the basic template includes "epoch" option in the basic template.

### Alternatives
<!-- A clear and concise description of any alternative solutions or features you've considered, if any. -->
Add "epoch" options to parser of 'basic_examples/lightning_module_template.py'
```python
parser.add_argument('--epochs', default=10, type=int, metavar='N',
                            help='number of total epochs to run')

trainer = pl.Trainer(max_epochs=hparams.epochs)
```

### Additional context
<!-- Add any other context or screenshots about the feature request here. -->
I am really enjoying PytorchLightning framework. Thanks 😄 



Discussion:
Hey, thanks for your contribution! Great first issue!
Possibly linked to #916 and would cover the alternative the op suggests.
yeah, it can be solved a part of that one, could you check on the follow-up PR? :]

Resolving patch:
diff --git a/pl_examples/basic_examples/lightning_module_template.py b/pl_examples/basic_examples/lightning_module_template.py
--- a/pl_examples/basic_examples/lightning_module_template.py
+++ b/pl_examples/basic_examples/lightning_module_template.py
@@ -250,6 +250,7 @@ def add_model_specific_args(parent_parser, root_dir):  # pragma: no cover
         parser.add_argument('--data_root', default=os.path.join(root_dir, 'mnist'), type=str)
 
         # training params (opt)
+        parser.add_argument('--epochs', default=20, type=int)
         parser.add_argument('--optimizer_name', default='adam', type=str)
         parser.add_argument('--batch_size', default=64, type=int)
         return parser


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
"RuntimeError: Address already in use" when running multiple multi-gpu training (DDP).
**Describe the bug**
I see "RuntimeError: Address already in use" error message if I try to run two multi-gpu training session (using ddp) at the same time.

**To Reproduce**
Run two multi-gpu training session at the same time.

**Expected behavior**
Able to run two multi-gpu training session at the same time.

**Screenshots**
![image](https://user-images.githubusercontent.com/1051900/67188345-15505980-f427-11e9-823c-08cafbd0289b.png)

**Desktop (please complete the following information):**
 - OS: Ubuntu 18.04
 - Pytorch 1.3.0
 - CUDA 10.1




Discussion:
you have to set master_port yourself if running 2 ddp sessions on the same machien at the same time. This is a PyTorch limitation. in all the other ddp settings you wouldn’t have to worry about this.  


https://pytorch-lightning.readthedocs.io/en/latest/Trainer/Distributed%20training/#multi-node
Great. Thanks!

Resolving patch:
diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py
--- a/pytorch_lightning/trainer/trainer.py
+++ b/pytorch_lightning/trainer/trainer.py
@@ -958,6 +958,7 @@ def fit(
                 task = int(os.environ['SLURM_LOCALID'])
                 self.ddp_train(task, model)
             else:
+                self.__set_random_port()
                 mp.spawn(self.ddp_train, nprocs=self.num_gpus, args=(model,))
 
         # 1 gpu or dp option triggers training using DP module
@@ -994,6 +995,18 @@ def fit(
         # used for testing or when we need to know that training succeeded
         return 1
 
+    def __set_random_port(self):
+        """
+        When running DDP NOT managed by SLURM, the ports might collide
+        :return:
+        """
+        try:
+            default_port = os.environ['MASTER_PORT']
+        except Exception:
+            import random
+            default_port = random.randint(10000, 19000)
+            os.environ['MASTER_PORT'] = str(default_port)
+
     def __set_fit_dataloaders(self, model, train_dataloader, val_dataloaders, test_dataloaders):
         # when dataloader is passed via fit, patch the train_dataloader
         # functions to overwrite with these implementations


Test patch: