internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: Lightning-AI/pytorch-lightning

Issue:
multi-gpu training triggers CUDA out of memory error
Hi - 

I am running into issues when going from single to multi-gpu training.  Specifically, if I switch the line

`pl.Trainer(gpus=1, precision=16, distributed_backend='ddp')`

to

`pl.Trainer(gpus=4, precision=16, distributed_backend='ddp')`

I get the dreaded CUDA out of memory error.  Is there any reason why the parallelism causes the GPU to receive more data?


Discussion:
Hi! thanks for your contribution!, great first issue!
Hi, what are your outputs of the validation_step? If there are any large tensors, it's likely they get synced back to root GPU by https://github.com/PyTorchLightning/pytorch-lightning/pull/2434 . We're working on that.

cc @williamFalcon  ^^

Resolving patch:
diff --git a/pytorch_lightning/trainer/evaluation_loop.py b/pytorch_lightning/trainer/evaluation_loop.py
--- a/pytorch_lightning/trainer/evaluation_loop.py
+++ b/pytorch_lightning/trainer/evaluation_loop.py
@@ -341,11 +341,6 @@ def _evaluate(
             elif self.is_overridden('validation_epoch_end', model=model):
                 eval_results = model.validation_epoch_end(outputs)
 
-        # aggregate ddp stats across
-        has_content = eval_results is not None and len(eval_results) > 0
-        if has_content and (self.use_ddp or self.use_ddp2):
-            self.reduce_eval_ddp(eval_results)
-
         # enable train mode again
         model.train()
 
@@ -354,19 +349,6 @@ def _evaluate(
 
         return eval_results
 
-    def reduce_eval_ddp(self, eval_results):
-        # ignore bad inputs
-        if eval_results is None or len(eval_results) == 0:
-            return
-
-        for k, v in eval_results.items():
-            if isinstance(v, dict):
-                self.reduce_eval_ddp(v)
-            elif isinstance(v, torch.Tensor):
-                dist.all_reduce(v, op=dist.reduce_op.SUM)
-                v = v / self.world_size
-                eval_results[k] = v
-
     def run_evaluation(self, test_mode: bool = False):
         # hook
         model = self.get_model()


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
DDP breaks when `python` does not refer to the correct interpreter
<!-- 
### Common bugs:
1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/PyTorchLightning/pytorch-lightning/issues/79).    
2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/PyTorchLightning/pytorch-lightning#faq)    
-->

## 🐛 Bug

If using the DDP distributed_backend, the program breaks if `python` refers to `python2` or does not exist.

### To Reproduce

Steps to reproduce the behavior:

1. Make sure the `python` command does not link to `python3`, such as on Ubuntu 18.04.
2. Run Trainer().fit with distributed_backed='ddp'

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

### Additional context

<!-- Add any other context about the problem here. -->
The problem lies at https://github.com/PyTorchLightning/pytorch-lightning/blob/0697dd306d578f0cd3c4d23b768da0106e54a095/pytorch_lightning/trainer/distrib_data_parallel.py#L422 The `python` command is hardcoded here. On many systems, `python` is a symlink to `python2`, or does not exist.


Discussion:
Hi! thanks for your contribution!, great first issue!
Same question!!!!!
it is based on your default python... @kl0211 @RitchieAlpha mind send a Pr which updating the actual python interpreter?
cc: @williamFalcon 

Resolving patch:
diff --git a/pytorch_lightning/trainer/distrib_data_parallel.py b/pytorch_lightning/trainer/distrib_data_parallel.py
--- a/pytorch_lightning/trainer/distrib_data_parallel.py
+++ b/pytorch_lightning/trainer/distrib_data_parallel.py
@@ -419,7 +419,8 @@ def spawn_ddp_children(self, model):
             full_path = abspath(command[0])
 
         command[0] = full_path
-        command = ['python'] + command
+        # use the same python interpreter and actually running
+        command = [sys.executable] + command
 
         # since this script sets the visible devices we replace the gpus flag with a number
         num_gpus = os.environ['CUDA_VISIBLE_DEVICES'].split(',').__len__()


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Utility rank_zero_only doesn't properly handle environment variable LOCAL_RANK
## 🐛 Bug

The `LOCAL_RANK` environment variable can be a string. This causes checks in the rank_zero_only decorator to fail here: https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/utilities/distributed.py#L11

### Expected behavior

We should cast `LOCAL_RANK` to an int both in the rank_zero_only check as well as in the trainer here: https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/trainer/trainer.py#L385

Then the decorator works as we expected



Discussion:

Resolving patch:
diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py
--- a/pytorch_lightning/trainer/trainer.py
+++ b/pytorch_lightning/trainer/trainer.py
@@ -382,7 +382,7 @@ def __init__(
         # we need to call this here or NVIDIA flags and other messaging in init will show on all ranks
         # this way we only show it on rank 0
         if 'LOCAL_RANK' in os.environ:
-            rank_zero_only.rank = os.environ['LOCAL_RANK']
+            rank_zero_only.rank = int(os.environ['LOCAL_RANK'])
 
         # training bookeeping
         self.total_batch_idx = 0
diff --git a/pytorch_lightning/utilities/distributed.py b/pytorch_lightning/utilities/distributed.py
--- a/pytorch_lightning/utilities/distributed.py
+++ b/pytorch_lightning/utilities/distributed.py
@@ -14,12 +14,8 @@ def wrapped_fn(*args, **kwargs):
     return wrapped_fn
 
 
-try:
-    # add the attribute to the function but don't overwrite in case Trainer has already set it
-    getattr(rank_zero_only, 'rank')
-except AttributeError:
-    rank_zero_only.rank = os.environ.get('LOCAL_RANK', 0)
-
+# add the attribute to the function but don't overwrite in case Trainer has already set it
+rank_zero_only.rank = getattr(rank_zero_only, 'rank', int(os.environ.get('LOCAL_RANK', 0)))
 
 def _warn(*args, **kwargs):
     warnings.warn(*args, **kwargs)


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
--gpus flag with add_argparse_args bug
## 🐛 Bug

When using

```
parser = Trainer.add_argparse_args(parser)
args = parser.parse_args()

trainer = Trainer.from_argparse_args(args)
```

if the user does not provide the --gpus flag, the code allocates some memory on the GPU and reports

GPU available: True, used: True

but does not use this GPU. This issue has been discovered in bolts: 

1. https://github.com/PyTorchLightning/pytorch-lightning-bolts/issues/35
2. https://github.com/PyTorchLightning/pytorch-lightning-bolts/issues/124

### To Reproduce

Steps to reproduce the behavior:

Use the following method to create the trainer object

```
parser = Trainer.add_argparse_args(parser)
args = parser.parse_args()

trainer = Trainer.from_argparse_args(args)
```

and don't pass the --gpus flag.

### Expected behavior

If --gpus flag is not provided in the script call, then Lightning should report 

GPU available: True, used: False 

with a warning, and not allocate any memory on the GPU. This way, the user can set the --gpus flag if they have missed out on it.


Discussion:

Resolving patch:
diff --git a/pytorch_lightning/trainer/distrib_data_parallel.py b/pytorch_lightning/trainer/distrib_data_parallel.py
--- a/pytorch_lightning/trainer/distrib_data_parallel.py
+++ b/pytorch_lightning/trainer/distrib_data_parallel.py
@@ -310,6 +310,9 @@ def set_distributed_mode(self, distributed_backend):
         num_cores = self.tpu_cores if self.tpu_cores is not None else 0
         rank_zero_info(f'TPU available: {XLA_AVAILABLE}, using: {num_cores} TPU cores')
 
+        if torch.cuda.is_available() and not self.on_gpu:
+            rank_zero_warn('GPU available but not used. Set the --gpus flag when calling the script.')
+
     def configure_slurm_ddp(self, num_gpu_nodes):
         self.is_slurm_managing_tasks = False
 
diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py
--- a/pytorch_lightning/trainer/trainer.py
+++ b/pytorch_lightning/trainer/trainer.py
@@ -463,8 +463,6 @@ def __init__(
                 "track_grad_norm can be an int, a float or 'inf' (infinity norm).")
         self.track_grad_norm = float(track_grad_norm)
 
-        self.on_gpu = True if (gpus and torch.cuda.is_available()) else False
-
         # tpu config
         if num_tpu_cores is not None:
             rank_zero_warn("Argument `num_tpu_cores` is now set by `tpu_cores` since v0.7.6"
@@ -532,6 +530,8 @@ def __init__(
         self.root_gpu = determine_root_gpu_device(self.data_parallel_device_ids)
         self.root_device = torch.device("cpu")
 
+        self.on_gpu = True if (self.data_parallel_device_ids and torch.cuda.is_available()) else False
+
         # tpu state flags
         self.use_tpu = False
         self.tpu_local_core_rank = None


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Subprocess launched in ddp have the wrong cwd when using hydra.
<!-- 
### Common bugs:
1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/PyTorchLightning/pytorch-lightning/issues/79).    
2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/PyTorchLightning/pytorch-lightning#faq)    
-->

## 🐛 Bug

Details: https://github.com/PyTorchLightning/pytorch-lightning/pull/2639#issuecomment-663601262. I've talked to @omry about the issue and I will send out a fix soon.

### To Reproduce

Please see the comment I posted above.

### Expected behavior

The CWD for subprocesses should be the same as that of the parent, and relative paths should work.


Discussion:

Resolving patch:
diff --git a/pytorch_lightning/trainer/distrib_data_parallel.py b/pytorch_lightning/trainer/distrib_data_parallel.py
--- a/pytorch_lightning/trainer/distrib_data_parallel.py
+++ b/pytorch_lightning/trainer/distrib_data_parallel.py
@@ -162,7 +162,8 @@ def train_fx(trial_hparams, cluster_manager, _):
 
 
 try:
-    from hydra.utils import to_absolute_path
+    from hydra.utils import to_absolute_path, get_original_cwd
+    from hydra.core.hydra_config import HydraConfig
 except ImportError:
     HYDRA_AVAILABLE = False
 else:
@@ -464,7 +465,12 @@ def spawn_ddp_children(self, model):
             env_copy['LOCAL_RANK'] = f'{local_rank}'
 
             # start process
-            proc = subprocess.Popen(command, env=env_copy)
+            # if hydra is available and initialized, make sure to set the cwd correctly
+            cwd: Optional[str] = None
+            if HYDRA_AVAILABLE:
+                if HydraConfig.initialized():
+                    cwd = get_original_cwd()
+            proc = subprocess.Popen(command, env=env_copy, cwd=cwd)
             self.interactive_ddp_procs.append(proc)
 
             # starting all processes at once can cause issues


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
to_categorical should go before get_num_classes in metrics/functional/classification.py
https://github.com/PyTorchLightning/pytorch-lightning/blob/d18b9ef9d95ffd92591f767808cc497af5bd4e1c/pytorch_lightning/metrics/functional/classification.py#L174-L178

`to_categorical` should go before `get_num_classes`, since `get_num_classes` assumes `pred` has already been turned into categories (it is doing `int(pred.max().detach().item() + 1)`).

Warnings are raised now and then with current code, for example:
```
UserWarning: You have set 10 number of classes if different from predicted (xx) and target (10) number of classes
```

If logits are passed in directly for metrics.



Discussion:
Hi! thanks for your contribution!, great first issue!
@pwwang mind send a PR? 🐰 

Resolving patch:
diff --git a/pytorch_lightning/metrics/functional/classification.py b/pytorch_lightning/metrics/functional/classification.py
--- a/pytorch_lightning/metrics/functional/classification.py
+++ b/pytorch_lightning/metrics/functional/classification.py
@@ -171,12 +171,12 @@ def stat_scores_multiple_classes(
         >>> sups
         tensor([1., 0., 1., 1.])
     """
-    num_classes = get_num_classes(pred=pred, target=target,
-                                  num_classes=num_classes)
-
     if pred.ndim == target.ndim + 1:
         pred = to_categorical(pred, argmax_dim=argmax_dim)
 
+    num_classes = get_num_classes(pred=pred, target=target,
+                                  num_classes=num_classes)
+
     tps = torch.zeros((num_classes,), device=pred.device)
     fps = torch.zeros((num_classes,), device=pred.device)
     tns = torch.zeros((num_classes,), device=pred.device)


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Use of shell=True could lead to shell injection
File: pytorch_lightning/trainer/training_io.py
Line Number: 227-233
Relevant Code: 

`            # find job id
            job_id = os.environ['SLURM_JOB_ID']
            cmd = 'scontrol requeue {}'.format(job_id)

            # requeue job
            log.info(f'requeing job {job_id}...')
            result = call(cmd, shell=True)`

From [here](https://docs.python.org/2/library/subprocess.html), "Executing shell commands that incorporate unsanitized input from an untrusted source makes a program vulnerable to shell injection, a serious security flaw which can result in arbitrary command execution. For this reason, the use of shell=True is strongly discouraged in cases where the command string is constructed from external input...shell=False disables all shell based features, but does not suffer from this vulnerability"

Meaning anything that can set the SLURM_JOB_ID environment variable can perform code execution.

The documentation also describes why you might need/want shell=True, "This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user’s home directory."

Looking at the code above, it doesn't look like we need any of these features and we can switch to shell=False with no change in functionality while gaining the security benefits of shell=False.


Discussion:
Hi! thanks for your contribution!, great first issue!
@bkhakshoor would you mind submitting a PR for it with the fix?

Resolving patch:
diff --git a/pytorch_lightning/trainer/training_io.py b/pytorch_lightning/trainer/training_io.py
--- a/pytorch_lightning/trainer/training_io.py
+++ b/pytorch_lightning/trainer/training_io.py
@@ -226,11 +226,11 @@ def sig_handler(self, signum, frame):  # pragma: no-cover
 
             # find job id
             job_id = os.environ['SLURM_JOB_ID']
-            cmd = 'scontrol requeue {}'.format(job_id)
+            cmd = ['scontrol', 'requeue', job_id]
 
             # requeue job
             log.info(f'requeing job {job_id}...')
-            result = call(cmd, shell=True)
+            result = call(cmd)
 
             # print result text
             if result == 0:


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
apex amp state dict
## 🐛 Bug

https://github.com/PyTorchLightning/pytorch-lightning/blob/25ee51bc570503f331dceecc610d0eb355e22327/pytorch_lightning/trainer/training_io.py#L310

It seems for native amp support, the scalar state dict is saved. But for non native amp, the amp state dict is not saved?



Discussion:
anyone think it should be fixed?

Resolving patch:
diff --git a/pytorch_lightning/trainer/training_io.py b/pytorch_lightning/trainer/training_io.py
--- a/pytorch_lightning/trainer/training_io.py
+++ b/pytorch_lightning/trainer/training_io.py
@@ -115,6 +115,13 @@
 else:
     XLA_AVAILABLE = True
 
+try:
+    from apex import amp
+except ImportError:
+    APEX_AVAILABLE = False
+else:
+    APEX_AVAILABLE = True
+
 try:
     import horovod.torch as hvd
 except (ModuleNotFoundError, ImportError):
@@ -317,6 +324,8 @@ def restore(self, checkpoint_path: str, on_gpu: bool):
         # restore amp scaling
         if self.use_amp and NATIVE_AMP_AVALAIBLE and 'native_amp_scaling_state' in checkpoint:
             self.scaler.load_state_dict(checkpoint['native_amp_scaling_state'])
+        elif self.use_amp and not NATIVE_AMP_AVALAIBLE and 'amp_scaling_state' in checkpoint:
+            amp.load_state_dict(checkpoint['amp_scaling_state'])
 
         # load training state (affects trainer only)
         self.restore_training_state(checkpoint)
@@ -368,6 +377,8 @@ def dump_checkpoint(self, weights_only: bool = False) -> dict:
             # save native amp scaling
             if self.use_amp and NATIVE_AMP_AVALAIBLE and not self.use_tpu:
                 checkpoint['native_amp_scaling_state'] = self.scaler.state_dict()
+            elif self.use_amp and not NATIVE_AMP_AVALAIBLE:
+                checkpoint['amp_scaling_state'] = amp.state_dict()
 
         # add the module_arguments and state_dict from the model
         model = self.get_model()
@@ -523,6 +534,8 @@ def hpc_load(self, folderpath, on_gpu):
         # restore amp scaling
         if self.use_amp and NATIVE_AMP_AVALAIBLE and 'native_amp_scaling_state' in checkpoint:
             self.scaler.load_state_dict(checkpoint['native_amp_scaling_state'])
+        elif self.use_amp and not NATIVE_AMP_AVALAIBLE and 'amp_scaling_state' in checkpoint:
+            amp.load_state_dict(checkpoint['amp_scaling_state'])
 
         if self.root_gpu is not None:
             model.cuda(self.root_gpu)


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
apex gradient clipping
#🐛 Bug

It seems when apex + gradient clipping is not handled explicitly.
I see note here https://nvidia.github.io/apex/advanced.html#gradient-clipping.
where amp.master_params needs to be used.

Is what's in this repo correct or not?


Discussion:

Resolving patch:
diff --git a/pytorch_lightning/trainer/training_loop.py b/pytorch_lightning/trainer/training_loop.py
--- a/pytorch_lightning/trainer/training_loop.py
+++ b/pytorch_lightning/trainer/training_loop.py
@@ -291,7 +291,7 @@ def transfer_batch_to_tpu(self, *args):
         """Warning: this is just empty shell for code implemented in other class."""
 
     @abstractmethod
-    def clip_gradients(self):
+    def clip_gradients(self, *args):
         """Warning: this is just empty shell for code implemented in other class."""
 
     @abstractmethod
@@ -817,7 +817,7 @@ def run_batch_backward_pass(self, split_batch, batch_idx, opt_idx, optimizer):
         # ------------------
         if self.use_amp and NATIVE_AMP_AVALAIBLE and not self.use_tpu:
             self.scaler.unscale_(optimizer)
-        self.clip_gradients()
+        self.clip_gradients(optimizer)
 
         # ------------------
         # .STEP + ZERO_GRAD
diff --git a/pytorch_lightning/trainer/training_tricks.py b/pytorch_lightning/trainer/training_tricks.py
--- a/pytorch_lightning/trainer/training_tricks.py
+++ b/pytorch_lightning/trainer/training_tricks.py
@@ -27,9 +27,17 @@
 from pytorch_lightning.core.lightning import LightningModule
 from pytorch_lightning.callbacks import GradientAccumulationScheduler
 from pytorch_lightning.loggers.base import DummyLogger
+from pytorch_lightning.utilities import NATIVE_AMP_AVALAIBLE
 from pytorch_lightning.utilities.exceptions import MisconfigurationException
 from pytorch_lightning.utilities.memory import is_oom_error, garbage_collection_cuda
 
+try:
+    from apex import amp
+except ImportError:
+    APEX_AVAILABLE = False
+else:
+    APEX_AVAILABLE = True
+
 EPSILON = 1e-6
 EPSILON_FP16 = 1e-5
 
@@ -60,14 +68,17 @@ def restore(self, *args):
     def fit(self, *args):
         """Warning: this is just empty shell for code implemented in other class."""
 
-    def clip_gradients(self):
+    def clip_gradients(self, optimizer):
 
         # this code is a modification of torch.nn.utils.clip_grad_norm_
         # with TPU support based on https://github.com/pytorch/xla/blob/master/TROUBLESHOOTING.md
         if self.gradient_clip_val <= 0:
             return
         model = self.get_model()
-        parameters = model.parameters()
+        if self.use_amp and not NATIVE_AMP_AVALAIBLE:
+            parameters = amp.master_params(optimizer)
+        else:
+            parameters = model.parameters()
         max_norm = float(self.gradient_clip_val)
         norm_type = float(2.0)
         if isinstance(parameters, torch.Tensor):


Test patch:
Repository: Lightning-AI/pytorch-lightning

Issue:
Can't use None (anymore) in checkpoint_callback
## 🐛 Bug
using None in checkpoint_callback now errors out

```
-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 562, in ddp_train
    q.put(self.checkpoint_callback.best_model_path)
AttributeError: 'NoneType' object has no attribute 'best_model_path'
```

### To Reproduce
`trainer = Trainer(checkpoint_callback=None)`

Ran into this issue from upgrading to masters, was using masters from a few commits ago before

Edit: `False` casuses the same error as well


Discussion:
@williamFalcon I saw that this issue was mentioned and supposedly fixed in the merge, but I just tested with master and I'm still getting the same error

Resolving patch:
diff --git a/pytorch_lightning/accelerator_backends/ddp_spawn_backend.py b/pytorch_lightning/accelerator_backends/ddp_spawn_backend.py
--- a/pytorch_lightning/accelerator_backends/ddp_spawn_backend.py
+++ b/pytorch_lightning/accelerator_backends/ddp_spawn_backend.py
@@ -49,7 +49,8 @@ def teardown(self, model):
         last_path = self.mp_queue.get()
 
         # transfer back the best path to the trainer
-        self.trainer.checkpoint_callback.best_model_path = best_path
+        if self.trainer.checkpoint_callback:
+            self.trainer.checkpoint_callback.best_model_path = best_path
         # todo, pass also bets score
 
         # load last weights


Test patch: