internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: microsoft/DeepSpeed

Issue:
[REQUEST] Remove deprecated PyTorch code
**Is your feature request related to a problem? Please describe.**
DeepSpeed uses code that's deprecated. This causes problems for libraries built on deepspeed when testing.

**Describe the solution you'd like**
```
/usr/local/lib/python3.10/dist-packages/deepspeed/comm/torch.py:171: in get_global_rank
    return torch.distributed.distributed_c10d._get_global_rank(group, group_rank)
```
should instead use `get_global_rank` as recommended by the warning
```
E       UserWarning: torch.distributed.distributed_c10d._get_global_rank is deprecated please use torch.distributed.distributed_c10d.get_global_rank instead
```



Discussion:

Resolving patch:
diff --git a/deepspeed/comm/__init__.py b/deepspeed/comm/__init__.py
--- a/deepspeed/comm/__init__.py
+++ b/deepspeed/comm/__init__.py
@@ -19,7 +19,10 @@ def get_world_group():
         return group.WORLD
 
     def get_global_rank(group, group_rank):
-        from torch.distributed.distributed_c10d import _get_global_rank
+        if hasattr(torch.distributed.distributed_c10d, "get_global_rank"):
+            from torch.distributed.distributed_c10d import get_global_rank as _get_global_rank
+        else:
+            from torch.distributed.distributed_c10d import _get_global_rank
         return _get_global_rank(group, group_rank)
 
     def allgather_fn(output_tensor, input_tensor, group=None, async_op=False):
diff --git a/deepspeed/comm/torch.py b/deepspeed/comm/torch.py
--- a/deepspeed/comm/torch.py
+++ b/deepspeed/comm/torch.py
@@ -168,7 +168,11 @@ def new_group(self, ranks):
         return torch.distributed.new_group(ranks)
 
     def get_global_rank(self, group, group_rank):
-        return torch.distributed.distributed_c10d._get_global_rank(group, group_rank)
+        if hasattr(torch.distributed.distributed_c10d, "get_global_rank"):
+            from torch.distributed.distributed_c10d import get_global_rank as _get_global_rank
+        else:
+            from torch.distributed.distributed_c10d import _get_global_rank
+        return _get_global_rank(group, group_rank)
 
     def get_world_group(self):
         return torch.distributed.group.WORLD


Test patch:
Repository: microsoft/DeepSpeed

Issue:
deepspeed/launcher/launch.py: add option enable_each_rank_log
It is not easy to check the console output because all the ranks output logs (with logs from underlying software libraries) into console. With this patch, the output of each rank are dumped into different log files.


Discussion:
One other request, in order to merge we need to address the formatting errors. Can you run our formatter? For details see: https://github.com/microsoft/DeepSpeed/blob/master/CONTRIBUTING.md#prerequisites
looks that we are still waiting for the formatting and unit-tests, anything I need to do to trigger them? thanks.
I take a look at the CI failure, and they are also there at both https://github.com/microsoft/DeepSpeed/pull/2429 and https://github.com/microsoft/DeepSpeed/pull/2428.  I think it is a CI system issue, not caused by this PR.

FAILED unit/inference/test_inference.py::TestModelTask::test[EleutherAI/gpt-j-6B-text-generation-fp16-noCG] 
good to see "All checks have passed"
any other comment? thanks.
any comment?  will click 'Update branch' in the webpage.

Resolving patch:
diff --git a/deepspeed/launcher/runner.py b/deepspeed/launcher/runner.py
--- a/deepspeed/launcher/runner.py
+++ b/deepspeed/launcher/runner.py
@@ -154,6 +154,12 @@ def parse_args(args=None):
         "where <main-pid> is the pid of the first process that invoked `deepspeed`. "
         "Useful when launching deepspeed processes programmatically.")
 
+    parser.add_argument(
+        "--enable_each_rank_log",
+        default="None",
+        type=str,
+        help="redirect the stdout and stderr from each rank into different log files")
+
     parser.add_argument(
         "--autotuning",
         default="",
@@ -481,6 +487,9 @@ def main(args=None):
             deepspeed_launch.append("--no_local_rank")
         if args.save_pid:
             deepspeed_launch += ["--save_pid", f"{os.getpid()}"]
+        if args.enable_each_rank_log:
+            deepspeed_launch.append(
+                f"--enable_each_rank_log={args.enable_each_rank_log}")
         if args.elastic_training:
             deepspeed_launch.append("--enable_elastic_training")
             deepspeed_launch.append(f"--max_elastic_nodes={args.max_elastic_nodes}")


Test patch:
Repository: microsoft/DeepSpeed

Issue:
[BUG] pydantic DeepSpeedConfigModel has no validator for <class:object>
**Describe the bug**
During ```from deepspeed.inference.config import DtypeEnum```, got error 
```
 File "pydantic/main.py", line 299, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/fields.py", line 411, in pydantic.fields.ModelField.infer
  File "pydantic/fields.py", line 342, in pydantic.fields.ModelField.__init__
  File "pydantic/fields.py", line 456, in pydantic.fields.ModelField.prepare
  File "pydantic/fields.py", line 670, in pydantic.fields.ModelField.populate_validators
  File "pydantic/validators.py", line 715, in find_validators
RuntimeError: no validator found for <class 'object'>, see `arbitrary_types_allowed` in Config
```

**To Reproduce**
Steps to reproduce the behavior:
1. Simple inference script to reproduce ```from deepspeed.inference.config import DtypeEnum```
2. pydantic 1.8.2, deepspeed 0.8.0+384f17b


**Expected behavior**
successful import with no error

**ds_report output**
cannot produce due to the same import error

**System info (please complete the following information):**
 - OS: Red Hat Enterprise Linux Server 7.9 (Maipo)
 - GPU count and types: one machine with 8 A100s, three machines with 8 A100s each
 - Hugging Face Transformers 4.19.2, no accelerate
 - Python version 3.8.13



Discussion:
There are many errors like this that can be fixed with adding ```arbitrary_types_allowed = True``` for the pydantic baseclass (see this Pydantic [issue](https://github.com/pydantic/pydantic/issues/3320)). However, there are many child classes of the base class, and adding this option one by one does not seem like the best solution. Did I have a package version mismatch?
Hi @Lavenderjiang, try updating pydantic with `pip install pydantic --upgrade`

Resolving patch:
diff --git a/deepspeed/runtime/config_utils.py b/deepspeed/runtime/config_utils.py
--- a/deepspeed/runtime/config_utils.py
+++ b/deepspeed/runtime/config_utils.py
@@ -109,6 +109,7 @@ class Config:
         use_enum_values = True
         allow_population_by_field_name = True
         extra = "forbid"
+        arbitrary_types_allowed = True
 
 
 class pp_int(int):


Test patch:
Repository: microsoft/DeepSpeed

Issue:
[BUG] the `benchmarks` folder is included upon installation
I noticed that while inspecting the conda package during my attempt to create a conda forge build.

![image](https://user-images.githubusercontent.com/528003/211929466-436decd1-a733-4db6-8a67-c948736575a5.png)

The fix is likely as simple as adding `benchmarks` to `packages=find_packages(exclude=[....])` in the `setup.py` file.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -291,7 +291,9 @@ def create_dir_symlink(src, dest):
           "release",
           "requirements",
           "scripts",
-          "tests"
+          "tests",
+          "benchmarks",
+          "accelerator"
       ]),
       include_package_data=True,
       scripts=[


Test patch:
Repository: microsoft/DeepSpeed

Issue:
[BUG] No module named 'torch._six'
Torch does not seem to support torch._six anymore and it has been removed.
Refer - https://github.com/pytorch/pytorch/pull/94709

DeepSpeed still has dependency on it. Example in `runtime/utils.py`

```
from torch._six import inf
```

Torch version to be precise - 2.0.0a0+git5a54537


Discussion:
Please apply fix " from torch import inf", please let us know if this does not fix your issue. 
I cloned git master (8be8c01) now, I did not modify deepspeed, ran into same problem. I used `torch==2.0.0.dev20230217+cu118`.

Changing to `from torch import inf` in all files works.

Resolving patch:
diff --git a/deepspeed/runtime/utils.py b/deepspeed/runtime/utils.py
--- a/deepspeed/runtime/utils.py
+++ b/deepspeed/runtime/utils.py
@@ -16,9 +16,13 @@
 from bisect import bisect_left
 
 import torch
-from torch._six import inf
 from deepspeed import comm as dist
 
+try:
+    from torch._six import inf as inf
+except ModuleNotFoundError:
+    from torch import inf as inf
+
 from deepspeed.utils import groups, logger
 from deepspeed.runtime.constants import PIPE_REPLICATED
 from numpy import prod
diff --git a/deepspeed/runtime/zero/stage3.py b/deepspeed/runtime/zero/stage3.py
--- a/deepspeed/runtime/zero/stage3.py
+++ b/deepspeed/runtime/zero/stage3.py
@@ -7,13 +7,12 @@
 import gc
 import collections
 from typing import Deque, Dict, Tuple
-from torch._six import inf
 
 from deepspeed.runtime import ZeROOptimizer
 from deepspeed.utils import logger
 from deepspeed.runtime.fp16.loss_scaler import LossScaler, DynamicLossScaler
 from deepspeed.runtime.comm.coalesced_collectives import reduce_scatter_coalesced
-from deepspeed.runtime.utils import get_global_norm, is_model_parallel_parameter
+from deepspeed.runtime.utils import inf, get_global_norm, is_model_parallel_parameter
 from deepspeed.runtime.zero.partition_parameters import *
 from deepspeed.runtime.zero.config import ZeroStageEnum
 from deepspeed.runtime.zero.offload_config import OffloadDeviceEnum
diff --git a/deepspeed/runtime/zero/stage_1_and_2.py b/deepspeed/runtime/zero/stage_1_and_2.py
--- a/deepspeed/runtime/zero/stage_1_and_2.py
+++ b/deepspeed/runtime/zero/stage_1_and_2.py
@@ -5,7 +5,6 @@
 import torch
 import os
 from deepspeed import comm as dist
-from torch._six import inf
 from packaging import version as pkg_version
 from collections import OrderedDict
 
@@ -15,6 +14,7 @@
                                      get_global_norm,
                                      empty_cache,
                                      see_memory_usage,
+                                     inf,
                                      is_model_parallel_parameter,
                                      align_dense_tensors,
                                      all_gather_dp_groups)


Test patch:
Repository: microsoft/DeepSpeed

Issue:
Add Full Apache License
#3111 did not include the full Apache 2.0 License text. Adding that here.

@jeffra 


Discussion:
Awesome :)

Resolving patch:
diff --git a/deepspeed/utils/zero_to_fp32.py b/deepspeed/utils/zero_to_fp32.py
--- a/deepspeed/utils/zero_to_fp32.py
+++ b/deepspeed/utils/zero_to_fp32.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python
+
 # Copyright (c) Microsoft Corporation.
 # SPDX-License-Identifier: Apache-2.0
 


Test patch:
Repository: microsoft/DeepSpeed

Issue:
Mention default launcher value in the help menu
Could you please add the launcher default value in the help menu?

https://github.com/microsoft/DeepSpeed/blob/ab1d2f826b5e1365b9e68b59e4d13e27c6f1065e/deepspeed/launcher/runner.py#L117


Discussion:

Resolving patch:
diff --git a/deepspeed/launcher/runner.py b/deepspeed/launcher/runner.py
--- a/deepspeed/launcher/runner.py
+++ b/deepspeed/launcher/runner.py
@@ -40,7 +40,8 @@
 
 def parse_args(args=None):
     parser = argparse.ArgumentParser(description="DeepSpeed runner to help launch distributed "
-                                     "multi-node/multi-gpu training jobs.")
+                                     "multi-node/multi-gpu training jobs.",
+                                     formatter_class=argparse.ArgumentDefaultsHelpFormatter)
 
     parser.add_argument("-H",
                         "--hostfile",


Test patch:
Repository: microsoft/DeepSpeed

Issue:
[BUG] FP16 used for all reduce even if BFLOAT16 is enabled
Even if BFLOAT16 is enabled, unless the communication datatype is also explicitly set to bfp16, fp16 is still used: https://github.com/microsoft/DeepSpeed/blob/9b70ce56e7af89d5226f9b06ebe1137407f371dc/deepspeed/runtime/engine.py#L702-L711

Moreover in zero, even if the communication datatype is set to bfp16 at some place the default value is fp16
https://github.com/microsoft/DeepSpeed/blob/9b70ce56e7af89d5226f9b06ebe1137407f371dc/deepspeed/runtime/zero/stage_1_and_2.py#L1334-L1338




Discussion:
@owmohamm, can you please try PR #2145?

Resolving patch:
diff --git a/deepspeed/runtime/engine.py b/deepspeed/runtime/engine.py
--- a/deepspeed/runtime/engine.py
+++ b/deepspeed/runtime/engine.py
@@ -778,6 +778,9 @@ def communication_data_type(self):
         if self.fp16_enabled():
             return torch.float16
 
+        if self.bfloat16_enabled():
+            return torch.bfloat16
+
         return torch.float32
 
     def postscale_gradients(self):


Test patch:
diff --git a/tests/unit/runtime/half_precision/test_bf16.py b/tests/unit/runtime/half_precision/test_bf16.py
--- a/tests/unit/runtime/half_precision/test_bf16.py
+++ b/tests/unit/runtime/half_precision/test_bf16.py
@@ -287,7 +287,7 @@ def test(self, stage=2):
 
 
 @pytest.mark.parametrize("comp_type", [torch.float16, torch.bfloat16, torch.float], ids=["fp16", "bfp16", "fp32"])
-@pytest.mark.parametrize("comm_type", [torch.float16, torch.bfloat16], ids=["fp16", "bfp16"])
+@pytest.mark.parametrize("comm_type", [torch.float16, torch.bfloat16, None], ids=["fp16", "bfp16", "default"])
 class TestZeroDtypeCocktail(DistributedTest):
     world_size = 2
 
@@ -312,8 +312,11 @@ def test(self, comp_type, comm_type):
             "zero_optimization": {
                 "stage": 2
             },
-            "communication_data_type": type_str[comm_type]
         }
+        if comm_type is not None:
+            config_dict["communication_data_type"] = type_str[comm_type]
+        else:
+            comm_type = comp_type
         hidden_dim = 10
 
         model = SimpleModel(hidden_dim)
Repository: microsoft/DeepSpeed

Issue:
[ERROR] AttributeErrorAttributeErrorAttributeError:  'NoneType' object has no attribute 'get_tensor_model_parallel_group'
https://github.com/microsoft/DeepSpeed/blob/d10b8ca011b18eba3a6ca56f4208a732d7fbb744/deepspeed/runtime/hybrid_engine.py#LL321C16-L321C37

Should the judgment condition be "self.mpu is None"?


Discussion:
I am seeing the same issue with DeepSpeed version 0.9.2:
```
self.mpu.get_tensor_model_parallel_group()
    engine = DeepSpeedHybridEngine(args=args,
AttributeError:   File "/home/ubuntu/miniconda3/envs/pytorch/lib/python3.10/site-packages/deepspeed/runtime/hybrid_engine.py", line 52, in __init__
'NoneType' object has no attribute 'get_tensor_model_parallel_group'
    self.create_inference_module()
  File "/home/ubuntu/miniconda3/envs/pytorch/lib/python3.10/site-packages/deepspeed/runtime/hybrid_engine.py", line 335, in create_inference_module
    self.mpu.get_tensor_model_parallel_group()
AttributeError: 'NoneType' object has no attribute 'get_tensor_model_parallel_group'
    actor_engine, *_ = deepspeed.initialize(model=actor_model,
  File "/home/ubuntu/miniconda3/envs/pytorch/lib/python3.10/site-packages/deepspeed/__init__.py", line 153, in initialize
    engine = DeepSpeedHybridEngine(args=args,
  File "/home/ubuntu/miniconda3/envs/pytorch/lib/python3.10/site-packages/deepspeed/runtime/hybrid_engine.py", line 52, in __init__
    self.create_inference_module()
  File "/home/ubuntu/miniconda3/envs/pytorch/lib/python3.10/site-packages/deepspeed/runtime/hybrid_engine.py", line 335, in create_inference_module
    self.mpu.get_tensor_model_parallel_group()
AttributeError: 'NoneType' object has no attribute 'get_tensor_model_parallel_group'
```
@sunhaohai yes I believe you are correct. This should be `if self.mpu is None`. I will add a PR to fix this. Thank you!

Resolving patch:
diff --git a/deepspeed/runtime/hybrid_engine.py b/deepspeed/runtime/hybrid_engine.py
--- a/deepspeed/runtime/hybrid_engine.py
+++ b/deepspeed/runtime/hybrid_engine.py
@@ -318,7 +318,7 @@ def create_inference_module(self):
         self._orig_fwds_others = []
 
         if self._config.hybrid_engine.inference_tp_size > 1:
-            if self.mpu is not None:
+            if self.mpu is None:
                 global_rank = dist.get_rank()
                 world_size = dist.get_world_size()
                 mp_group_id = global_rank // self._config.hybrid_engine.inference_tp_size


Test patch:
Repository: microsoft/DeepSpeed

Issue:
`TypeError: WarmupLR.step() got an unexpected keyword argument 'increment'`
- Related to:

  https://github.com/microsoft/Megatron-DeepSpeed/issues/215

- Relevant line:

  https://github.com/microsoft/DeepSpeed/blob/9d79cfd1e90cae9306dc1b5837d374b2c9489ac8/deepspeed/runtime/engine.py#L2037

- Traceback:

  ```python
  ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
  │ /lus/grand/projects/datascience/foremans/locations/thetaGPU/miniconda3/envs/2023-04-26/lib/pytho │
  │ n3.10/site-packages/deepspeed/runtime/engine.py:2032 in _take_model_step                         │
  │                                                                                                  │
  │   2029 │   │   │   self.compression_scheduler.step()                                             │
  │   2030 │   │   │   if self.lr_scheduler is not None:                                             │
  │   2031 │   │   │   │   try:                                                                      │
  │ ❱ 2032 │   │   │   │   │   self.lr_scheduler.step(**(lr_kwargs or {}))                           │
  │   2033 │   │   │   │   except TypeError:                                                         │
  │   2034 │   │   │   │   │   # XXX Hack to work with Megatron 2.0 and DeepSpeed pipelines.         │
  │   2035 │   │   │   │   │   # We don't currently have a way to specify lr_kwargs from             │
  ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
  TypeError: WarmupLR.step() got an unexpected keyword argument 'increment'
  ```

- Fix:

  - Replace:

    ```python
    self.lr_scheduler.step(increment=self.train_batch_size())
    ```

    with

    ```python
    self.lr_scheduler.step(self.train_batch_size())
    ```
    (i.e. passing increment as an argument without the increment= keyword) seems to fix it.


Discussion:
@saforem2, your proposal is a good solution. Do you mind submitting a PR? Thanks! 
Yeah no problem.

I was actually about to but I think my fork at [`saforem2/DeepSpeed`](https://github.com/saforem2/DeepSpeed) is mangled somehow

edit: actually, maybe I'll just fork it to another organization really quick. 1 sec

Resolving patch:
diff --git a/deepspeed/runtime/engine.py b/deepspeed/runtime/engine.py
--- a/deepspeed/runtime/engine.py
+++ b/deepspeed/runtime/engine.py
@@ -2034,7 +2034,7 @@ def _take_model_step(self, lr_kwargs, block_eigenvalue={}):
                     # XXX Hack to work with Megatron 2.0 and DeepSpeed pipelines.
                     # We don't currently have a way to specify lr_kwargs from
                     # pipe_engine.train_batch()
-                    self.lr_scheduler.step(increment=self.train_batch_size())
+                    self.lr_scheduler.step(self.train_batch_size())
 
         if report_progress and (self.global_steps + 1) % self.steps_per_print() == 0:
             self._report_progress(self.global_steps + 1)


Test patch:
Repository: microsoft/DeepSpeed

Issue:
[REQUEST] Handle SIGTERM
Command deepspeed can catch SIGINT and stop the subprocess ([code](https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/launcher/runner.py#L580)).

In Kubernetes, kubelet sends process SIGTERM, which is not handled by deepspeed, before closing a container. If deepspeed can handle SIGTERM and transmit to training processes, it will be helpful.


Discussion:
Hi @tswangdi - what launcher are you using?  To make sure I understand your ask correctly, you're seeing a scenario where the SIGTERM isn't being handled but it is in the code you pointed out?
Thanks for the reply.

`what launcher are you using?`  PDSH_LAUNCHER

I am running deepspeed in a Kubernetes Pod. When i delete the pod, kubelet will send SIGTERM to deepspeed. But deepspeed runner only catch SIGINT, and send SIGINT/SIGTERM to subprocesses.  Deepspeed runner should catch more signals (like SIGSTOP, SIGTERM) and stop subprocesses gracefully.

Now, when the runner receives a SIGTERM, it will shut down, but the subprocesses will still keep running.


Resolving patch:
diff --git a/deepspeed/launcher/runner.py b/deepspeed/launcher/runner.py
--- a/deepspeed/launcher/runner.py
+++ b/deepspeed/launcher/runner.py
@@ -578,6 +578,7 @@ def sigkill_handler(signum, frame):
 
     if args.launcher == PDSH_LAUNCHER and multi_node_exec:
         signal.signal(signal.SIGINT, sigkill_handler)
+        signal.signal(signal.SIGTERM, sigkill_handler)
 
     result.wait()
 


Test patch:
Repository: microsoft/DeepSpeed

Issue:
[REQUEST] Add timeout as entry-point option or environment variable
**Is your feature request related to a problem? Please describe.**
I am using Hugging Face `transformers` for my deep learning, and it has a nice option to restrict specific processing to the main process only. This is useful if a function caches the result: the main process does the processing while the other processes wait, and when main is done, the other processes can just load from the cache. That's pretty neat.

The problem arises when these are long running processes. In distributed environment (torch or deepspeed, for instance), the communication between processes has a default timeout. If no communication has occurred for `timeout` seconds, the whole program will exit. 

**Describe the solution you'd like**

Both [`torch`](https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group) and [`deepspeed`](https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group) provide options in the Python init methods to set the timeout parameter to a higher value than the default 30 minutes, but this option is not available from the command-line or through an environment, which is what I would like.

**Describe alternatives you've considered**
I could make a custom fork but I think that this is something that more people might need as soon as they scale to larger projects.

**Additional context**

I can work on this, depending on what you suggest as a solution (CLI argument for the `deepspeed` command or as environment variable).


Discussion:
I think the solution can be as simple as turning this line

https://github.com/microsoft/DeepSpeed/blob/60bf78454c976c59eb9a77ea9ef92ca863669886/deepspeed/constants.py#L18

Into 

```python
default_pg_timeout = timedelta(minutes=os.getenv("DEEPSPEED_TIMEOUT", default=30))
```

Any thoughts on this?

Resolving patch:
diff --git a/deepspeed/constants.py b/deepspeed/constants.py
--- a/deepspeed/constants.py
+++ b/deepspeed/constants.py
@@ -3,6 +3,7 @@
 
 # DeepSpeed Team
 
+import os
 from datetime import timedelta
 
 #############################################
@@ -15,6 +16,6 @@
 # (only if NCCL_BLOCKING_WAIT or NCCL_ASYNC_ERROR_HANDLING is set to 1).
 # To make an attempt at backwards compatibility with THD, we use an
 # extraordinarily high default timeout, given that THD did not have timeouts.
-default_pg_timeout = timedelta(minutes=30)
+default_pg_timeout = timedelta(minutes=int(os.getenv("DEEPSPEED_TIMEOUT", default=30)))
 INFERENCE_GENERIC_MODE = 'generic'
 INFERENCE_SPECIALIZED_MODE = 'specialized'


Test patch: