internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: NVIDIA/TransformerEngine

Issue:
The fc2.bias of LayerNormMLP is not used
Hi there, thanks for your excellent work!

I observed that the parameter `fc2.bias` is not used when forwarding `te.LayerNormMLP`.

The reason is that `use_bias` is set to `False` when calling [`_LayerNormMLP.apply`](https://github.com/NVIDIA/TransformerEngine/blob/main/transformer_engine/pytorch/module.py#L2361), therefore `fc2.bias` is not used for both FP8 (https://github.com/NVIDIA/TransformerEngine/blob/main/transformer_engine/pytorch/module.py#L1727) and FP32 (https://github.com/NVIDIA/TransformerEngine/blob/main/transformer_engine/pytorch/module.py#L1761).


Discussion:

Resolving patch:
diff --git a/transformer_engine/pytorch/module.py b/transformer_engine/pytorch/module.py
--- a/transformer_engine/pytorch/module.py
+++ b/transformer_engine/pytorch/module.py
@@ -2372,7 +2372,7 @@ def forward(
             self.weight2_fp8 if self.fp8 else None,
             self.weight2_t_fp8 if self.fp8 else None,
             self.fc2_bias,
-            False,  # use_bias set to False for RPL
+            self.use_bias,
             self.eps,
             is_first_microbatch,
             self.fp8,


Test patch:
Repository: NVIDIA/TransformerEngine

Issue:
Upgrade `flash-attn` to 1.0.7?
#254 set the version at 1.0.6. It seems that now the container uses version 1.0.7, so to stay compatible, should we update as well, or is there something stopping us from doing that?


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -290,7 +290,7 @@ def add_unique(l: List[str], vals: Union[str, List[str]]) -> None:
 
     # Framework-specific requirements
     if "pytorch" in frameworks():
-        add_unique(install_reqs, ["torch", "flash-attn==1.0.6"])
+        add_unique(install_reqs, ["torch", "flash-attn>=1.0.6, <=1.0.7"])
         add_unique(test_reqs, ["numpy", "onnxruntime", "torchvision"])
     if "jax" in frameworks():
         if not found_pybind11():
diff --git a/transformer_engine/pytorch/attention.py b/transformer_engine/pytorch/attention.py
--- a/transformer_engine/pytorch/attention.py
+++ b/transformer_engine/pytorch/attention.py
@@ -35,7 +35,7 @@
 from transformer_engine.pytorch.export import is_in_onnx_export_mode
 
 _flash_attn_version = packaging.version.Version(version("flash-attn"))
-_flash_attn_version_required = packaging.version.Version("1.0.2")
+_flash_attn_version_required = packaging.version.Version("1.0.6")
 
 
 __all__ = ["DotProductAttention"]


Test patch:
Repository: NVIDIA/TransformerEngine

Issue:
Check that the layer number is not 0
This parameter https://github.com/NVIDIA/TransformerEngine/blob/574f1b41e7473c3108b305b3ddef271392ddd9d5/transformer_engine/pytorch/attention.py#L719-L721

should indicate that the number must be 1-index based. An assertion could be added to enforce it.

Otherwise, if the user accidentally passes `0`, then https://github.com/NVIDIA/TransformerEngine/blob/main/transformer_engine/pytorch/attention.py#L206 will raise a division by 0 error after this multiplication: https://github.com/NVIDIA/TransformerEngine/blob/main/transformer_engine/pytorch/attention.py#L198


Discussion:

Resolving patch:
diff --git a/transformer_engine/pytorch/attention.py b/transformer_engine/pytorch/attention.py
--- a/transformer_engine/pytorch/attention.py
+++ b/transformer_engine/pytorch/attention.py
@@ -977,7 +977,7 @@ def __init__(
         bias: bool = True,
     ) -> None:
         super().__init__()
-        self.layer_number = (layer_number,)
+        self.layer_number = layer_number
         self.input_layernorm = input_layernorm
         self.attention_type = attention_type
         self.get_rng_state_tracker = get_rng_state_tracker
@@ -991,9 +991,9 @@ def __init__(
             qkv_weight_interleaved = False
         self.qkv_weight_interleaved = qkv_weight_interleaved
 
-        assert (
-            attention_type in AttnTypes
-        ), f"attention_type {attention_type} not supported"
+        assert attention_type in AttnTypes, f"attention_type {attention_type} not supported"
+        if layer_number is not None:
+            assert layer_number > 0, "layer_number must be a positive integer"
 
         tp_size = tp_size if tp_group is None else get_distributed_world_size(tp_group)
         self.tp_size = tp_size
@@ -1090,7 +1090,7 @@ def __init__(
             attn_mask_type=attn_mask_type,
             sequence_parallel=sequence_parallel,
             tp_group=tp_group,
-            layer_number=layer_number,
+            layer_number=self.layer_number,
         )
 
         # Linear


Test patch:
Repository: NVIDIA/TransformerEngine

Issue:
The version v1.2 has compatibility issues with PyTorch 2.0.1, causing errors during execution.
Hello,

I encountered the following error while using version v1.2 of Transformer Engine: 

`no_torch_dynamo = lambda recursive=True: lambda f: torch._dynamo.disable(f, recursive=recursive). The error message is TypeError: disable() got an unexpected keyword argument 'recursive'.
`

My environment setup is as follows: CUDA 11.8, PyTorch 2.0.1, and Python 3.10. I am experiencing this issue specifically in the context of using MegatronLM at commit fab0bd6 for Large Language Model (LLM) training. The error occurs right at the start of the training.

Possible cause of the issue:
The `disable` function in PyTorch 2.0.1 is defined as `def disable(f)`, and the recursive parameter was introduced only in later versions starting from PyTorch 2.1.0.


Discussion:
Thank you @jindajia for reporting this issue. I will work on a solution to this issue. In the meantime you can work around it by reverting commit https://github.com/NVIDIA/TransformerEngine/commit/7e7f09207a0b05c6e262075386d02e1743dff6c7

> Thank you @jindajia for reporting this issue. I will work on a solution to this issue. In the meantime you can work around it by reverting commit [7e7f092](https://github.com/NVIDIA/TransformerEngine/commit/7e7f09207a0b05c6e262075386d02e1743dff6c7)

Hello @ptrendx ,

The commit [7e7f092] appears to conflict with transformers.
  ```
File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1354, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1366, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.trainer because of the following error (look up to see its traceback):
Failed to import transformers.integrations.integration_utils because of the following error (look up to see its traceback):
disable() got an unexpected keyword argument 'recursive'
```

My env:
```
python 3.8
torch 2.0.1+cu118
transformer-engine-1.2.0.dev0+7e7f092
transformer 4.36.2/4.38.0.dev0

```

Thank you!




Resolving patch:
diff --git a/transformer_engine/pytorch/jit.py b/transformer_engine/pytorch/jit.py
--- a/transformer_engine/pytorch/jit.py
+++ b/transformer_engine/pytorch/jit.py
@@ -22,7 +22,12 @@
 no_torch_dynamo = lambda recursive=True: lambda func: func
 if torch.__version__ >= "2":
     import torch._dynamo
-    no_torch_dynamo = lambda recursive=True: lambda f: torch._dynamo.disable(f, recursive=recursive)
+    if torch.__version__ >= "2.1":
+        no_torch_dynamo = lambda recursive=True: lambda f: \
+                                                    torch._dynamo.disable(f, recursive=recursive)
+    else:
+        # no "recursive" option in pyTorch 2.0 - it acts as if recursive was True
+        no_torch_dynamo = lambda recursive=True: torch._dynamo.disable
 
 
 def set_jit_fusion_options() -> None:


Test patch:
Repository: NVIDIA/TransformerEngine

Issue:
Scaling factor for fwd and bwd mismatch with PP enabled
https://github.com/NVIDIA/TransformerEngine/blob/da30634a6c9ccdbb6c587b6c93b1860e4b038204/transformer_engine/pytorch/module/layernorm_linear.py#L487-L493

Currently, the scaling factor in backward is retrieved by refering fp8_meta via `ctx.fp8_meta["scaling_fwd"]` .

If enabling PP, the fp8_meta will be overwrited by another microbatch, causing mismatch between cast_to_fp8 in forward and cast_from_fp8 in backward.


Discussion:

Resolving patch:
diff --git a/transformer_engine/pytorch/module/layernorm_linear.py b/transformer_engine/pytorch/module/layernorm_linear.py
--- a/transformer_engine/pytorch/module/layernorm_linear.py
+++ b/transformer_engine/pytorch/module/layernorm_linear.py
@@ -484,9 +484,9 @@ def backward(
                         )
                         clear_tensor_data(ln_out_total_t, grad_output_t)
                     else:
-                        ln_out_total_c = tex.cast_from_fp8(
+                        ln_out_total_c = torch.ops.tex_ts.cast_from_fp8_ts(
                             ln_out_total,
-                            ctx.fp8_meta["scaling_fwd"],
+                            fwd_scale_inverses,
                             tex.FP8FwdTensors.GEMM1_INPUT,
                             fp8_dtype_forward,
                             TE_DType[ctx.activation_dtype],
diff --git a/transformer_engine/pytorch/module/layernorm_mlp.py b/transformer_engine/pytorch/module/layernorm_mlp.py
--- a/transformer_engine/pytorch/module/layernorm_mlp.py
+++ b/transformer_engine/pytorch/module/layernorm_mlp.py
@@ -676,9 +676,9 @@ def backward(
                     clear_tensor_data(fc1_out)
                 else:
                     if fc2_weight.requires_grad:
-                        gelu_out_c = tex.cast_from_fp8(
+                        gelu_out_c = torch.ops.tex_ts.cast_from_fp8_ts(
                             gelu_out,
-                            ctx.fp8_meta["scaling_fwd"],
+                            fwd_scale_inverses,
                             tex.FP8FwdTensors.GEMM2_INPUT,
                             fp8_dtype_forward,
                             TE_DType[ctx.activation_dtype],
@@ -875,9 +875,9 @@ def backward(
                         )
                         clear_tensor_data(ln_out_total_t, dgelu_t)
                     else:
-                        ln_out_total_c = tex.cast_from_fp8(
+                        ln_out_total_c = torch.ops.tex_ts.cast_from_fp8_ts(
                             ln_out_total,
-                            ctx.fp8_meta["scaling_fwd"],
+                            fwd_scale_inverses,
                             tex.FP8FwdTensors.GEMM1_INPUT,
                             fp8_dtype_forward,
                             TE_DType[ctx.activation_dtype],


Test patch:
Repository: NVIDIA/TransformerEngine

Issue:
`warnings.simplefilter('default')` in global scope causes excessive DeprecationWarnings
https://github.com/NVIDIA/TransformerEngine/blob/f85553ea369da15fd726ab279818e415be48a228/transformer_engine/common/utils.py#L9

Importing the `transformer_engine.common.utils` resets the warning filters to default settings using `warnings.simplefilter('default')` in the global scope. This results in the console being flooded with DeprecationWarnings, which are normally ignored by Python by default.

Would it be possible to move setting the warning filter config to a more controlled scope in this module?


Discussion:
Makes sense.

@ksivaman Could you take a look at that?

Resolving patch:
diff --git a/transformer_engine/common/utils.py b/transformer_engine/common/utils.py
--- a/transformer_engine/common/utils.py
+++ b/transformer_engine/common/utils.py
@@ -6,7 +6,8 @@
 import warnings
 from enum import Enum
 
-warnings.simplefilter('default')
+warnings.filterwarnings(
+    "module", category=DeprecationWarning, module="transformer_engine.common.utils")
 
 
 class DeprecatedEnum:    # pylint: disable=too-few-public-methods


Test patch:
Repository: dask/distributed

Issue:
Wait/sync doesn't break on KeyboardInterrupt
Calling a function like `wait` that uses `sync` can not be broken with `KeyboardInterrupt`.  Two questions
1.  How does one test keyboard interrupts?  Is there a way to simulate the signal?
2.  What could be happening here?  We are waiting on a threading `Event` when the `KeyboardInterrupt` _should_ interrupt us.  https://github.com/blaze/distributed/blob/master/distributed/utils.py#L104



Discussion:
For 1.) you can send the `signal.SIGINT` to the process

Hrm, so this works fine on my machine:

``` python

In [1]: from distributed import Executor
e = ^[[A
In [2]: e = Executor('127.0.0.1:8786')

In [3]: from time import sleep

In [4]: futures = e.map(sleep, range(100))

In [5]: from distributed import Executor, wait

In [6]: wait(futures)
KeyboardInterrupt                         Traceback (most recent call last)

In [7]: 
```

Any thoughts on how to reproduce this?  cc @danielfrg 

What didn't worked for me was the interrupt button on the jupyter notebook, note sure if thats the same signal.

OK, this all works perfectly in Python 3 and crashes in Python 2.  I can reproduce this.  Thanks!


Resolving patch:
diff --git a/distributed/utils.py b/distributed/utils.py
--- a/distributed/utils.py
+++ b/distributed/utils.py
@@ -101,7 +101,8 @@ def f():
             e.set()
 
     a = loop.add_callback(f)
-    e.wait()
+    while not e.is_set():
+        e.wait(1000000)
     if error[0]:
         raise result[0]
     else:


Test patch:
Repository: dask/distributed

Issue:
S3 anon=False fails
This looks like a trivial typo that was missed somehow?

``` python
In [18]: fs = s3fs.S3FileSystem(anon=False)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-18-e677ad34f89c> in <module>()
----> 1 fs = s3fs.S3FileSystem(anon=False)

/home/mrocklin/Software/anaconda/lib/python3.4/site-packages/distributed-1.9.1-py3.4.egg/distributed/s3fs.py in __init__(self, anon, key, secret, **kwargs)
     67         self.secret = secret
     68         self.kwargs = kwargs
---> 69         self.connect(anon, key, secret, kwargs)
     70         self.dirs = {}
     71         self.s3 = self.connect(anon, key, secret, kwargs)

/home/mrocklin/Software/anaconda/lib/python3.4/site-packages/distributed-1.9.1-py3.4.egg/distributed/s3fs.py in connect(self, anon, key, secret, kwargs)
     86                               read_timeout=self.read_timeout)
     87                 s3 = boto3.Session(self.key, self.secret, config=conf
---> 88                                    **self.kwargs).client('s3')
     89             self._conn[tok] = s3
     90         return self._conn[tok]

TypeError: unsupported operand type(s) for ** or pow(): 'Config' and 'dict'
```



Discussion:
cc @martindurant 

One test short, it seems.

I guess it's hard to test non-anonymous connections on travis.ci.  I'm mostly pointing this out as a cautionary tale.


Resolving patch:
diff --git a/distributed/s3fs.py b/distributed/s3fs.py
--- a/distributed/s3fs.py
+++ b/distributed/s3fs.py
@@ -84,8 +84,8 @@ def connect(self, anon, key, secret, kwargs):
             else:
                 conf = Config(connect_timeout=self.connect_timeout,
                               read_timeout=self.read_timeout)
-                s3 = boto3.Session(self.key, self.secret, config=conf
-                                   **self.kwargs).client('s3')
+                s3 = boto3.Session(self.key, self.secret,
+                                   **self.kwargs).client('s3', config=conf)
             self._conn[tok] = s3
         return self._conn[tok]
 


Test patch:
diff --git a/distributed/tests/test_s3fs.py b/distributed/tests/test_s3fs.py
--- a/distributed/tests/test_s3fs.py
+++ b/distributed/tests/test_s3fs.py
@@ -3,6 +3,9 @@
 from distributed.s3fs import S3FileSystem
 from distributed.s3 import seek_delimiter
 from distributed.utils_test import slow
+from distributed.utils import ignoring
+
+from botocore.exceptions import NoCredentialsError
 
 # These get mirrored on s3://distributed-test/
 test_bucket_name = 'distributed-test'
@@ -31,6 +34,12 @@ def s3():
     yield S3FileSystem(anon=True)
 
 
+def test_non_anonymous_access():
+    with ignoring(NoCredentialsError):
+        fs = S3FileSystem(anon=False)
+        fs.ls('distributed-test')
+
+
 def test_s3_file_access(s3):
     fn = 'distributed-test/nested/file1'
     data = b'hello\n'
Repository: dask/distributed

Issue:
Requires internet connection
I was surprised that `dscheduler` and even `dscheduler --help` require an internet connection. Maybe a try/except around `global_ip` that falls back to localhost with a warning? Things should still work locally right?



Discussion:
Sure, I see a couple of options
1.  Add a timeout to get_ip
2.  Don't call `get_ip` at all if the user supplies a hostname/ip

Thoughts?  Preferences?


Resolving patch:
diff --git a/distributed/cli/dscheduler.py b/distributed/cli/dscheduler.py
--- a/distributed/cli/dscheduler.py
+++ b/distributed/cli/dscheduler.py
@@ -20,8 +20,6 @@
 
 logger = logging.getLogger('distributed.scheduler')
 
-global_ip = ip = get_ip()
-
 import signal
 
 def handle_signal(sig, frame):
@@ -39,13 +37,13 @@ def handle_signal(sig, frame):
 @click.option('--bokeh/--no-bokeh', '_bokeh', default=True, show_default=True,
               required=False, help="Launch Bokeh Web UI")
 @click.option('--host', type=str, default=None,
-              help="Serving host defaults to %s" % ip)
+              help="IP or hostname of this server")
 @click.option('--show/--no-show', default=False, help="Show web UI")
 @click.option('--bokeh-whitelist', default=None, multiple=True,
               help="IP addresses to whitelist for bokeh.")
 def main(center, host, port, http_port, bokeh_port, show, _bokeh, bokeh_whitelist):
     given_host = host
-    host = host or global_ip
+    host = host or get_ip()
     ip = socket.gethostbyname(host)
     loop = IOLoop.current()
     scheduler = Scheduler(center, ip=ip,
diff --git a/distributed/utils.py b/distributed/utils.py
--- a/distributed/utils.py
+++ b/distributed/utils.py
@@ -32,8 +32,11 @@ def funcname(func):
 
 
 def get_ip(host='8.8.8.8', port=80):
-    return [(s.connect((host, port)), s.getsockname()[0], s.close())
-            for s in [socket.socket(socket.AF_INET, socket.SOCK_DGRAM)]][0][1]
+    try:
+        return [(s.connect((host, port)), s.getsockname()[0], s.close())
+                for s in [socket.socket(socket.AF_INET, socket.SOCK_DGRAM)]][0][1]
+    except OSError:
+        return '127.0.0.1'
 
 
 @contextmanager


Test patch:
Repository: dask/distributed

Issue:
Executor.map on Iterator hangs
The helper thread spawned when Executor.map is invoked on an Iterator never puts a StopIteration into the queue, so the iterator returned (queue_to_iterator) blocks indefinitely.
I will attach a pull request with a test which shows this and a fix (the same fix, btw, is already implemented in scatter, which follows the same pattern).

Note that without the bugfix, the test hangs, and because it is blocking in multithreading, a timeout on the event loop doesn't help. I hope to reimplement this with coroutines and PR soon.



Discussion:

Resolving patch:
diff --git a/distributed/executor.py b/distributed/executor.py
--- a/distributed/executor.py
+++ b/distributed/executor.py
@@ -524,6 +524,7 @@ def _threaded_map(self, q_out, func, qs_in, **kwargs):
             try:
                 args = [get(q) for q in qs_in]
             except StopIteration:
+                q_out.put(StopIteration)
                 break
             f = self.submit(func, *args, **kwargs)
             q_out.put(f)


Test patch:
diff --git a/distributed/tests/test_executor.py b/distributed/tests/test_executor.py
--- a/distributed/tests/test_executor.py
+++ b/distributed/tests/test_executor.py
@@ -2021,11 +2021,25 @@ def test_map_iterator(e, s, a, b):
     future = next(f2)
     result = yield future._result()
     assert result == (1 + 10) * 2
+    futures = list(f2)
+    results = []
+    for f in futures:
+        r = yield f._result()
+        results.append(r)
+    assert results == [(2 + 20) * 2, (3 + 30) * 2]
 
     items = enumerate(range(10))
     futures = e.map(lambda x: x, items)
+    assert isinstance(futures, Iterator)
+
     result = yield next(futures)._result()
     assert result == (0, 0)
+    futures_l = list(futures)
+    results = []
+    for f in futures_l:
+        r = yield f._result()
+        results.append(r)
+    assert results == [(i, i) for i in range(1,10)]
 
 
 @gen_cluster(executor=True)
Repository: dask/distributed

Issue:
WindowsError on dscheduler
So it appears that Bokeh is not found, but I have Bokeh and everything is up to date.

Interesting...

here lieth the traceback:

```
distributed.scheduler - INFO - Start Scheduler at:          192.168.0.7:8786
distributed.scheduler - INFO -            http at:          192.168.0.7:9786
distributed.scheduler - WARNING - Could not start Bokeh web UI
Traceback (most recent call last):
  File "C:\Anaconda\lib\site-packages\distributed\cli\dscheduler.py", line 88, in main
    bokeh_proc = subprocess.Popen(args)
  File "C:\Anaconda\lib\subprocess.py", line 710, in __init__
    errread, errwrite)
  File "C:\Anaconda\lib\subprocess.py", line 958, in _execute_child
    startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
Traceback (most recent call last):
  File "C:\Anaconda\Scripts\dscheduler-script.py", line 5, in <module>
    sys.exit(distributed.cli.dscheduler.go())
  File "C:\Anaconda\lib\site-packages\distributed\cli\dscheduler.py", line 107, in go
    main()
  File "C:\Anaconda\lib\site-packages\click\core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "C:\Anaconda\lib\site-packages\click\core.py", line 696, in main
    rv = self.invoke(ctx)
  File "C:\Anaconda\lib\site-packages\click\core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Anaconda\lib\site-packages\click\core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "C:\Anaconda\lib\site-packages\distributed\cli\dscheduler.py", line 100, in main
    bokeh_proc.terminate()
UnboundLocalError: local variable 'bokeh_proc' referenced before assignment
```



Discussion:
Indeed.  That looks quite grim.  Resolving now...


Resolving patch:
diff --git a/distributed/cli/dscheduler.py b/distributed/cli/dscheduler.py
--- a/distributed/cli/dscheduler.py
+++ b/distributed/cli/dscheduler.py
@@ -62,7 +62,8 @@ def main(center, host, port, http_port, bokeh_port, show, _bokeh, bokeh_whitelis
             dirname = os.path.dirname(distributed.__file__)
             paths = [os.path.join(dirname, 'bokeh', name)
                      for name in ['status', 'tasks']]
-            binname = sys.argv[0][:-len('dscheduler')] + 'bokeh'
+            binname = 'bokeh.bat' if 'win' in sys.platform else 'bokeh'
+            binname = os.path.join(os.path.dirname(sys.argv[0]), binname)
             args = ([binname, 'serve'] + paths +
                     ['--log-level', 'warning',
                      '--check-unused-sessions=50',


Test patch: