gopikrsmscs/torch-issues huggingface dataset:
The difference between input grad computed by channels last backward and the input grad computed by channels first backward of Hardswish on MPS is too large
Labels: high priority, module: nn, triaged, module: memory format, module: correctness (silent), module: mps

### 🐛 Describe the bug

I added a backward test for test_memory_format : https://github.com/pytorch/pytorch/pull/106104, and found:
The difference between input grad computed by channels last backward and the input grad computed by channels first backward of Hardswish on MPS is too large. I have no mac machine to reproduce it, and you can refer to https://github.com/pytorch/pytorch/actions/runs/5862978195/job/15895774563.

LPPool2d, ReLU, Sigmoid, Tanh, and Tanhshrink seem to have the same issue.

### Versions

PyTorch master

cc @ezyang @gchanan @zou3519 @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @jamesr66a @kulinseth @malfet @DenisVieriu97 @razarmehr @abhudev
[ONNX] ONNX doesn't support exporting non-persistent buffer included models in FakeMode
Labels: module: onnx, triaged, onnx-triaged

To avoid out of memory issue during exporting models to ONNX, we need to detach the parameters and persistent buffers with state_dict(). 

```python
  # Create the toy model with real weight.
  real_model = create_model()

  with tempfile.NamedTemporaryFile(
      prefix=model_name, suffix=".pt"
  ) as tmp_checkpoint_file:
      # Dump state_dict to a file to simulate how HuggingFace model is initialized.
      # The file will be loaded via .load_state_dict(...)
      state_dict = real_model.state_dict()
      torch.save(state_dict, tmp_checkpoint_file.name)

      with torch.onnx.enable_fake_mode() as fake_context:
          fake_args = create_args()
          fake_kwargs = create_kwargs()
          fake_model = create_model()
          if load_checkpoint_during_init:
              fake_model.load_state_dict(torch.load(tmp_checkpoint_file.name))

          # Export the model with fake inputs and parameters
          export_options = torch.onnx.ExportOptions(
              dynamic_shapes=self.dynamic_shapes,
              op_level_debug=self.op_level_debug,
              fake_context=fake_context,
          )

          export_output = torch.onnx.dynamo_export(
              fake_model,
              *fake_args,
              **fake_kwargs,
              export_options=export_options,
          )
```

However, some models, for example, GPT2, there is non-persistent buffer which can't be detached to state_dict(). Subsequently, ONNX graph complains about the missing buffers, but we don't have it in external data of the model initializer. This kind of case can be be reproduced when we use Config to `create_model()`.

cc @BowenBao @thiagocrepaldi @wschin
The difference between channels last backward and channels first backward of AvgPool2d on CUDA is too large
Labels: module: nn, module: cuda, triaged, module: memory format

### 🐛 Describe the bug

The difference between channels last backward and channels first backward of AvgPool2d on CUDA is too large.

```
### reproducer
import torch
import copy
import torch.utils._pytree as pytree

x = torch.randn(2, 3, 6, 6).cuda().requires_grad_()
x2 = x.contiguous(memory_format=torch.channels_last).cuda().detach().requires_grad_()
model = torch.nn.AvgPool2d((2, 2), (2, 2), (1, 1)).cuda()
model2 = copy.deepcopy(model).cuda()
output = model(x)
grad_out = torch.randn(output.shape).cuda()
grad_out2 = grad_out.clone().contiguous(memory_format=torch.channels_last).cuda()

output2 = model2(x2)

if isinstance(output2, torch.Tensor):
    outputs2 = (output2,)
    outputs = (output,)

# === Do backward pass. ===

ref_diff_outputs = tuple(t for t in outputs)
ref_params = tuple(p for p in model.parameters())
ref_inputs=(x,)
ref_diff_inputs = tuple(
    t
    for t in pytree.tree_flatten((ref_inputs, ref_params))[0]
)
ref_grad_outputs = tuple(
    torch.rand_like(t)
    for t in ref_diff_outputs
)

ref_grad_inputs = torch.autograd.grad(
    ref_diff_outputs,
    ref_diff_inputs,
    grad_outputs=ref_grad_outputs,
    allow_unused=True,
)

diff_outputs = tuple(t for t in outputs2)
params = tuple(p for p in model2.parameters())
inputs=(x2,)
diff_inputs = tuple(
    t
    for t in pytree.tree_flatten((inputs, params))[0]
)
grad_outputs = tuple(
    torch.rand_like(t)
    for t in diff_outputs
)
grad_outputs = tuple(
    t1.copy_(t2)
    for (t1, t2) in zip(grad_outputs, ref_grad_outputs)
)

grad_inputs = torch.autograd.grad(
    diff_outputs,
    diff_inputs,
    grad_outputs=grad_outputs,
    allow_unused=True,
)

torch.testing.assert_close(grad_inputs, ref_grad_inputs, atol=None, rtol=None)
```

The difference is too large:
```
AssertionError: Tensor-likes are not close!

Mismatched elements: 162 / 216 (75.0%)
Greatest absolute difference: 0.2320583015680313 at index (1, 2, 1, 4) (up to 1e-05 allowed)
Greatest relative difference: 245.96385192871094 at index (1, 2, 1, 4) (up to 1.3e-06 allowed)
```

### Versions

PyTorch master.

cc @ezyang @gchanan @zou3519 @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @ptrblck @jamesr66a
DISABLED test_make_fx_symbolic_exhaustive_special_entr_cpu_float32 (__main__.TestProxyTensorOpInfoCPU)
Labels: triaged, module: flaky-tests, skipped, module: ProxyTensor

Platforms: asan, linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_make_fx_symbolic_exhaustive_special_entr_cpu_float32&suite=TestProxyTensorOpInfoCPU) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/15889444552).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 2 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_make_fx_symbolic_exhaustive_special_entr_cpu_float32`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_proxy_tensor.py`
DISABLED test_RNN_dropout_state (__main__.TestNN)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on main branch ([recent examples](https://hud.pytorch.org/failure/test_nn.py%3A%3ATestNN%3A%3Atest_RNN_dropout_state))

Specific example: https://github.com/pytorch/pytorch/actions/runs/5859121053/job/15885247751

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
Timeout during NCCL initialization due to store
Labels: oncall: distributed, better-engineering

## Context

Pytorch barrier for the NCCL backend is implemented by calling [allReduce](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/api/colls.html#ncclallreduce) from NCCL. Because we initialize NCCL lazily, before the allreduce we need to set up the NCCL communicators. To initialize communicator, a unique nccl id is broadcasted to all ranks in the group. To do this, we use a key-value store where rank 0 sets the ID, and all other ranks read from this.

## Issue
There is a possibility that some rank is trying to read from the store but no key is set (rank 0 is not setting it). This can happen when rank 0 is a straggler. The default timeout for store is 5 mins which is different that the default nccl watchdog timeout of 30 mins.

the error would look something like:

```
  "RuntimeError: [6] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: Socket Timeout",
```

We should rethink our nccl initialization design and how we want to timeout for initialization

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @kwen2501 @awgu @penguinwu
sdp_kernel causes dynamo error on torch.compile(model, fullgraph=True)
Labels: good first issue, triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

Applying torch.compile(model, fullgraph=True) to a model using sdp_kernel results with a dynamo error.

### Error logs

from user code:
   File ... , line 14, in forward
    with torch.backends.cuda.sdp_kernel(

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information

### Minified repro

```
import torch
import torch._dynamo
import torch._inductor


class Repro(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.enable_flash = True
        self.enable_math = True
        self.enable_mem_efficient = True

    def forward(self, q, k, v):
        with torch.backends.cuda.sdp_kernel(
            enable_flash=self.enable_flash,
            enable_math=self.enable_math,
            enable_mem_efficient=self.enable_mem_efficient,
        ):
            out = torch.nn.functional.scaled_dot_product_attention(
                q,
                k,
                v,
                attn_mask=None,
                dropout_p=0.0,
                is_causal=False,
            )
            return out

rep = Repro()
rep_opt = torch.compile(rep, fullgraph=True)
q = torch.empty_strided(
        (1, 48, 64, 64), (0, 4096, 64, 1), device="cuda", dtype=torch.half
    )
 k = torch.empty_strided(
        (1, 48, 64, 64), (0, 4096, 64, 1), device="cuda", dtype=torch.half
    )
 v = torch.empty_strided(
        (1, 48, 64, 64), (0, 4096, 64, 1), device="cuda", dtype=torch.half
    )
 with torch.no_grad():
     out = rep_opt(q, k, v)


```

### Versions

NA

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @aakhundov
Surface NCCL and CUDA version incompatibility
Labels: oncall: distributed, better-engineering

When there is an incompatibly of the NCCL version with the CUDA version installed on the system, the user sees an error: 

```
torch.distributed.DistBackendError: NCCL error in: fbcode/caffe2/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1190, invalid usage, NCCL version 2.10.3            
ncclInvalidUsage: This usually reflects invalid usage of NCCL library.
```

Ideally we can surface this as a check earlier or provide a better error message.

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @kwen2501 @awgu @penguinwu
Create fastpath backend context manager, similar to SDPA kernel backend manager
Labels: fb-exported, release notes: nn, release notes: distributed (fsdp)

Summary: 

1. create context manager infrastructure with aten-managed context with room for 64 settings
   - as Python builtin for Python eager and torch.compile
   - as TS builtins guarded by torch.hit.is_scripting() for legacy compatibility with TS legacy code. (since "normal" Python builtins that are part of the Pytorch distro are not accessible in TS.)
2. Ability to start with True or False initialization (we always init the state to 0, but will invert some bits based on 
       desired startup/default polarity of a flag)
3. Create fastpath backend context manager using 4 context bits from 1 
       (similar to SDPA kernel backend manager, to give users instant familiarity with the mechanism)
4. Use context manager in FSDP test code (this is prep for a future update that broadens use of fastpath, and would
      otherwise break this FSDP test when the executions are performed using different kernels, showing divergence in
      the test, not thru an error, but the use of different kernels, with different FP rounding characteristics etc)

Test Plan: sandcastle, github

Differential Revision: D48325593
DISABLED test_learnable_forward_per_channel_cpu (quantization.core.test_workflow_ops.TestFakeQuantizeOps)
Labels: triaged, module: macos, skipped

Platforms: macos, mac

This test was disabled because it is failing on main branch ([recent examples](http://torch-ci.com/failure/test_quantization.py%3A%3ATestFakeQuantizeOps%3A%3Atest_learnable_forward_per_channel_cpu)).

cc @malfet @albanD
torch.inverse throws error when DP but not in DDP or single GPU
Labels: needs reproduction, triaged, module: data parallel, module: linear algebra, module: edge cases

### 🐛 Describe the bug

## Description
When I use `torch.inverse` function, it throws an error when I train the model in DataParallel mode, but it will not appear if I use only a single GPU or in DDP mode. I use two RTX4090 on one machine and the OS is Ubuntu 22.04.1 LTS.

## Code
```
proj = torch.matmul(src_proj, torch.inverse(ref_proj))
```

## Error
```
Traceback (most recent call last):
  File "train.py", line 345, in <module>
    train(model, model_loss, optimizer, TrainImgLoader, TestImgLoader, start_epoch, args)
  File "train.py", line 49, in train
    loss, scalar_outputs, image_outputs = train_sample(model, model_loss, optimizer, sample, args)
  File "train.py", line 119, in train_sample
    outputs = model(
  File "/Anaconda3/envs/mymvsnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Anaconda3/envs/mymvsnet/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 168, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/Anaconda3/envs/mymvsnet/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 178, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/Anaconda3/envs/mymvsnet/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/Anaconda3/envs/mymvsnet/lib/python3.8/site-packages/torch/_utils.py", line 457, in reraise
    raise exception
torch._C._LinAlgError: Caught _LinAlgError in replica 1 on device 1.
Original Traceback (most recent call last):
  File "/Anaconda3/envs/mymvsnet/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/Anaconda3/envs/mymvsnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/models/mymvsnet.py", line 101, in forward
    outputs_stage = self.StageNet(
  File "/Anaconda3/envs/mymvsnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/models/mymvsnet.py", line 151, in forward
    warped_src = homo_warping(src_fea, src_proj_new, ref_proj_new, depth_hypo)  # B C D H W
  File "/models/submodules.py", line 133, in homo_warping
    proj = torch.matmul(src_proj, torch.inverse(ref_proj))
torch._C._LinAlgError: torch.linalg.inv: (Batch element 0): The diagonal element 1 is zero, the inversion could not be completed because the input matrix is singular.
```

### Versions

## Pytorch Version
```
conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch
```

cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano
[docs] Document dtype conversions dtype.to_complex() dtype.to_real()
Labels: module: docs, triaged

Edit by @colesbury 

We have helpful conversions between real and complex dtypes:

```
torch.float32.to_complex() -> torch.complex64
torch.complex64.to_real() -> torch.float32
```

But these conversions are not documented. We should document them!

### Original request below:

### 🚀 The feature, motivation and pitch

Builtin helpers would be nice for dtype-polymorphic code of the type:
```python
def f(x : torch.Tensor):
  assert x.is_floating_point()
  return torch.zeros_like(x, dtype = x.dtype.complex()) # or some other bikeshedding
```

Same for conversion of complex dtype to real dtype of the corresponding width


cc @svekars @carljparker
DISABLED test_make_fx_symbolic_exhaustive_special_bessel_y1_cpu_float32 (__main__.TestProxyTensorOpInfoCPU)
Labels: triaged, module: flaky-tests, skipped, module: fx, module: unknown

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_make_fx_symbolic_exhaustive_special_bessel_y1_cpu_float32&suite=TestProxyTensorOpInfoCPU) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/15858837799).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 2 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_make_fx_symbolic_exhaustive_special_bessel_y1_cpu_float32`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_proxy_tensor.py`

ConnectionTimeoutError: Connect timeout for 5000ms, GET https://raw.githubusercontent.com/pytorch/pytorch/main/test/test_proxy_tensor.py -2 (connected: false, keepalive socket: false, socketHandledRequests: 1, socketHandledResponses: 0)
headers: {}

cc @ezyang @SherlockNoMad @EikanWang @jgong5 @wenzhe-nrv
memoryview support for `torch._C.import_ir_module_from_buffer`
Labels: oncall: jit

### 🚀 The feature, motivation and pitch

Currently `torch._C.import_ir_module_from_buffer` require buffer to be `std::string`(corresponding to `str` and `bytes` via pybind11), and immediately convert it to `std::istringstream`.
https://github.com/pytorch/pytorch/blob/b897c57d47f99807528feb022a06e7f5a9d88d00/torch/csrc/jit/python/script_init.cpp#L1887-L1911

However, copy is inevitable to obtain a `bytes`. `memoryview.tobytes()` will copy the data, `io.BytesIO`'s initialization will copy the data(there has been [copy-on-write optimization](https://github.com/python/cpython/issues/66202) since Python 3.5, but only for `bytes` initialization), and `.read()` will copy the data again!

If `torch._C.import_ir_module_from_buffer` supports memoryview, I can do zero-copy `torch.jit.load` after receiving scriptmodule binary through network:
```python
class BufferWrapper:
    def __init__(self, buffer):
        self.buffer = buffer
    
    def read(self):
        return self.buffer

mem: memoryview = network_recv(...)
model = torch.jit.load(BufferWrapper(mem))
```
BTW, the parameter description of `torch.jit.load` seems too strict: `f` only need to support `read`, instead of "has to implement read, readline, tell, and seek".
https://github.com/pytorch/pytorch/blob/b897c57d47f99807528feb022a06e7f5a9d88d00/torch/jit/_serialization.py#L86-L169

### Alternatives

_No response_

### Additional context

There might be a concern that memoryview can be mutable, however, regular files are mutable too.

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Revert "Revert "Add `_foreach_clamp` (#106574)""
Labels: triaged, open source, release notes: foreach_frontend, module: inductor, ciflow/inductor

This reverts commit 354484ea6d85ce2da90fd3e1e5fa9524987bcfe6.


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
Build a check we can defer to runtime, potentially add to the graph
Labels: triaged, oncall: pt2, module: dynamo

See discussion here https://github.com/pytorch/pytorch/pull/106886/files#r1291433908

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @aakhundov
Extend dict  and by extension __dict__ modeling in dynamo to support `setdefault`, `get`
Labels: triaged, oncall: pt2, module: dynamo

ex:

`return module.__dict__.setdefault(REGISTRY_KEY, default_registry)  # type: ignore[call-overload]`

`_module_state_mapping.get(module, None)`

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @aakhundov
Dynamo x FSDP - Issue Tracking Master Task
Labels: triaged, oncall: pt2, module: dynamo

- [ ] https://github.com/pytorch/pytorch/issues/107054

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @aakhundov
UNSTABLE pull / linux-bionic-py3.8-clang9 / test (dynamo)
Labels: module: ci, triaged, oncall: pt2, module: functorch, unstable

Multiple similar failures on trunk recently:
https://hud.pytorch.org/failure/RuntimeError%3A%20functorch%2Ftest_vmap%202%2F2%20failed

```
 functorch/test_vmap 2/2 failed! Received signal: SIGIOT
2023-08-11T05:39:49.5188676Z Traceback (most recent call last):
2023-08-11T05:39:49.5189105Z   File "test/run_test.py", line 1709, in <module>
2023-08-11T05:39:49.5191726Z     main()
2023-08-11T05:39:49.5192146Z   File "test/run_test.py", line 1678, in main
2023-08-11T05:39:49.5195732Z     run_tests(general_tests, test_directory, options, general_failures)
2023-08-11T05:39:49.5196271Z   File "test/run_test.py", line 1585, in run_tests
2023-08-11T05:39:49.5198237Z     raise RuntimeError(failure.message)
2023-08-11T05:39:49.5198751Z RuntimeError: functorch/test_vmap 2/2 failed! Received signal: SIGIOT
2023-08-11T05:39:49.9126288Z 
2023-08-11T05:39:49.9126735Z real	72m46.379s
2023-08-11T05:39:49.9127130Z user	233m22.949s
2023-08-11T05:39:49.9127446Z sys	8m15.861s
2023-08-11T05:39:49.9168650Z ##[error]Process completed with exit code 1.
```

The failure started to appear after #106852, but it's not consistent and looks unrelated to this PR.

cc @seemethere @malfet @pytorch/pytorch-dev-infra @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @zou3519 @Chillee @samdow @kshitij12345 @janeyx99
[RFC] Option to check eager if compile fails
Labels: Stale, module: dynamo, ciflow/inductor

Suppress errors is problematic because it swallows exceptions and keeps going. The question for this RFC is - do want a regime wherein we can catch failures in the compiler, and then quickly check eager to see if it passes or fails?

There's 2 ways we can take this:

1) Replace suppress errors w/ this, and make the code here actually return the eager result - this acts as a fallback mechanism, and disable in one, without routing out to eval_frame.c OR reusing the buggy/failed compilation

OR

2) Agree that an exception is an exception, and just run eager to give the user signal on where the failure is (in eager AND compiled, or just in eager), and then always fail at this point.

I prefer the latter, but wanted to discuss.

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #107044



cc @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Dynamo not handling a NamedTuple 
Labels: high priority, triaged, oncall: pt2, module: dynamo, mlperf

### 🐛 Describe the bug

Context: https://discuss.pytorch.org/t/struggling-to-get-pytorch-fast-enough-to-use-in-public-competition/186015

Tried to poke around the NamedTupleVariable https://gist.github.com/msaroufim/24c0da855b28c8acbea4ddf2d628e977 to get this fixed in dynamo but haven't managed to solve it yet. Opening this while i work on librispeech now instead 

## Repro

Setup instructions here https://github.com/mlcommons/algorithmic-efficiency

```
python3 datasets/dataset_setup.py \
  --data_dir=~/data \
  --ogbg
```

```
python submission_runner.py \
    --framework=pytorch \
    --workload=ogbg \
    --experiment_dir=/home/ubuntu/algorithmic-efficiency \
    --experiment_name=ogbg_pytorch \
    --submission_path=reference_algorithms/development_algorithms/ogbg/ogbg_pytorch/submission.py \
    --tuning_search_space=reference_algorithms/development_algorithms/ogbg/tuning_search_space.json
```

### Error logs

Full logs: https://gist.github.com/msaroufim/36e68408c6cb156cf8cb38baabd17bd3

Error message: `torch._dynamo.exc.InternalTorchDynamoError: GraphsTuple.__new__() missing 6 required positional arguments: 'edges', 'receivers', 'senders', 'globals', 'n_node', and 'n_edge'`

Here's what a `GraphsTuple` is: https://github.com/deepmind/jraph/blob/master/jraph/_src/graph.py#L26


### Minified repro

_No response_

### Versions

n/a

cc @ezyang @gchanan @zou3519 @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @aakhundov
[TEST] Revert "NumPy support in torch.compile  (#106211)"
Labels: Stale, ciflow/binaries_conda, module: inductor, module: dynamo, ciflow/inductor

This reverts commit a9dca53438ca5a3c71d3cbdd4d701b91a61fb7d1.

Windows nightlies are broken, testing to see if revert can fix this: https://github.com/pytorch/pytorch/actions/runs/5830091528/job/15810775269

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @ngimel @anijain2305
Doc for dtensor?
Labels: oncall: distributed, module: dtensor

### 🚀 The feature, motivation and pitch

I noticed that torch dtensor has been released, is there a revised standard documentation for dtensor  related API?😊

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu
a bug about tensor stride
Labels: triaged, module: viewing and reshaping

### 🐛 Describe the bug

>>> a=torch.rand(4,4)
>>> a.T.unsqueeze(0).transpose(-2,-1).contiguous().stride()
(4, 4, 1)
>>> a.T.unsqueeze(0).transpose(-2,-1).contiguous().is_contiguous()
True
>>> a.T.unsqueeze(0).transpose(-2,-1).stride()
(4, 4, 1)
>>> a.T.unsqueeze(0).contiguous().transpose(-2,-1).contiguous().stride()
(16, 4, 1)>>> torch.__version__
'2.0.0a0+gitc263bd4'

the stride of  a.T.unsqueeze(0).transpose(-2,-1).contiguous() should be (16,4,1) instead of (4, 4, 1)

os:ubuntu 18.04

### Versions

'2.0.0a0+gitc263bd4'
[feature request] [onnx] Support QuantLinear/DequantLinear float16 inputs (opset19 and maybe "backport"-support them for opset17)
Labels: module: onnx, triaged, enhancement

### 🚀 The feature, motivation and pitch

They added standardized support for float16 inputs for QuantLinear/DequantLinear: https://github.com/onnx/onnx/blob/main/docs/Changelog.md#QuantizeLinear-19 and https://github.com/onnx/onnx/blob/main/docs/Changelog.md#dequantizelinear-19

When doing a hack to force it to be float16, got `UserWarning: The exported ONNX model failed ONNX shape inference.The model will not be executable by the ONNX Runtime.If this is unintended and you believe there is a bug,please report an issue at https://github.com/pytorch/pytorch/issues.Error reported by strict ONNX shape inference: [ShapeInferenceError] (op_type:QuantizeLinear, node name: /qkv/_input_quantizer/QuantizeLinear): x typestr: T1, has unsupported type: tensor(float16) (Triggered internally at /opt/pytorch/pytorch/torch/csrc/jit/serialization/export.cpp:1410.)`

Related: https://github.com/pytorch/pytorch/issues/102430

### Alternatives

_No response_

### Additional context

_No response_
torchrun： RendezvousConnectionError when use C10d on multi nodes
Labels: oncall: distributed, oncall: r2p

### 🐛 Describe the bug

torchrun \
    --nnodes=1:3\
    --nproc_per_node=4\
    --max_restarts=3\
    --rdzv_id=1\
    --rdzv_backend=c10d\
    --rdzv_endpoint="192.0.0.1:1234"\
    train.py

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
[E socket.cpp:860] [c10d] The client socket has timed out after 60s while trying to connect to 192.0.0.1:1234
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.



cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @dzhulgakov
Memory tracker does not report the module name correctly. 
Labels: Stale, topic: not user facing

If a module contains a sub module followed by some functional method. The name of the functional method is mistakenly labeled to be within the sub module. It should be the parent module. It can be shown in the following example:

```
import torch

from torch.distributed._tools import MemoryTracker

class MyModel(torch.nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.linear = torch.nn.Linear(10, 10)


    def forward(self, input):
        res = self.linear(input)
        res = torch.relu(res)
        res = res.softmax(-1)
        return res


class Parent(torch.nn.Module):
    def __init__(self):
        super(Parent, self).__init__()
        self.child = MyModel()

    def forward(self, input):
        return self.child(input)

def run():
    model = Parent()
    memory_tracker = MemoryTracker()
    memory_tracker.start_monitor(model)
    input = torch.randn(5, 10)
    res = model(input)
    memory_tracker.stop()
    allocated = memory_tracker.memories_allocated
    for i in range(0, len(allocated)):
        print(f"{allocated[i][0]}\t{allocated[i][1]}")


if __name__ == "__main__":
    run()
```

In current master, the output is:
```
.randn.default_0        0.0
child.linear.forward.t.default_0        0.0
child.linear.forward.addmm.default_0    0.0
child.linear.forward.relu.default_0     0.0
child.linear.forward._softmax.default_0 0.0
```

It shows that relu and _softmax are part of the linear. However, they should be part of child. 

With the PR, the output is:

```
.randn.default_0        0.0
child.linear.forward.t.default_0        0.0
child.linear.forward.addmm.default_0    0.0
child.forward.relu.default_0    0.0
child.forward._softmax.default_0        0.0
```

The name is shown correctly.
Create fastpath backend context manager, similar to SDPA kernel backend manager
Labels: fb-exported, Stale, release notes: nn, release notes: distributed (fsdp)

Summary: Create fastpath backend context manager, similar to SDPA kernel backend manager.  This allows us control over reproducibility by giving us control over the backend/kernel that will be selected

PRs like #106824 demonstrate the need for controlling what kernels are used in various situations, this gives users a general solution to the problem.

cc: @drisspg  

Test Plan: sandcastle, github

Differential Revision: D48256968
Add flake8-bugbear code B007
Labels: module: mkldnn, open source, release notes: quantization, release notes: releng, ciflow/mps, module: inductor, module: dynamo, ciflow/inductor

Fixes flake8-bugbear code B007, from #106571


cc @gujinghui @PenghuiCheng @XiaobingSuper @jianyuh @jgong5 @mingfeima @sanchitintel @ashokei @jingxu10 @min-jean-cho @yanbing-j @Guobing-Chen @Xia-Weiwen @voznesenskym @penguinwu @EikanWang @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @anijain2305
max_pool3d_with_indices_backward_cuda does not have a deterministic implementation
Labels: triaged, module: determinism, module: pooling

### 🚀 The feature, motivation and pitch

The operation "max_pool3d_with_indices_backward_cuda" does not have a deterministic implementation in the current context. I would like to have a deterministic implementation so that I can conduct fair algorithm comparison experiments.

### Alternatives

_No response_

### Additional context

The experiment cannot be reproduced.

cc @mruberry @kurtamohler
[WIP][CI Test] Inline through skipfiles
Labels: Stale, ciflow/trunk, topic: not user facing, module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #107009



cc @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @anijain2305 @ipiszy
Apply fusion more aggressively in NAdam and Adagrad compilation
Labels: good first issue, triaged, oncall: pt2, module: inductor

### 🚀 The feature, motivation and pitch

Currently NAdam compiles to > 5 kernels and Adagrad does as well. See https://github.com/pytorch/pytorch/blob/4df84c3b4d93ea53c9ca169f32a0dc4619bee797/test/inductor/test_compiled_optimizers.py#L4

for current kernel counts. Ideally these should fully fuse, but likely there are issues due to the presence of mutation. A good place to start is https://github.com/pytorch/pytorch/blob/4df84c3b4d93ea53c9ca169f32a0dc4619bee797/torch/_inductor/scheduler.py#L4

Specifically https://github.com/pytorch/pytorch/blob/4df84c3b4d93ea53c9ca169f32a0dc4619bee797/torch/_inductor/scheduler.py#L644

Ideally the rules should be able to be modified to soundly allow these to fuse fully. 

Contact @mlazos for more details

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
Dynamic shapes support for inductor foreach codegen
Labels: good first issue, triaged, oncall: pt2, module: dynamic shapes, module: inductor

### 🚀 The feature, motivation and pitch

Currently foreach op compilation doesn't support dynamic shapes. 

See the restriction here: https://github.com/pytorch/pytorch/blob/4df84c3b4d93ea53c9ca169f32a0dc4619bee797/torch/_inductor/lowering.py#L419

Since this feature was originally intended for optimizers, dynamic shapes aren't strictly necessary, but in order for this codegen to be used elsewhere in the stack (think input concatenation, more generalized horizontal fusion) dynamic shapes support is necessary. 

### Alternatives

_No response_

### Additional context

A good place to get started is here: https://github.com/pytorch/pytorch/blob/4df84c3b4d93ea53c9ca169f32a0dc4619bee797/torch/_inductor/codegen/triton_foreach.py#L142

This file is what handles codegen for the foreach operators. At a high level additional runtime args and a more sophisticated kernel launch grid calculation will be needed to take into account runtime sizes vs static sizes. Contact @mlazos for more details

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
tmp test
Labels: Stale, module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #107003
* #107004
* #106912
* #106911



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @ngimel
Only check for fusions within a node distance of 64
Labels: Stale, ciflow/trunk, module: inductor, ciflow/inductor, release notes: inductor

After Animesh's change to help with peak memory, nodes farther than 64 edges in the graph won't be fused anyway, this ensures the scheduler doesn't add those nodes to the list possible fusions. Makes a fusion pass O(N) instead of O(N^2)

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @ngimel
Use aot_compile instead of compile_fx_aot in _export.aot_compile
Labels: triaged, open source, Stale, topic: not user facing, module: export

`torch._inductor` module provides function [aot_compile()](https://github.com/pytorch/pytorch/blob/main/torch/_inductor/__init__.py#L30) which internally uses `_inductor.compile_fx.compile_fx_aot()`.

Since `torch._inductor` module has AOT compilation function `_inductor.aot_compile()` we supposed to use it instead of using module internal function `_inductor.compile_fx.compile_fx_aot()`.

Also, `_inductor.aot_compile()` has a logic to process `compile_fx_aot()` output to get actual lib path.

cc @avikchaudhuri @gmagogsfm @zhxchen17 @tugsbayasgalan
Add cutlass as an alternative backend of PT2 Inductor
Labels: triaged, oncall: pt2, module: inductor

### 🚀 The feature, motivation and pitch

### Motivation
[Cutlass](https://github.com/NVIDIA/cutlass) is an efficient template library for compute-heavy GPU operations like Gemm, Conv and others. It has decent support for H100. Some important kernels (e.g. Flash attention v2, XFormer attention) are implemented based on Cutlass, too. It would be a good complement to Triton to generate Inductor fused kernels for some compute heavy operations.

### Proposal
The proposal is to add Cutlass as an alternative backend of Inductor, as demonstrated in the prototype PR (https://github.com/pytorch/pytorch/pull/106607). As shown in the PR:

1) In torch/_inductor/config.py, "Cutlass" could be configured as one of the Inductor max-autotune backend through `max_autotune_gemm_backends`. (In the future, we may want to extend it to cover other cases beyond gemm.) This option can be set via options in `torch.compile()` (https://pytorch.org/docs/stable/generated/torch.compile.html#torch-compile).

2) Once "Cutlass" is added as a candidate, max-autotune will also tune and select cutlass kernels, together with Aten and Triton kernels.

3) We'll also utilize Cutlass epilogue visitor to support flexible gemm and epilogue fusions in later PRs. More features will come in the future.

### Release / Dependency
1) Pytorch release: to properly release cutlass together with Pytorch, we need to find a way to pack cutlass into Pytorch package distribution. This includes both C++ header files (third_party/cutlass/include) as well as Python scripts and modules (third_party/cutlass/tools/library/scripts).

2) NVCC dependency: The prototype implementation relies on NVCC. We need to figure out ways to switch to NVRTC. Cutlass team has a proposal to use NVCC to do pre-processing at Pytorch packaging stage, and make end-users only rely on NVRTC. This needs to be discussed in details.

3) Meta internal deployment: We need to figure out a way to properly deploy cutlass dependency in Meta internal env.

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @atalman @malfet @alband @ptrblck @seemethere @jansel @gottbrath
cc @jackkosaian @mnicely @hwu36 



### Alternatives

_No response_

### Additional context

_No response_
Reland "Make adding buffers more like adding parameters (#104069)" (take #2)
Labels: ciflow/trunk, release notes: quantization, release notes: nn, topic: new features, ciflow/mps, module: inductor, module: dynamo, ciflow/inductor, module: export

Merged in forward fix from https://github.com/pytorch/pytorch/pull/106783, not sure whether we are relanding in this state but opening for CI

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #106981


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @anijain2305

Differential Revision: [D49443325](https://our.internmc.facebook.com/intern/diff/D49443325)
Sdpa higher order op
Labels: Stale, release notes: fx, module: inductor, module: dynamo, ciflow/inductor

@Chillee I am going to start to address some of the comments in the original PR: #105596

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @ngimel @anijain2305
Found two conflicting CUDA installs
Labels: module: build, triaged

### 🐛 Describe the bug

When I build FBGEMM in our CUDA 12.1 Docker image, it fails with this error message:

```
CMake Error at /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:65 (mess
age):                                                                                                                  
  Found two conflicting CUDA installs:                                                                                 
                                                                                                                       
  V12.1.66 in '/usr/local/cuda/include' and

  V12.1.66 in '/usr/local/cuda-12.1/include'
Call Stack (most recent call first):
  /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:87 (include)
  /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:113 (find_package)
```

Workaround is to run with env var CUDA_PATH=/usr/local/cuda-12.1


### Versions

main

cc @malfet @seemethere
Enable Mypy Checking in torch/_inductor/kernel/conv.py
Labels: triaged, open source, module: inductor, ciflow/inductor

Fixes #105230 

Summary:

As suggested in https://github.com/pytorch/pytorch/issues/105230 mypy checking is enabled in torch/_inductor/kernel/conv.py

After Fix:

mypy --follow-imports=skip torch/_inductor/kernel/conv.py Success: no issues found in 1 source file

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
`tensor.repeat` quirks: has no `torch.` variant, no `out=` variant, no inplace variant | `torch.tile` also does not have `out=` variant and uses `dims=` instead of `dim=`
Labels: triaged, module: viewing and reshaping, module: python frontend

### 🐛 Describe the bug

subj :)

given that https://github.com/pytorch/pytorch/issues/31980 is slow, we are obliged to sometimes use `tensor.repeat` in loops, so this revealed some `tensor.repeat` issues

### Versions

nightly

cc @albanD
Enable Mypy Checking in torch/_inductor/fx_passes/mkldnn_fusion.py
Labels: open source, module: inductor, ciflow/inductor

Fixes #105230 

Summary:

As suggested in https://github.com/pytorch/pytorch/issues/105230 mypy checking is enabled in torch/_inductor/fx_passes/mkldnn_fusion.py

After Fix:

mypy --follow-imports=skip torch/_inductor/fx_passes/mkldnn_fusion.py Success: no issues found in 1 source file

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
Readily available python wheels for windows ARM
Labels: oncall: binaries, module: windows, feature

### 🚀 The feature, motivation and pitch

Currently, [PyPI](https://pypi.org/project/torch/#files) does not support arm64 wheels for windows

### Alternatives

Building from source

### Additional context

_No response_

cc @seemethere @malfet @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite
Enable Mypy Checking in torch/_inductor/freezing.py
Labels: triaged, open source, module: inductor, ciflow/inductor

Fixes #105230 

Summary:

As suggested in https://github.com/pytorch/pytorch/issues/105230 mypy checking is enabled in torch/_inductor/freezing.py

After Fix:

mypy --follow-imports=skip torch/_inductor/freezing.py Success: no issues found in 1 source file


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
Enable Mypy Checking in torch/_inductor/exc.py
Labels: triaged, open source, module: inductor, ciflow/inductor

Fixes #105230 

Summary:

As suggested in https://github.com/pytorch/pytorch/issues/105230 mypy checking is enabled in torch/_inductor/exc.py

After Fix:

mypy --follow-imports=skip torch/_inductor/exc.py Success: no issues found in 1 source file


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
Enable Mypy Checking in torch/_inductor/codegen/triton_foreach.py
Labels: triaged, open source, topic: not user facing, module: inductor, ciflow/inductor

See #105230

Summary:

As suggested in https://github.com/pytorch/pytorch/issues/105230 mypy checking is enabled in torch/_inductor/codegen/triton_foreach.py

After Fix:

mypy --follow-imports=skip torch/_inductor/codegen/triton_foreach.py Success: no issues found in 1 source file


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
Other overloads of `_foreach_clamp`
Labels: module: optimizer, triaged, enhancement, actionable, module: mta

### 🚀 The feature, motivation and pitch

Implement the overloads such as `Tensor`, `TensorList`, and `ScalarList` for `_foreach_clamp`. Currently only `_foreach_clamp(Tensor[], Scalar, Scalar)` is being implemented -- https://github.com/pytorch/pytorch/pull/106574/files#diff-2f3dbd85efb9b5172f2264eedd3be47dd765e6ab7cc8bf3ade5e62c28ae35991R10130

### Alternatives

_No response_

### Additional context

_No response_

cc @vincentqb @jbschlosser @albanD @janeyx99 @mcarilli
[Dynamo x FSDP][5/x] Fix bug in __class__ sources
Labels: Stale, module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #106738
* __->__ #106906
* #106890
* #106888
* #106886
* #106884



cc @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @anijain2305 @ipiszy
Port "Fix kDefaultTimeout multiple definition build failure (#97270)" to release/2.0 branch
Labels: open source, Stale, release notes: distributed (c10d)

#97270 fixed #90448 on the master branch by making the namespace explicit to avoid the constexpr conflict on GCC 11. Unfortunately, the issue persists on release/2.0 causing lost productivity across the Pytorch developer community to manually apply this patch. Let's please include it in the release branch.

@ezyang