gopikrsmscs/torch-issues huggingface dataset:
Issue with torch.distributed.launch
Labels: oncall: distributed

### 🐛 Describe the bug

I am trying to run an image colorization code on my VS code, specifically "https://github.com/piddnad/DDColor/tree/master#-news"

I have to train this model on my custom dataset. For which i have followed the steps provided in the repo, which is mainly to change a few things in the .yml file as i have only one GPU while the repo has code with 4 GPUs being used.

the train.sh on repo is: 

_**CUDA_VISIBLE_DEVICES=0,1,2,3 \
python3 -m torch.distributed.launch --nproc_per_node=4 --master_port=3721 basicsr/train.py \
    -opt options/train/train_ddcolor.yml --auto_resume --launcher pytorch**_

i have changed it to:

_**CUDA_VISIBLE_DEVICES=0 python3 -m torch.distributed.launch --nproc_per_node=1 --master_port=1024 basicsr/train.py -opt options/train/train_ddcolor.yml --auto_resume --launcher pytorch**_

i checked the availability of port 1024 and it is available and idle.

but im still getting two prime errors:

**1. [W socket.cpp:663] [c10d] The client socket has failed to connect to [[WPUCTBNVP3W.jdnet.deere.com](http://wpuctbnvp3w.jdnet.deere.com/)]:1024 (system error: 10049 - The requested address is not valid in its context.).**

**2. train.py: error: unrecognized arguments: --local-rank=0
[2023-10-09 15:51:10,033] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 2) local_rank: 0 (pid: 1844) of binary: C:\Program Files\Python310\python.exe**

im unable to find a solution, i dont want to change the codes given on github repo, i want to follow their steps and train the model but im not aware of how to setup connection with an idle socket and my windows machine.

Most solutions online recommend using **torch.run** in place of torch.distributed.launch but that might require me to change a few scripts in the code. 

Hoping to find some guidance as im not well versed with DDP yet.


### Versions

addict
cython
future
lmdb
numpy
opencv-python
Pillow
pyyaml
requests
scikit-image
scipy
tb-nightly
timm
torch>=1.7
torchvision
tqdm
yapf

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu
tried removing extraneous faketensorprop
Labels: module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #111005
* __->__ #110932



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
Solving pickle error when saving CyclicLR state_dict
Labels: triaged, open source, release notes: optim

## How to reproduce:
```py
import os
import tempfile

import torch
from torch import nn
from torch.optim import SGD
from torch.optim.lr_scheduler import CyclicLR


model = nn.Linear(100, 100)
opt = SGD(model.parameters(), lr=1.)
scheduler = CyclicLR(opt, base_lr=0.1, max_lr=0.2, scale_fn=lambda x: 0.99)

tmp = tempfile.NamedTemporaryFile(delete=False)
try:
    torch.save(scheduler.state_dict(), tmp.name)
    scheduler.load_state_dict(torch.load(tmp.name))
finally:
    tmp.close()
    os.unlink(tmp.name)
```
Error: 
```
_pickle.PicklingError: Can't pickle <function <lambda> at 0x000001A51DF67600>: attribute lookup <lambda> on __main__ failed
```
## Fix:
Saving `scale_fn` to the state dict only if it is a callable object and not if it is a function or lambda.
[dynamo] add config to report guard failure values
Labels: triaged, open source, module: dynamo, ciflow/inductor, release notes: dynamo

Follow up to https://github.com/pytorch/pytorch/pull/110927

Example output:

With `report_guard_failure_values=True`.

With all failures enabled:
```
[2023-10-10 02:04:37,366] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function inner in /home/jonch/Desktop/Programming/mlsys/pytorch/torch/_dynamo/external_utils.py:15', 'triggered by the following guard failures: [\'___guarded_code.valid\', "___check_obj_id(L[\'args\'][1], 140039813517312), with LHS value: {\'id\': 140040114487792}", "___check_obj_id(L[\'args\'][2], 140039813517168), with LHS value: {\'id\': 140040124395392}"]')

```

With default first failure:

```
.[2023-10-10 01:56:07,749] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function fn in /home/jonch/Desktop/Programming/mlsys/pytorch/test/dynamo/test_misc.py:1354', "triggered by the following guard failure: L['a'] == 0, with LHS value: 1")
[2023-10-10 01:56:07,757] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function fn in /home/jonch/Desktop/Programming/mlsys/pytorch/test/dynamo/test_misc.py:1354', "triggered by the following guard failure: tensor '__as_tensor(L['i'])' size mismatch at index 0. expected 1, actual 5")
[2023-10-10 01:56:07,767] torch._dynamo.convert_frame.__recompiles: [DEBUG] ('Recompiling function fn in /home/jonch/Desktop/Programming/mlsys/pytorch/test/dynamo/test_misc.py:1354', "triggered by the following guard failure: ___check_type_id(L['a'], 94852103176992), with LHS value: {'type': <class 'NoneType'>}")
```

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @lezcano @eellison
Expose CompPhoto library to Fbcode for Consumption by IQCloud Service
Labels: fb-exported

Summary:
Bacground:

For Photo processing on IQCloud service on Photo GraphNode, we want to expose CompPhoto SDK code in xplat to be available in fbcode.

I have exported all necessary CompPhoto dependencies to fbcode.

I had to handle conflict with global variable names for
Bool and Complex types between platform X11 and Caffe2 library. The #undef directive is used to make sure the conflict doesn't exist.

I was able to compile and build the following

buck2 build //xplat/compphoto/sdk/pipeline/graphs/stella/stella_background/image:StellaBackgroundImageProcessingGraphFbcode

Test Plan:
buck2 build //xplat/compphoto/sdk/pipeline/graphs/stella/stella_background/image:StellaBackgroundImageProcessingGraphFbcode

File changed: fbsource//xplat/caffe2/c10/core/ScalarType.h
File changed: fbcode//caffe2/c10/core/ScalarType.h
Buck UI: https://www.internalfb.com/buck2/da8ed23b-0735-4325-881c-796bb01b0ce4
Network: Up: 10MiB  Down: 511MiB  (reSessionID-1fda6d02-0545-4135-ba2d-57e5d55b5971)
Jobs completed: 4438. Time elapsed: 8:06.7s.
Cache hits: 1%. Commands: 2189 (cached: 25, remote: 2156, local: 8)
BUILD SUCCEEDED


********************

buck2 build //xplat/compphoto/sdk/pipeline/graphs/stella/stella_background/image:StellaBackgroundImageProcessingGraph
Buck UI: https://www.internalfb.com/buck2/191b4f4a-2bc7-436b-9d3b-d322bad5c6f5
Network: Up: 232KiB  Down: 416KiB  (reSessionID-3fb50098-a64d-4cf2-923f-375953beac6e)
Jobs completed: 12. Time elapsed: 19.9s.
Cache hits: 0%. Commands: 3 (cached: 0, remote: 0, local: 3)
BUILD SUCCEEDED

Differential Revision: D50036045
[inductor][fx pass] Add new split cat pattern detection
Labels: fb-exported, module: inductor, ciflow/inductor

Summary: We add a new pattern to merge getitem_cat to enable further split merges

Test Plan:
### test mcf model
Patch D49972740
```
buck2 run mode/opt //scripts/jackiexu0313/pt2:local_model_with_pt2 -- --test_mode split-only -c
```
split_cat_fx_passes: True
batch_fusion:  False
group_fusion:  False
Running pattern matcher pass: PatternMatcherPass
After pattern matcher pass:  https://www.internalfb.com/intern/everpaste/?color=0&handle=GMXjOBMnJY_5KOEFAMde4eDevKYEbr0LAAAz
Running pattern matcher pass: PatternMatcherPass
merge_getitem_cat pass
After pattern matcher pass:  https://www.internalfb.com/intern/everpaste/?color=0&handle=GAnJ_hKkhC4gQUIBAJvJy1M8EGRkbr0LAAAz
Running pattern matcher pass: PatternMatcherPass
After pattern matcher pass:  https://www.internalfb.com/intern/everpaste/?color=0&handle=GPj0tgSYVtonz3oCAFtxH-GKZWszbr0LAAAz
Running pattern matcher pass: PatternMatcherPass
After pattern matcher pass:  https://www.internalfb.com/intern/everpaste/?color=0&handle=GEAO2gQvoiCWzIAEAAPNkJ6Ns-1bbr0LAAAz
Running pattern matcher pass: PatternMatcherPass
After pattern matcher pass:  https://www.internalfb.com/intern/everpaste/?color=0&handle=GBX8DxArHaki_1sAADm7Vw8N9NY7br0LAAAz
Running pattern matcher pass: PatternMatcherPass
After pattern matcher pass:  https://www.internalfb.com/intern/everpaste/?color=0&handle=GLXE_BZbXITGQKkCALO3xqXvGtV0br0LAAAz
 compare parameters. Numerical result :True
 compare loss/predict. Numerical result :True
 compare param grad. Numerical result :True

P850153017

### unit test
```
buck2 test mode/dev-nosan //caffe2/test/inductor:split_cat_fx_passes -- test_getitem_cat_merge
```
Buck UI: https://www.internalfb.com/buck2/eb7411a5-a6bd-46bc-bf66-756341e3ce10
Test UI: https://www.internalfb.com/intern/testinfra/testrun/13792273864439068
Network: Up: 48KiB  Down: 15KiB  (reSessionID-39ca57cc-5743-423e-b94f-9d0f642010f8)
Jobs completed: 8. Time elapsed: 1:44.7s.
Cache hits: 0%. Commands: 2 (cached: 0, remote: 0, local: 2)
Tests finished: Pass 1. Fail 0. Fatal 0. Skip 0. Build failure 0

### before vs after transformation
https://www.internalfb.com/intern/diffing/?paste_number=847958889

Differential Revision: D50100667




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
The NCCL kernel did not start as expected
Labels: oncall: distributed

### 🐛 Describe the bug

Hello, I apologize for the interruption. I've encountered a puzzling issue. I initiated an asynchronous batch_send_recv communication as per the code below, but when observing on nsys, the kernel launch doesn't entirely align with my expectations. In some loops, both batch_send_recv and matmul are perfectly initiated simultaneously, but in others, the communication is significantly delayed. This unpredictable behavior is not what I intended. Is there a way to ensure that the kernels are launched concurrently?
The nsys graph is here.
<img width="1444" alt="image" src="https://github.com/pytorch/pytorch/assets/29300150/6d5f31dd-14fa-457a-8052-430bc4570e48">

``` python
for idx, t in enumerate(cal_index):
        start = t * micro_size
        end = start + micro_size
        # slice and calculate matmul
        yi = torch.matmul(x[:,start:end,:], weight)
        
        # xi = torch.slice_copy(x, dim=-2, start=start, end=end)
        # yi = torch.matmul(xi, weight)
        # we need to sync and get received
        if idx > 0:
            for req in reqs:
                req.wait()
            yi_send = yi + yi_recv
        else:
            yi_send = yi
        # launch async send and recv
        if idx < mp_ranks-1:
            if mp_rank % 2 == 0:
                send_op = torch.distributed.P2POp(dist.isend, yi_send, (mp_rank + 1) % mp_ranks, group=parallel_state.ALL_SMP_GROUP)
            else:
                yi_recv = torch.empty_like(yi)
                recv_op = torch.distributed.P2POp(dist.irecv, yi_recv, (mp_rank - 1 + mp_ranks) % mp_ranks, group=parallel_state.ALL_SMP_GROUP)

            if mp_rank % 2 == 0:
                yi_recv = torch.empty_like(yi)
                recv_op = torch.distributed.P2POp(dist.irecv, yi_recv, (mp_rank - 1 + mp_ranks) % mp_ranks, group=parallel_state.ALL_SMP_GROUP)
            else:
                send_op = torch.distributed.P2POp(dist.isend, yi_send, (mp_rank + 1) % mp_ranks, group=parallel_state.ALL_SMP_GROUP)
            reqs = torch.distributed.batch_isend_irecv([send_op,recv_op])
```

### Versions

torch  Version :2.1.0a0+4136153

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu
[inductor]: improve support for pow codegen to reciprocal powers of 2
Labels: triaged, open source, topic: not user facing, module: inductor, ciflow/inductor

fixes spda autocast fused attention codegen on my local (i.e. `test_autocast_sdpa_dynamic_shapes`)

Local compiled without flash attention nor mem-efficient attention


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @yanboliang
[inductor] WIP: cumsum template kernel
Labels: open source, module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110911
* #109132
* #106581
* #109601



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
[Inductor] `ConstantFolder` Utility Breaking in Recent Nightly
Labels: triaged, module: inductor

### 🐛 Describe the bug

Encountered the following error when using the Inductor `ConstantFolder` utility on a `GraphModule` with the latest nightly. It appears to arise from a no-longer-existent op referenced by name in the `is_impure` function, relevant section here:
https://github.com/pytorch/pytorch/blob/6c8096ec31df6cb5009dedfe2295099d19f9ea54/torch/_inductor/constant_folding.py#L53-L63

**Error**:
```python
  File "/root/.pyenv/versions/3.10.13/lib/python3.10/site-packages/torch/_ops.py", line 829, in __getattr__
    raise AttributeError(
AttributeError: '_OpNamespace' 'quantized_decomposed' object has no attribute 'dequantize_per_channel'
```

**Code to reproduce**: 
```python
import torch
from torch._inductor.constant_folding import ConstantFolder

gm = torch.fx.symbolic_trace(lambda x: x + 1)
cf = ConstantFolder(gm)
cf.is_impure(list(gm.graph.nodes)[-1]))
```

### Versions

```bash
torch==2.2.0.dev20231009+cu121
```

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
Reland "[C10] PG observability hooks. (#108815)"
Labels: ciflow/trunk, release notes: distributed (c10d)

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #109739
* #110307
* __->__ #110897

Add missing docs for new APIs

This reverts commit ff0358b0384d6a3a5b8ceeae625c93221612ba8e.
[PyTorch] AOTI: add CPU fast path in aoti_torch_empty_strided

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* (to be filled)

This seems to reduce benchmark time by 15-20%. Supersedes D49835545.

Differential Revision: [D49974460](https://our.internmc.facebook.com/intern/diff/D49974460/)
[PyTorch] -DNDEBUG in inductor codecache builds
Labels: module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* (to be filled)

Things like TORCH_INTERNAL_ASSERT_DEBUG_ONLY care about this!

Differential Revision: [D49972742](https://our.internmc.facebook.com/intern/diff/D49972742/)

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
[torch op][xs] verbose error message for type mismatch in toList()
Labels: fb-exported, release notes: mobile

Summary:
Currently error message doesn't give you details on nature of mismatch:
  Output annotation element type and runtime tensor element type must match for tolist()

After update, error becomes actionable:
  RuntimeError: Output annotation element type and runtime tensor element type must match for tolist(): Long vs Int

Test Plan: existing unit tests

Reviewed By: iseeyuan

Differential Revision: D50082858
[dynamo] Add asserts to prevent user defined objects/classes from going into ConstantVariable
Labels: triaged, module: dynamo

### 🐛 Describe the bug

This is the root cause of https://github.com/pytorch/pytorch/pull/110794 but it's a bit hard to educate everyone not to put random crap in ConstantVariable, so it would be good if we have more asserts that help people figure out.

Primary barrier to adding the assert is we'll probably have to fix a pile of latent bugs while we're at it.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @Fidget-Spinner 

### Versions

main
Gradients (Jacobian) in inference
Labels: module: onnx, triaged, onnx-triaged

### 🚀 The feature, motivation and pitch

Neural Nets are often applied in physical or related fields. Here, one usually not only need an output, put also its derivative. With Pytorch one can use Autograd, but I have not found a way to export a Pytorch model to onnx and then get the results from a backwards call. So if there is no way to export any Pytorch gradient calculations, is this planned for the feature? It wouldn't only be a great help for me, I am sure many others would appreciate this functionality!

(If someone found a way to achieve it, please let me know!)

### Alternatives

_No response_

### Additional context

_No response_
BCEWithLogitsLoss: Check if labels / targets are within zero and one
Labels: module: nn, triaged

### 🚀 The feature, motivation and pitch

`BCEWithLogitsLoss` currently performs no kind of checking wrt. the targets whatsoever. This means, that calling the loss function with targets not contained within the interval [0,1] succesfully computes values without throwing an error. However, these values don't have any kind of meaning.

In practice, though, passing invalid targets to `BCEWithLogitsLoss` can easily happen and can go unnoticed! Such mistakes can render whole experiments invalid; **if** they are even noticed in the first place!

For instance, this can happen if multi-class labels and predictions (i.e. 0,1,2,...,n) are erroneously used with BCEWithLogitsLoss. *Shouldn't this result in a shape-mismatch due to the prediction being a categorical distribution?* No, both predictions and targets will be broadcastable to each other in that case. In fact, commonly, their shape will be identical.

### Alternatives

_No response_

### Additional context

Admittedly, performing an additional check will have some performance impact. However, relatively to the rest of the computation, this impact should be neglectable. Furthermore, it would be possible to add a flag to the `BCEWithLogitsLoss` constructor that disables range-checks if desired by the user.

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
Improve Best Practices to Edit and Compile Pytorch Source Code On Windows
Labels: module: windows, module: docs, triaged

### 📚 The doc issue

The current [Best Practices to Edit and Compile Pytorch Source Code On Windows](https://github.com/pytorch/pytorch/wiki/Best-Practices-to-Edit-and-Compile-Pytorch-Source-Code-On-Windows) wiki page does not contain much value and is rather outdated.

### Suggest a potential alternative/fix

One can write down how to modify, build and run PyTorch codebase in different scenairos, both with Conda, and without, on Windows utilitzing tools provided by the poplular IDEs like VSCode.

cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @cristianPanaite @svekars @carljparker
Hash mismatch when installing torch
Labels: oncall: binaries

### 🐛 Describe the bug

Installation:
```
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 --no-cache-dir
```
Tried to follow [this solution](https://stackoverflow.com/questions/40183108/python-packages-hash-not-matching-whilst-installing-using-pip) to solve hash mismatch, therefore add `--no-cache-dir` option.

Response:
```
Looking in indexes: https://download.pytorch.org/whl/cu118
Collecting torch
  Downloading https://download.pytorch.org/whl/cu118/torch-2.1.0%2Bcu118-cp311-cp311-linux_x86_64.whl (2325.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━ 2.1/2.3 GB 3.8 MB/s eta 0:01:04
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    torch from https://download.pytorch.org/whl/cu118/torch-2.1.0%2Bcu118-cp311-cp311-linux_x86_64.whl#sha256=bcb17e2de6ca634d326203694d0bfb552587335e536c1917be3f28c5664b5506:
        Expected sha256 bcb17e2de6ca634d326203694d0bfb552587335e536c1917be3f28c5664b5506
             Got        f1a1a503fb72a82a21fcc46f9feb1b93c189f5744c64bc511963a055342ba644
```

### Versions

torch 2.1.0+cu118
Python 3.11
Ubuntu 22.04 LTS

cc @seemethere @malfet @osalpekar @atalman
AOTAutograd: set_ under no_grad still triggers "a view of a leaf Variable that requires grad is being used in an in-place operation"
Labels: triaged, oncall: pt2, module: aotdispatch

### 🐛 Describe the bug

Extracted from tracing FSDP

```
import torch

@torch.compile(backend="aot_eager", fullgraph=True)
def f(flat_p, p):
    with torch.no_grad():
        p.set_(flat_p[0:2])
    return flat_p

flat_p = torch.randn(4, requires_grad=True)
f(flat_p, flat_p[0:2])
```

fails with

```
  File "/data/users/ezyang/a/pytorch/torch/_functorch/aot_autograd.py", line 3936, in forward
    return compiled_fn(full_args)
  File "/data/users/ezyang/a/pytorch/torch/_functorch/aot_autograd.py", line 1513, in g
    return f(*args)
  File "/data/users/ezyang/a/pytorch/torch/_functorch/aot_autograd.py", line 2626, in runtime_wrapper
    original_inpt.as_strided_(
RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.
```

Fixing this in full generality may be annoying, but a simplified solution that identifies that the set_ is a no-op would suffice for FSDP's case.  To tell if it's a no-op you need to know that flat_p and p alias.

cc @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym 

### Versions

main
AOTAutograd: set_ on input that ultimately no-ops fails in runtime_wrapper copy_
Labels: triaged, bug, oncall: pt2, module: aotdispatch

### 🐛 Describe the bug

This simplified repro is extracted from Voz's FSDP tracing branch:

```
import torch

@torch.compile(backend="aot_eager", fullgraph=True)
def f(p):
    p2 = p.detach()
    p.set_(torch.cat([p, p]))
    x = p * 2
    p.set_(p2)
    return x

f(torch.randn(8))
```

Fails with

```
  File "/data/users/ezyang/a/pytorch/torch/_functorch/aot_autograd.py", line 3936, in forward
    return compiled_fn(full_args)
  File "/data/users/ezyang/a/pytorch/torch/_functorch/aot_autograd.py", line 1513, in g
    return f(*args)
  File "/data/users/ezyang/a/pytorch/torch/_functorch/aot_autograd.py", line 2655, in runtime_wrapper
    original_inpt.copy_(updated_inpt)
RuntimeError: The size of tensor a (8) must match the size of tensor b (16) at non-singleton dimension 0
```

Doesn’t fail on Voz’s branch, because we manually forced AOTAutograd to assume there are no (metadata) mutations on any inputs: https://github.com/pytorch/pytorch/blob/419dcc95a29d8da2232432d5431cacc7024e852d/torch/_functorch/aot_autograd.py#L781 

cc @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym 

### Versions

main
[TP][Inference] Add decompose for matmul op
Labels: ciflow/trunk, module: dtensor, release notes: distributed (dtensor)

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110833
* #110751

When we set the bias to False in the linear, it will trigger `matmul` during the decomposition of  `linear`. So we want to add matmul op in the decomposition as well. Also since we didn't cover this case in unit test, we add a new unit test case so that we have it covered in our UT.
Enable flake8-bugbear B020 lint
Labels: triaged, open source, release notes: quantization, release notes: fx, module: inductor, ciflow/inductor

Fixes part of https://github.com/pytorch/pytorch/issues/106571

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
The derivation of swish activation function is wrong.
Labels: caffe2, triaged

The derivation of swish now in swish_op.cc line 43 is: https://github.com/pytorch/pytorch/blob/12f97bb2e94d6748d3d39d8f8debefde5b6b8dce/caffe2/operators/swish_op.cc#L43
dx = dy * (y + \sigma (x)*(1-y))

But it should be:
dy = dx * (y + \sigma (x)*(1-y))

Besides, in swish.cu line 32 https://github.com/pytorch/pytorch/blob/12f97bb2e94d6748d3d39d8f8debefde5b6b8dce/caffe2/operators/swish_op.cu#L32
it seems as wrong as this.
M2 Failing to build example-app in c++
Labels: needs reproduction, oncall: binaries, module: build, module: cpp, triaged, module: macos

### 🐛 Describe the bug

Following the exact steps in https://pytorch.org/cppdocs/installing.html, I created the following file structure as indicated 
example-app/
  CMakeLists.txt
  example-app.cpp
  
  when I run 
mkdir build
cd build
cmake -DCMAKE_PREFIX_PATH=/absolute/path/to/libtorch ..
cmake --build . --config Release

I get the following linker error 

[ 50%] Building CXX object CMakeFiles/example-app.dir/example-app.cpp.o
[100%] Linking CXX executable example-app
ld: warning: directory not found for option '-L/lib/intel64'
ld: warning: directory not found for option '-L/lib/intel64_win'
ld: warning: directory not found for option '-L/lib/win-x64'
ld: unknown option: --no-as-needed
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [example-app] Error 1
make[1]: *** [CMakeFiles/example-app.dir/all] Error 2
make: *** [all] Error 2


### Versions

Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: macOS 14.1 (arm64)
GCC version: Could not collect
Clang version: 15.0.0 (clang-1500.0.28.1.1)
CMake version: version 3.27.3
Libc version: N/A

Python version: 3.9.6 (default, Aug 12 2023, 13:13:19)  [Clang 15.0.0 (clang-1500.0.37.1) ] (64-bit runtime)
Python platform: macOS-14.1-arm64-arm-64bit
Is CUDA available: N/A
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

CPU:
Apple M2 Pro

Versions of relevant libraries:
[pip3] numpy==1.19.0


cc @seemethere @malfet @osalpekar @atalman @jbschlosser @albanD
[Easy] Eagerly propagate guards for functools partial
Labels: module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110802



cc @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Build process failure with torch_shm_manager
Labels: module: build, triaged

### 🐛 Describe the bug

sudo apt-get install --reinstall libicu-dev  # did not help

neither did pointing /usr/bin/ld  # to Conda compatibility linker


[ 97%] Linking CXX executable ../../../../bin/torch_shm_manager
/usr/bin/ld: /home/developer/mambaforge/envs/FinGPT/lib/libicuuc.so.73: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
collect2: error: ld returned 1 exit status
make[2]: *** [caffe2/torch/lib/libshm/CMakeFiles/torch_shm_manager.dir/build.make:114: bin/torch_shm_manager] Error 1
make[1]: *** [CMakeFiles/Makefile2:5981: caffe2/torch/lib/libshm/CMakeFiles/torch_shm_manager.dir/all] Error 2
make: *** [Makefile:146: all] Error 2


### Versions

Ubunti 20.04, running inside a docker container.

cc @malfet @seemethere
[dynamo] Properly track user-defined types for `type()`
Labels: triaged, open source, topic: not user facing, module: dynamo, ciflow/inductor, module: export

Closes https://github.com/pytorch/pytorch/issues/110315.

Thanks to @ezyang for the easy repro!

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @avikchaudhuri @gmagogsfm @zhxchen17 @tugsbayasgalan @Xia-Weiwen
functools.partial can result in KeyError
Labels: triaged, bug, oncall: pt2, module: dynamo

### 🐛 Describe the bug

Creating an issue so we don't forget about this. Discovered in https://github.com/pytorch/pytorch/pull/109427.

See https://github.com/pytorch/pytorch/pull/110772 for an explanation.

### Error logs

_No response_

### Minified repro

Reproducer in https://github.com/pytorch/pytorch/pull/110772/files#diff-943bb0459b4be9e657b9e1dbedc2f3b6f4479b8aa533e3be575e2d46c5148e74R1327

### Versions

main

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
[Inductor] remove extra buffer creation in realize into
Labels: open source, ciflow/trunk, topic: not user facing, module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110787

**Summary**
Fix the llama performance regression reported in https://github.com/pytorch/pytorch/issues/109874

**Test Plan**
```
python -u -m pytest -s -v test_torchinductor.py -k test_input_mutation5
python -u -m pytest -s -v test_torchinductor.py -k  test_zero_element_mutation
```

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
.lldbinit formatters only work when building with clang
Labels: triaged, release notes: devx, topic: devs

# Summary
I wrote a few formatters for working c++ aten code which can be found here: https://github.com/pytorch/pytorch/blob/c2e7a0d6897bbf75239d238a49197bfd3970142e/tools/lldb/pytorch_lldb.py#L73

trying to run this command from within lldb returns None. I not entirely sure why this doesn't work with GCC but works as expected with Clang... hmmm
[DTensor] Fix DTensor.from_local() returns DTensor with wrong size for uneven sharded tensor
Labels: ciflow/trunk, topic: bug fixes, ciflow/periodic, release notes: distributed (dtensor)

Fixes #110762 

This PR:
1) fixes issue described in #110762 by adding kwarg for shape and stride when creating DTensor using `DTensor.from_local()`. When `shape` and `stride` are provided, we skip calcualtion for `tensor_shape` and `tensor_stride` using `compute_global_tensor_info()`, as `compute_global_tensor_info()` always assume even sharding.
2) updates one use case for `DTensor.from_local()` in fsdp/_shard_utils.py by passing in the `shape` and `stride` of the global tensor. 

cc. @wanchaol
backward and grad behave inconsistently w.r.t. set_ on leaf variable
Labels: module: autograd, triaged, needs research, has workaround

### 🐛 Describe the bug

```

import torch

x = torch.randn(2, requires_grad=True)
y = x.sum()
with torch.no_grad():
    x.set_(torch.randn(2))

y.backward()  # OK
print(x.grad)

x = torch.randn(2, requires_grad=True)
y = x.sum()
with torch.no_grad():
    x.set_(torch.randn(2))

print(torch.autograd.grad(y, (x,)))  # Errors!
```

I would have expected these to both pass or both error, but backward works while grad does not. I print out x.grad to show that the x.grad field is indeed filled in.

cc @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 

### Versions

main
torch.distributed.pipeline skip module throws assert error that portal.grad is not None
Labels: oncall: distributed

### 🐛 Describe the bug

I am exploring `torch.distributed.pipeline` module specifically the skip feature. My standard manual micro batch pipelines implemented using torch distributed rcp with manual micro batching seem to work okay, its when I convert it to `Pipe` with `skip` when I start to see the following error:

```
python3.8/site-packages/torch/distributed/pipeline/sync/skip/portal.py", line 228, in backward
    assert portal.grad is not None
AssertionError
```
I have tried a few toy problems to narrow it down and can confirm that skip works okay with `nn.Linear` layers. In this case I am using it in combination with `nn.Embedding` where I run into this issue. I have tried the same on both torch 2.0 and torch 2.1 and see the same error.

A mock code simulating the error can be found here for reproduction purposes (py3.8): 
https://gist.github.com/suneeta-mall/a7d1aa86732e67470758fa75db134a0b

### Versions

torch.__version__: '2.1.0+cu118'
Python 3.8.10 

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu
Dynamo forward hooks registered before compile region (inlined) w/o tensor ops, don't respect is_compiling control flow
Labels: good first issue, triaged, module: dynamo

### 🐛 Describe the bug
```
class MyModule(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.register_forward_pre_hook(self.pre_forward, with_kwargs=True)

    def pre_forward(self, module, args, kwargs):
        if torch._utils.is_compiling():
            return args, kwargs

        print("path B")
        return args, kwargs

    def forward(self, x):
        return x

mod = MyModule()
m = torch.compile(mod, backend="eager")
m(torch.tensor([1])) # prints "path A"

```
If we force the `is_compiling` region to have something in it, then it works.

```
class MyModule(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.register_forward_pre_hook(self.pre_forward, with_kwargs=True)

    def pre_forward(self, module, args, kwargs):
        if torch._utils.is_compiling():
            global _dummy_compiler_variable
            _dummy_compiler_variable += 1     
            return args, kwargs

 
        logging.warning("I don't want this to be printed")
        return args, kwargs
```

This probably happens because the order is:

1) See hook, start inlining
2) Analysis on hook - no tensor compute or torch ops or anything we care about
3) Delegate to eager



### Versions

main

cc @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
[WIP] Added consistent default values for torch.dropout and related functions.
Labels: open source

Fixes #110356.

This PR makes the functional-style `torch.*` versions of `nn.Dropout` and related functions consistent with their `torch.nn.functional.*` siblings.

The C++ functions in `RNN.cpp` had to be renamed, as they are now ambiguous with the introduction of default values. The impact of this should be minimal, as they are only helper functions used within the file.
Using ChainedScheduler with ReduceLROnPlateau leads to unexpected keyword argument error
Labels: module: optimizer, triaged, needs design, module: LrScheduler

### 🚀 The feature, motivation and pitch

I want to use ReduceLROnPlateau together with another schedule, say a warm-up or cosine schedule.
My understanding is that the suggested way to do that is through ``ChainedScheduler``.
However, when I try to call ``step`` with a validation loss, I get a ``TypeError``, since ``ChainedScheduler`` doesn't accept ``kwargs`` in ``step``.
I'm still pretty new to pytorch, so please let me know if I'm overlooking some other solution.

I don't see a super obvious solution. One would be to pass all ``kwargs`` to all steps in the chain, but that would mean all the schedulers would have to accept arbitrary ``kwargs`` which would hide errors. The other option that comes to mind would be to use inspect to figure out which parameters to pass to which scheduler. Or the ``step`` of ``ChainedScheduler`` could take a dict mapping from schedulers to parameters, but that seems more complicated than just calling all the schedules explicitly and not using ``ChainedScheduler``.

### Alternatives

_No response_

### Additional context

_No response_

cc @vincentqb @jbschlosser @albanD @janeyx99 @crcrpar
Add padding support for dense matrices in (semi-structured sparse @dense) matmul
Labels: module: sparse, triaged

### 🚀 The feature, motivation and pitch

Currently there are shape constraints for both the sparse **and** the dense matrix when doing semi-structured sparse matmul. 

This applies for both cuSPARSELt and CUTLASS. 
For cuSPARSELt the shapes constraints are as follows:
```
rows, cols must be a multiple of:
16 for int8
8 for fp16/bfloat16
4 for fp32
```
for CUTLASS the shape constraints are:
```
rows, cols must be a multiple of:
   (32, 128) for int8
   (32, 64) for fp16/bf16
```

We should add support for padding the dense tensor if we are passed in a tensor that does not satisfy shape constraints. 
One particular use case that would need this is text generation with LLMs, as the matrix muliplications are of the shape (1, hidden_dim, output_dim). 



### Alternatives

_No response_

### Additional context

_No response_

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer
[inductor] decomposition for complex addition
Labels: module: inductor, ciflow/inductor

Tracks https://github.com/pytorch/pytorch/issues/98161 

Complex number support in Pytorch isn't ideal today as complex operations will mostly end up taken care of by the aten runtime, except for `torch.angle` which is handled in [105609](https://github.com/pytorch/pytorch/pull/105609). In general a better way to handle that could be to decompose complex operations first so that more opportunities for fusion could be unveiled, and then to have Triton take care of non-continuous (strided) tensor operations more efficiently. This change adds support to decompose complex addtions.



```
@triton.jit
def triton_(in_ptr0, in_ptr1, out_ptr0, xnumel, XBLOCK : tl.constexpr):
    xnumel = 6
    xoffset = tl.program_id(0) * XBLOCK
    xindex = xoffset + tl.arange(0, XBLOCK)[:]
    xmask = xindex < xnumel
    x0 = xindex
    tmp0 = tl.load(in_ptr0 + (x0), xmask)
    tmp1 = tl.load(in_ptr1 + (x0), xmask)
    tmp2 = tmp0 + tmp1
    tl.store(out_ptr0 + (x0), tmp2, xmask)
```

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
Support using SymBool in arithmetics
Labels: good first issue, triaged, bug, oncall: pt2, module: dynamic shapes

### 🐛 Describe the bug

Currently expressions like  `(x > 1) + (x < 2)`  are not handled by `torch.compile` with dynamic shapes. 

### Error logs

```

RuntimeError: Failed running call_function <built-in function add>(*(s0 > 1, s0 < 3), **{}):
unsupported operand type(s) for +: 'SymBool' and 'SymBool'
(scroll up for backtrace)

The above exception was the direct cause of the following exception:

```

### Minified repro

```python
import torch
compiled = torch.compile(lambda x: (x.shape[0] > 1) + (x.shape[0] < 3), dynamic=True)
compiled(torch.zeros([2]))
```

### Versions

tested on pytorch 2.0.1 and 2.1.0 
<other information irrelevant>

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
[dynamo][guard-refactor] TupleIteratorGetItemAccessor using LambdaAccessor
Labels: module: dynamo

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #110590
* __->__ #110735
* #110589
* #110265
* #108839



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Debug Mode c++ macro
Labels: better-engineering, release notes: cpp

# Summary
Implements a minimal C++ macro that can support the base case of features described here: #99918

And adds to embedding table.


### Perf:
Ran a script to see if adding this path impacts perf: https://gist.github.com/drisspg/d9abae0f7f40f7b924edf9cffff2b12e
Seems like it is below noise levels on my machine
[dynamo]: `assert counter.frame_count == 1` is a bad practice when checking for no graph breaks
Labels: triaged, oncall: pt2, module: dynamo

### Proposed Solution
Prefer `counters['graph_breaks']` or `fullgraph=True`/`nopython=True`

### 🐛 Describe the bug

First identified: https://github.com/pytorch/pytorch/pull/110683/files?diff=split&w=0#r1349139286

Bad uncaught graph breaks:

Graph break happens at beginning of function.
```python

from torch._dynamo.testing import CompileCounter
from torch._dynamo.utils import counters

counters.clear()
counter = CompileCounter()

def f(x):
    torch._dynamo.graph_break()
    x += 1
    return x

compiled_f = torch._dynamo.optimize(counter)(f)

x = torch.tensor([1])
compiled_f(x)

print("frame count", counter.frame_count) # 1
print("Graph breaks", len(counters["graph_break"])) # 1 
assert counter.frame_count - 1 == len(counters["graph_break"]) # fail
```

Graph break occurs at end of function:
```python
from torch._dynamo.testing import CompileCounter
from torch._dynamo.utils import counters

counters.clear()
counter = CompileCounter()

def f(x):
    x += 1
    return torch._dynamo.graph_break()

compiled_f = torch._dynamo.optimize(counter)(f)

x = torch.tensor([1])
compiled_f(x)

print("frame count", counter.frame_count) # 1
print("Graph breaks", len(counters["graph_break"])) # 1
assert counter.frame_count - 1 == len(counters["graph_break"]) # fail
```

Current usage:
1. https://github.com/search?q=repo%3Apytorch%2Fpytorch+assertequal+frame_count&type=code
2. https://github.com/search?q=repo%3Apytorch%2Fpytorch+assert+frame_count&type=code

CC: @voznesenskym @ezyang 

### Versions

main

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Only set grad_fn when mirroring if original was not leaf.
Labels: ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110724

I'm not sure how the old code worked. I am not sure this is right, but it seems more right because things remain non-leaf if they weren't leaf before.

Signed-off-by: Edward Z. Yang <ezyang@meta.com>
Make torch._check work in Dynamo
Labels: triaged, ezyang's list, module: dynamic shapes, module: dynamo

### 🐛 Describe the bug

as stated

### Versions

main

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
fix(dynamo): `Optimizer._init_group` did not handle return value
Labels: triaged, open source, module: dynamo, ciflow/inductor

blocks: https://github.com/pytorch/pytorch/pull/110706

Causes a bug for all optimizers that use _init_group return value. 

`compile` + _init_group ret value is not on testing path. So we also add test.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @lezcano @mlazos
sliding_window attention in scaled_dot_product
Labels: oncall: transformer/mha

### 🚀 The feature, motivation and pitch

Support for sliding window attention (Mistral 7B)

It is not very difficult, just add the setting sliding_window, and add 2 lines to modify the mask.

Here is an example how the mask is computed in OpenNMT-py: (see the two lines in bold)

    def _compute_dec_mask(self, tgt_pad_mask, future):
        tgt_len = tgt_pad_mask.size(-1)
        if not future:  # apply future_mask, result mask in (B, T, T)
            future_mask = torch.ones(
                [tgt_len, tgt_len],
                device=tgt_pad_mask.device,
                dtype=torch.uint8,
            )
            future_mask = future_mask.tril_(0)
            **if self.sliding_window > 0:
                future_mask = future_mask.triu_(-self.sliding_window)**
            future_mask = future_mask.bool()
            future_mask = ~future_mask.view(1, tgt_len, tgt_len)

            dec_mask = torch.gt(tgt_pad_mask + future_mask, 0)
        else:  # only mask padding, result mask in (B, 1, T)
            dec_mask = tgt_pad_mask
        return dec_mask



### Alternatives

if the plan is to implement flash attention 2 in scaled_dot_product, then sliding_window is supported in 2.3

### Additional context

_No response_

cc @jbschlosser @bhosmer @cpuhrsch @erichan1 @drisspg @mikaylagawarecki
More informative variable names in AOTAutograd
Labels: triaged, module: fx, module: aotdispatch

### 🐛 Describe the bug

Two main improvements:

1. It may be that the input GraphModule to AOTAutograd has meaningful names. make_fx should preserve those names
2. When performing backwards, we every gradient should be associated with a name from the original. We should use a name like `grad_original_name` to make clear the association between primal and tangent

### Versions

main

cc @SherlockNoMad @EikanWang @jgong5 @wenzhe-nrv
max_pool3d_with_indices_backward_cuda and avg_pool3d_backward_cuda does not have a deterministic implementation
Labels: feature, module: nn, triaged, module: determinism

### 🚀 The feature, motivation and pitch

The operation max_pool3d_with_indices_backward_cuda and avg_pool3d_backward_cuda does not have a deterministic implementation in the current context. Can this be fixed? So that I can conduct fair comparison experiments? 

### Alternatives

_No response_

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @kurtamohler
[dtensor] reuse dtensor spec as much as possible
Labels: ciflow/trunk, release notes: distributed (dtensor)

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110685
* #110774

Given that we already save tuple of placements instead of list, there's
no risk of modifying the placements inplace, DTensorSpec becomes almost
immutable after construction, we can reuse it when appropriate.

This PR tries to reuse DTensorSpec whenever possible
[WIP][DDP] Use compiled_autograd to trace DDP backward allreduce
Labels: release notes: distributed (c10d), module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110662
* #109647

Use compiled_autograd to trace DDP backward allreduce.

Differential Revision: [D49428482](https://our.internmc.facebook.com/intern/diff/D49428482/)

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
[ROCm] Properly set atol/rtol settings for test_Conv2d_groups tests
Labels: module: rocm, triaged, open source, ciflow/trunk, topic: not user facing, rocm

This test was using default 1e-5 atol even for torch.half and torch.bfloat16 data types, which is impractical for half/bfloat16 hardware like MI200 and Navi 2x/3x.


cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang