gopikrsmscs/torch-issues huggingface dataset:
[doc] torch.scalar_tensor doc is missing
Labels: module: docs, triaged

### 📚 The doc issue

```python
>>> import torch
>>> k = 4
>>> torch.scalar_tensor(k)
tensor(4.)
>>> torch.scalar_tensor.__doc__
>>> 
```

Main branch: `2.1.0a0+git2ac6ee7`

cc @svekars @carljparker
Add support for aten::tril_indices for MPS backend 
Labels: feature, triaged, module: linear algebra, module: mps

### 🐛 Describe the bug

First time contributors are welcome! 🙂
Add support for 'aten::tril_indices' for MPS backend. Generic support for adding operations to MPS backend is captured here: https://github.com/pytorch/pytorch/wiki/MPS-Backend#adding-op-for-mps-backend

### Versions

N/A

cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
[ONNX] Opset 18 support for TorchScript exporter
Labels: module: onnx, triaged

* Modified ONNX Ops
  - [ ] Reduce ops. E.g., https://onnx.ai/onnx/operators/text_diff_ReduceMax_13_18.html#l-onnx-op-reducemax-d13-18
  - [ ] Pad. https://onnx.ai/onnx/operators/text_diff_Pad_13_18.html#l-onnx-op-pad-d13-18
  - [ ] Resize. https://onnx.ai/onnx/operators/text_diff_Resize_13_18.html#l-onnx-op-resize-d13-18
  - [ ] ScatterElements, ScatterND. Native reduction mode https://onnx.ai/onnx/operators/text_diff_ScatterElements_16_18.html#l-onnx-op-scatterelements-d16-18
  - [ ] Split. New attribute `num_outputs`. https://onnx.ai/onnx/operators/text_diff_Split_13_18.html#l-onnx-op-split-d13-18

* New ONNX Ops
  - [ ] CenterCropPad
  - [ ] Col2Im
  - [ ] Mish
  - [ ] BitwiseAnd, BitwiseNot, BitwiseOr, BitwiseXor
  - [ ] GroupNormalization
DISABLED test_inplace_grad_div_floor_rounding_cuda_float64 (__main__.TestBwdGradientsCUDA)
Labels: triaged, module: flaky-tests, skipped, oncall: pt2, module: inductor

Platforms: inductor

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_inplace_grad_div_floor_rounding_cuda_float64&suite=TestBwdGradientsCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/undefined).

Over the past 3 hours, it has been determined flaky in 7 workflow(s) with 7 failures and 7 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_inplace_grad_div_floor_rounding_cuda_float64`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_ops_gradients.py`

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @soumith @desertfire @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
Accuracy repro extraction, constants in graph are not preserved exactly
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

If the repro graph contains a constant (e.g., due to use of torch.tensor), this constant is not serialized faithfully when we extract repros. I plan on fixing it so that if the constant is not too big (<= 4) we'll just print it into the graph, but ideally we would use the same ContentStore mechanism to store these constants exactly.

Comment on this issue if this is affecting you.

### Versions

master

cc @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 @soumith
Arithmetic of single-element Tensors with different dtypes on 'cpu' and 'mps' results in obscure/unhelpful `TypeError`
Labels: triaged, module: mps

### 🐛 Describe the bug

The following snippet raises a TypeError with a rather unhelpful error message
```python
import torch

cpu_tensor = torch.mean(torch.rand(2).double()) # float64 tensor on cpu
mps_tensor = torch.mean(torch.rand(2, device='mps')) # float32 tensor on mps

cpu_tensor + mps_tensor # TypeError: unsupported operand type(s) for +: 'Tensor' and 'Tensor'
```

It would be more sensible to raise either of the usual `RuntimeError`s stating that the operation is impossible because the Tensors are on different devices and/or that the data types are incompatible.

### Versions

```
Collecting environment information...
PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.0 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.102)
CMake version: version 3.25.2
Libc version: N/A

Python version: 3.10.0 (default, Mar  3 2022, 03:54:28) [Clang 12.0.0 ] (64-bit runtime)
Python platform: macOS-13.0-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] pytorch-ident==0.2.5
[pip3] pytorch-lightning==1.9.4
[pip3] torch==2.0.0
[pip3] torchcast==0.3.0
[pip3] torchfilter==0.0
[pip3] torchmetrics==0.11.4
[conda] numpy                     1.23.5          py310hb93e574_0  
[conda] numpy-base                1.23.5          py310haf87e8b_0  
[conda] pytorch-ident             0.2.5                    pypi_0    pypi
[conda] pytorch-lightning         1.9.4              pyhd8ed1ab_0    conda-forge
[conda] torch                     2.0.0                    pypi_0    pypi
[conda] torchcast                 0.3.0                    pypi_0    pypi
[conda] torchfilter               0.0                      pypi_0    pypi
[conda] torchmetrics              0.11.4             pyhd8ed1ab_0    conda-forge
```

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
DISABLED test_wait_i_3 (__main__.TestMultiThreadedWait)
Labels: oncall: distributed, module: flaky-tests, skipped

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_wait_i_3&suite=TestMultiThreadedWait) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/undefined).

Over the past 3 hours, it has been determined flaky in 7 workflow(s) with 14 failures and 7 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_wait_i_3`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `distributed/test_store.py` or `distributed/test_store.py`

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
DISABLED test_wait_i_4 (__main__.TestMultiThreadedWait)
Labels: oncall: distributed, module: flaky-tests, skipped

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_wait_i_4&suite=TestMultiThreadedWait) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/undefined).

Over the past 3 hours, it has been determined flaky in 11 workflow(s) with 22 failures and 11 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_wait_i_4`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `distributed/test_store.py` or `distributed/test_store.py`

ConnectionTimeoutError: Connect timeout for 5000ms, GET https://raw.githubusercontent.com/pytorch/pytorch/master/test/distributed/test_store.py -2 (connected: false, keepalive socket: false, socketHandledRequests: 1, socketHandledResponses: 0)
headers: {}

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Higher GPU consumption for Lenet-5 and LSTM models when compared to other frameworks
Labels: module: cuda, module: memory usage, triaged

### 🐛 Describe the bug

Hello everybody.

I’ve been experimenting with different models and different frameworks, and I’ve noticed that, when using GPU, training Lenet-5 and LSTM models on the IMDB dataset and the MNIST, respectively, leads to higher GPU usage to the Keras and TensorFlow v1.X implementations. Moreover, I’ve also noticed that higher GPU consumption leads to faster training and inference times, and improved accuracy for the Lenet-5 model, while it leads to a slower training time, faster inference time, and consistent accuracy, when compared to the other frameworks

I'm on a Ubuntu 18.04.4 system equipped with an NVIDIA Quadro RTX 4000 GPU with 8GB of VRAM and an Intel(R) Core(TM) i9-9900K CPU running at 3.60GHz.

Here are boxplots that showcase the performance of numerous Lenet5 and LSTM models. Note that the blue boxplots represent training time while the orange ones represent the inference time:


![gpu cons](https://user-images.githubusercontent.com/132307143/235548546-7b9aeb70-1408-4b78-a3ca-fc3f63d8f21e.png)
 
Any ideas on what may be causing this?


### Versions

The version of relevant libraries are:

numpy==1.19.5
torch==1.10.0
torchaudio==0.10.0
torchvision==0.11.1
mkl==2022.2.1
mkl-fft==1.3.0
mkl-random==1.2.1
mkl-service==2.4.0

cc @ngimel
Subgraph rewriter: Unable to match constant args
Labels: triaged, module: dynamo

**Summary:**

I want to match `nn.Conv2d` with arbitrary strides, paddings, and other constant fields. However, when I try to do this the constants are inlined into the graph, and `replace_pattern` complains that these constant args are dead code (i.e. have no users in the graph). For my use case, both the original model and patterns need to be dynamo exported.

**Minimal repro:**
```
import torch
import torch._dynamo as torchdynamo
import torch.nn.functional as F
from torch.fx import replace_pattern

class M(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = torch.nn.Conv2d(1, 1, 1, stride=(1, 1), padding=(1, 1))

    def forward(self, x):
        return self.conv(x)

def match_pattern(x, weight, bias, stride, padding):
    return F.conv2d(x, weight, bias, stride, padding)

def replacement_pattern(x, weight, bias, stride, padding):
    return F.relu(F.conv2d(x, weight, bias, stride, padding))

model_example_inputs = (torch.randn(1, 1, 3, 3),)
pattern_example_inputs = (
    torch.randn(1, 1, 3, 3),  # x
    torch.randn(1, 1, 1, 1),  # weight
    torch.randn(1),           # bias  
    (1, 1),                   # stride
    (0, 0),                   # padding
)

m = M()
m, _ = torchdynamo.export(m, *model_example_inputs, aten_graph=True)
match_pattern , _ = torchdynamo.export(match_pattern, *pattern_example_inputs, aten_graph=True)
replacement_pattern , _ = torchdynamo.export(replacement_pattern, *pattern_example_inputs, aten_graph=True)
replace_pattern(m, match_pattern, replacement_pattern)  # This line fails with the following stacktrace
```

**Outcome:**
```
Traceback (most recent call last):
  File "/home/andrewor/repro.py", line 33, in <module>
    replace_pattern(m, match_pattern, replacement_pattern)
  File "/home/andrewor/local/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/fx/subgraph_rewriter.py", line 195, in replace_pattern
    match_and_replacements = _replace_pattern(gm, pattern, replacement)
  File "/home/andrewor/local/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/fx/subgraph_rewriter.py", line 245, in _replace_pattern
    matcher = SubgraphMatcher(pattern_graph, match_output=False, match_placeholder=False,
  File "/home/andrewor/local/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/fx/passes/utils/matcher_utils.py", line 81, in __init__
    assert len(node.users) > 0, \
AssertionError: SubgraphMatcher cannot be initialized with an pattern with dead code
```

cc @soumith @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
Stop importing HuggingFace transformers in DataClassVariable
Labels: triaged, oncall: pt2, module: dynamo, module: startup time

### 🐛 Describe the bug

There are some HF specific hacks here:

```
class DataClassVariable(ConstDictVariable):
    """             
    This is a bit of a hack to deal with
    transformers.file_utils.ModelOutput() from huggingface.
        
    ModelOutput causes trouble because it a a mix of a dataclass and a
    OrderedDict and it calls super() methods implemented in C.
    """ 
        
    # ModelOutput() excludes None, though generic datclasses don't
    include_none = False
                
    @staticmethod
    @functools.lru_cache(None)
    def _patch_once():
        from transformers.file_utils import ModelOutput
                
        for obj in ModelOutput.__dict__.values():
            if callable(obj):
                skip_code(obj.__code__)
            
    @staticmethod
    def is_matching_cls(cls):
        try:
            from transformers.file_utils import ModelOutput
    
            return issubclass(cls, ModelOutput)
        except ImportError:
            return False

```

Importing transformers takes about 0.5s; it would be much better if we could avoid importing it at all if the model in question doesn't involve transformers.

### Versions

master

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
Import setuptools.command.build_ext from torch.utils.cpp_extension somehow indirectly imports Cython when it is installed
Labels: triaged, oncall: pt2, module: startup time

### 🐛 Describe the bug

I noticed this when inspecting profiles of a simple torch.compile invocation. cpp_extension is pulled in by torch/_inductor/codecache.py. I'm not exactly sure why Cython is automatically getting imported from setuptools. The Cython import is somewhat expensive. A dirty workaround that saves you a half second or so is to uninstall Cython.

### Versions

master

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh
VecISA.__bool__ is very expensive (nearly a second) on startup
Labels: triaged, oncall: pt2, module: cpu inductor, module: startup time

### 🐛 Describe the bug

VecISA currently requires compiling a cpp program to test for capabilities, similar to `./configure`. We do this compilation **on every new process invocation**, dissimilar to `./configure`. After performing the capability query, we should cache it and reuse the cache entry. This will save us about 1s cold start time on CPU inductor. 

### Versions

master

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh
_sfdp_init is extremely expensive for startup time, even on networks that don't benefit from it
Labels: triaged, oncall: pt2, module: startup time

### 🐛 Describe the bug

<img width="1488" alt="image" src="https://user-images.githubusercontent.com/13564/235464215-6867ef63-1cdf-4054-a622-d5da14642e18.png">

In this speedscope profile, compiling a relatively trivial CPU program from cold start, sfdp_init is taking up a whopping three seconds, out of eight seconds end to end compile time.

To reproduce, I was just profiling this script:

```
import torch
import torch._dynamo
torch._inductor.config.cpp.inject_relu_bug_TESTING_ONLY = 'accuracy'
torch._dynamo.config.repro_after = "aot"
torch._dynamo.config.repro_level = 4
torch._dynamo.config.debug_dir_root = "/tmp/tmp5qmh2x01"
@torch.compile()
def inner(x):
    for _ in range(3):
        x = torch.sin(x)
    x = torch.relu(x)
    for _ in range(3):
        x = torch.cos(x)
    return x

inner(torch.randn(20, 20).to("cpu"))
```

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh @jansel 

### Versions

master
[BUG] add 1 to different tensor but get same value
Labels: triaged, module: correctness (silent), module: mps

## Issue description

Generating a list of random tensors using device mps (M2 gpu), then I add a value on each by index, but it feedbacks the same value. Which means `data[1] + 1 = data[2] + 1` but of course `data[1]` is not equal to `data[2]`

So so so weird!!!

## Code example

```
import torch
d = torch.randint(0, 50000, size=(32, ), device=torch.device("mps"))
# d = torch.rand(size=(32, ), device=torch.device("mps"))

print(d)
print(d.dtype)
print("d1", d[1])
print("d1 + 1", d[1] + 1)
print("d2", d[2])
print("d2 + 1", d[2] + 1)
```
the output is 
```
tensor([ 6343,  1605, 28616,  2264, 12151, 13417,  1629, 32512,  4441, 43177,
         6520, 18301,  3608, 35951, 17655,  6629, 21160, 35743, 44574, 16854,
        24114, 26402, 16203, 26529, 45812, 46216, 19513, 30223, 37758, 14296,
         5785, 19335], device='mps:0')
torch.int64
d1 tensor(1605, device='mps:0')
d1 + 1 tensor(6344, device='mps:0')
d2 tensor(28616, device='mps:0')
d2 + 1 tensor(6344, device='mps:0')
```
So when you add 1 on each item in the tensor, it gives back the same value.

other Arithmetic operations are not correct as well
```
import torch
d = torch.randint(0, 50000, size=(32, ), device=torch.device("mps"))
# d = torch.rand(size=(32, ), device=torch.device("mps"))

print(d)
print(d.dtype)
print("d1", d[1])
print("d1 * 2", d[1] * 2)
print("d2", d[2])
print("d2 * 2", d[2] * 2)
```

## System Info

PyTorch version: 1.13.0.dev20220718
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.3 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.12 (main, Apr  5 2022, 01:52:34)  [Clang 12.0.0 ] (64-bit runtime)
Python platform: macOS-13.3-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M2



cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
some of the enteries in the previous version of pytorch section are invalid 
Labels: oncall: binaries, module: docs

### 🐛 Describe the bug

The links used for installing some of the previous versions of pytorch are invalid and fail to install the packages.
1.10.0 and 1.9.0 have these issues.
For example to install 1.10.0+cu113 or 1.10.0+cu111 the following given statements fail:

```
# CUDA 11.3
pip install torch==1.10.0+cu113 torchvision==0.11.0+cu113 torchaudio==0.10.0 -f https://download.pytorch.org/whl/torch_stable.html
```
Or
```
# CUDA 11.1
pip install torch==1.10.0+cu111 torchvision==0.11.0+cu111 torchaudio==0.10.0 -f https://download.pytorch.org/whl/torch_stable.html
```

To fix that the url at the end must be fixed accordingly so the following now work:
 ```
# CUDA 11. 3

pip install torch==1.10.0+cu113 torchvision==0.11.0+cu113 torchaudio==0.10.0 -f https://download.pytorch.org/whl/cu113/torch_stable.html
```

```
# CUDA 11. 3

pip install torch==1.10.0+cu111 torchvision==0.11.0+cu111 torchaudio==0.10.0 -f https://download.pytorch.org/whl/cu111/torch_stable.html
```


### Versions

1.10.0, 1.9.0 

cc @seemethere @malfet @svekars @carljparker
nn.MultiheadAttention doesn't use efficient scaled_dot_product_attention
Labels: module: docs, oncall: transformer/mha

### 📚 The doc issue

The default `forward()` of `nn.MultiheadAttention` has `need_weights==True`. But in this case `scaled_dot_product_attention` does not use one of the efficient fused kernels and silently uses the native implementation. It would be good if this was documented since it took me a while to figure out why I was not seeing the expected performance.

### Suggest a potential alternative/fix

_No response_

cc @svekars @carljparker @jbschlosser @bhosmer @cpuhrsch @erichan1 @drisspg
MPS device inference all same value
Labels: triaged, module: mps

### 🐛 Describe the bug

Hi all,

I have the problem where inference in my model gives different results if I use `mps` device. The inference result are all the same, just for instance. I will share 2 images, they are using **cpu** and  **mps** respectively. (The code is exactly the same, except the device ofc)

<img width="619" alt="torch-cpu" src="https://user-images.githubusercontent.com/3484029/235360481-ee0982b7-75f1-42c1-95b2-7416baa04b3e.png">
<img width="617" alt="torch-mps" src="https://user-images.githubusercontent.com/3484029/235360488-f58e4b97-a1ef-4289-9304-78f1085a3dcb.png">


### Versions

ollecting environment information...
PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.3.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: Could not collect
Libc version: N/A

Python version: 3.11.0 | packaged by conda-forge | (main, Oct 25 2022, 06:21:25) [Clang 14.0.4 ] (64-bit runtime)
Python platform: macOS-13.3.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==2.0.0
[pip3] torchaudio==2.0.1
[pip3] torchvision==0.15.1
[conda] numpy                     1.24   py311hb8f3215_0      conda-forge
[conda] torch                       2.0.0                    pypi_0                    pypi
[conda] torchaudio              2.0.1                    pypi_0                    pypi
[conda] torchvision             0.15.1                   pypi_0                    pypi

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
[inductor] Move loop ordering after fusion
Labels: topic: not user facing, module: inductor, module: dynamo, ciflow/inductor, ciflow/inductor-perf-compare

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #108755
* __->__ #100331
* #108193

cc @soumith @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire
Using ddp training with different machine
Labels: oncall: distributed, module: elastic

### 🐛 Describe the bug

I train the project with different machine
https://github.com/ultralytics/yolov5

machine 1
```
docker run -it --gpus all --rm -v $(pwd):/mnt --network=host nvcr.io/nvidia/pytorch:21.09-py3
python -m torch.distributed.launch --nproc_per_node=2 --nnodes=2 --node_rank=0 --master_addr="192.0.1.1" --master_port=1234 train.py --device 0,1 --workers 1
```

machine 2
```
docker run -it --gpus all --rm -v $(pwd):/mnt --network=host nvcr.io/nvidia/pytorch:21.09-py3
python -m torch.distributed.launch --nproc_per_node=2 --nnodes=2 --node_rank=1 --master_addr="192.0.1.1" --master_port=1234 train.py --device 2,3 --workers 1
```

The machine 2  error then it hang
```
  warnings.warn(
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -7) local_rank: 0 (pid: 449) of binary: /opt/conda/bin/python
```

I try to ping the master address, it is ok.

### Versions

using ngc docker 
FROM pytorch/pytorch:2.0.0-cuda11.7-cudnn8-runtime


If I use the command with the same machine, it will be training successfully.

machine 1  on  docker1 
```
docker run -it --gpus all --rm -v $(pwd):/mnt --network=host nvcr.io/nvidia/pytorch:21.09-py3
python -m torch.distributed.launch --nproc_per_node=2 --nnodes=2 --node_rank=0 --master_addr="192.0.1.1" --master_port=1234 train.py --device 0,1 --workers 1
```

machine 1  on  docker2
```
docker run -it --gpus all --rm -v $(pwd):/mnt --network=host nvcr.io/nvidia/pytorch:21.09-py3
python -m torch.distributed.launch --nproc_per_node=2 --nnodes=2 --node_rank=1 --master_addr="192.0.1.1" --master_port=1234 train.py --device 2,3 --workers 1
```

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @dzhulgakov
graph._export_onnx() incorrect data types in the binary string representation
Labels: module: onnx, triaged

### 🐛 Describe the bug

This bug was aroused in the comments sections of this [post](https://github.com/pytorch/pytorch/issues/100097)

The model converted from torch to onnx is represented by a binary string through the `torch.onnx.export()` function, that function uses a private bounded function [graph._export_onnx](https://github.com/pytorch/pytorch/blob/9e1f46d55b9dfb48effdaf8eb3aa87ba5d072662/torch/onnx/utils.py#L1606) to construct the binary string. It works as expected except for the the data type of the output tensors, it is not accurate, it casts the int32 dtype to int64.

you can find tests [here](https://github.com/pytorch/pytorch/issues/100097)


### Versions


```
pytorch version: 2.0.0+cu118
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: None
onnx version: 1.13.1
```
[Performance] Potential Performance optimization for SDPA
Labels: triaged, module: inductor, module: multi-headed-attention

# Summary
### Eager
The interface for SDPA is (batch, num_heads, seq_len, head_dim) to more closely align with the Attention is All You Need Paper. For most transformers architectures SDPA is called following an input Linear projection and this projection is done on a tensor of shape (batch, seq_len, num_heads, head_dim).

For instance take a look at the CausalSelfAttention module used in the tutorial:
https://pytorch.org/tutorials/intermediate/scaled_dot_product_attention_tutorial.html#causal-self-attention.

This requires the query, key, value's to be transposed prior to calling SDPA. 

Both FlashAttention and MemEfficientAttention accept input q,k,v in the form: (batch, seq_len, num_heads, head_dim).  Within sdpa{flash, mem_eff} we call transpose here: 

For flash the following is the code pointer:
https://github.com/pytorch/pytorch/blob/c4bed869d16e9e0b761b82dda9130cfc14479141/aten/src/ATen/native/transformers/cuda/attention.cu#L715

In eager this manifests itself as two back-to-back view operations. This is not ideal but view ops are generally fairly cheap.

### PT2
Now we go to torch_compile which is great at removing this framework overhead. However, since https://github.com/pytorch/pytorch/blob/c4bed869d16e9e0b761b82dda9130cfc14479141/aten/src/ATen/native/transformers/cuda/attention.cu#L688
is an aten operation that inductor doesn't know how to compile( this is good), inductor will fallback to the aten op. However the transposes inside this ops which are potentially fusable can not be. 

### Question:
We could probably make sdpa_flash composite and just have at::_flash_forward as an aten op. This should in theory allow for the transposes to be compiled. Is there a good way to figure out the potential of this change before doing it? -> There is potential complexity with handling NestedTensors but I think we could avoid the dispatcher and just do it manually.
[discussion] "TensorList" as first-class abstraction (including python frontend) and as key for dispatch for merging `torch._foreach_*` into regular `torch.*` functions
Labels: feature, triaged, module: nestedtensor

### 🚀 The feature, motivation and pitch

There currently exist NestedTensor which accepts a list of tensors but copies them into a new storage

There currently are a bunch of _foreach methods which accept and return tensor lists (e.g. https://github.com/pytorch/pytorch/issues/100248) - these do not force copies into a new storage and often operate inplace. Another usecase: https://github.com/pytorch/pytorch/issues/27522 where copies can be more costly than the operation itself + when copies are taking the precious memory (which is important in academic settings)

I'm proposing to make lists of tensors somehow a first-class abstraction similar to NestedTensor. Maybe it could be done by introducing a new class TensorList or by making existing dispatch to foreach functions if they are fed a python list of tensors

This was discussed in some other places before with @cpuhrsch, but maybe now with proliferation of _foreach methods, it may make sense to absorb them into existing `torch.*` methods and just dispatch properly? As I understand, some TensorList abstraction already exists at the C++ / dispatch level

### Alternatives

_No response_

### Additional context

_No response_

cc @cpuhrsch @jbschlosser @bhosmer @drisspg
Interaction of torch.no_grad and torch.autocast context managers with torch.compile
Labels: triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

Originally discussed at:
https://discuss.pytorch.org/t/torch-compile-interaction-with-autocast-and-no-grad/178648/2?u=vadimkantorov

`Where should I put no_grad and autocast contexts? Inside the function that will be torch.compile’d? Or should I torch.compile first and then wrap the calls with autocast / no_grad? In theory knowing the autocast context and if we don’t need to save intermediate tensors can lead to stonger optimization and more aggressive inplace / economic memory placement`

I think this is an important question as during optimizing for best fp16 perf, e.g. inductor can pre-allocate smaller buffers and use inplace codegen if it knows that it's in no_grad. So the question is coming up naturally. @ezyang @albanD What are your current ideas on interaction of compilation with these important aspects: no_grad-ness and low_precision-ness? And how can we currently hint compilation for MAX brrr (in training) and smaller memory footprints (in inference) :) Also, if we keep using the compiled function in these contexts, it should not trigger a re-compilation every time

@bdhirsh in the original discuss thread:
`Today, if you try to torch.compile a module / function that internally uses autocast/no_grad context managers, dynamo will graph break on them.`

So I’d recommend putting them outside for now:
```python
@torch.compile
def f(args):
    ...

with torch.cuda.amp.autocast():
    out = f(...)

with torch.no_grad():
    out = f(...)
```
`I think we want to fix this though, and avoid graph breaking on these context managers. So longer term the answer is “it shouldn’t matter” - feel free to file an issue though!`

### Versions

N/A

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
Add missing `OpInfo`s for prims ops
Labels: module: tests, triaged, module: primTorch

### 🚀 The feature, motivation and pitch

Some prims ops don't have OpInfos:

- [x] bessel_i0
- [x] bessel_i0e
- [x] bessel_i1
- [x] bessel_i1e
- [x] bessel_j0
- [x] bessel_j1
- [ ] broadcast_in_dim
- [ ] cbrt
- [ ] collapse
- [ ] collapse_view
- [ ] convert_element_type
- [ ] copy_strided
- [ ] copy_to
- [ ] device_put
- [x] empty_strided
- [ ] erf_inv
- [x] erfcx
- [ ] fft_c2c
- [ ] fft_c2r
- [ ] fft_r2c
- [ ] iota
- [x] item
- [ ] maximum_value
- [ ] minimum_value
- [ ] name
- [x] ndtri
- [ ] resize
- [ ] rev
- [ ] shift_left
- [ ] shift_right_arithmetic
- [ ] slice_in_dim
- [x] spherical_bessel_j0
- [ ] split_dim
- [ ] view_of
- [x] zeta

### Alternatives

_No response_

### Additional context

_No response_

cc @mruberry @ezyang @ngimel @Lezcano @peterbell10
tensor with dims marked with torch._dynamo.mark_dynamic loses dynamic dim marks after being moved to a different device
Labels: triaged, oncall: pt2, module: dynamic shapes

### 🐛 Describe the bug

I'm trying to mark some tensor dims as dynamic with `torch._dynamo.mark_dynamic`, and later move it to a target device. If the target device and the device the tensor was on when `torch._dynamo.mark_dynamic` called, `_dynamo_dynamic_indices` is lost on the new tensor.

Repro1 (cpu -> cuda): 
```
import torch._dynamo


t = torch.ones((1, 20))
assert not hasattr(t, "_dynamo_dynamic_indices")

torch._dynamo.mark_dynamic(t, 0)
assert t._dynamo_dynamic_indices == {0}

t_cuda = t.to("cuda")
assert t_cuda._dynamo_dynamic_indices == {0}
```

Repro2 (cuda -> cpu): 
```
import torch._dynamo


t = torch.ones((1, 20)).to("cuda")
assert not hasattr(t, "_dynamo_dynamic_indices")

torch._dynamo.mark_dynamic(t, 0)
assert t._dynamo_dynamic_indices == {0}

t_cpu = t.to("cpu")
assert t_cpu._dynamo_dynamic_indices == {0}
```

Both repros yield the same error log
```
Traceback (most recent call last):
  File "*****", line *, in <module>
    assert t_cuda._dynamo_dynamic_indices == {0}
AttributeError: 'Tensor' object has no attribute '_dynamo_dynamic_indices'
```

### Expected Output:

No Errors

### Versions

torch version: `2.1.0.dev20230427+cu117`

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
Illegal instruction in ARM64 (ver 2.0.0)
Labels: oncall: binaries, triaged, module: arm

### 🐛 Describe the bug

Illegal instruction while importing torch. (version 2.0.0)
AWS Machine: a1.large
AMI: amzn2-ami-kernel-5.10-hvm-2.0.20230418.0-arm64-gp2

python code:
import torch

example:
>>> import torch
Illegal instruction

Note: Downgrading to 1.13.1 solves the issue.

### Versions

Can't run collect_env.py on version 2.0.0 cause I can't import torch.
But, downgrading to 1.13.1 solves the issue, so here are the details:

Collecting environment information...
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Amazon Linux 2 (aarch64)
GCC version: (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15)
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.26

Python version: 3.9.0 (default, Apr 27 2023, 08:22:43)  [GCC 7.3.1 20180712 (Red Hat 7.3.1-15)] (64-bit runtime)
Python platform: Linux-5.10.177-158.645.amzn2.aarch64-aarch64-with-glibc2.26
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        aarch64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              2
On-line CPU(s) list: 0,1
Thread(s) per core:  1
Core(s) per socket:  2
Socket(s):           1
NUMA node(s):        1
Vendor ID:           ARM
Model:               3
Model name:          Cortex-A72
Stepping:            r0p3
BogoMIPS:            166.66
L1d cache:           32K
L1i cache:           48K
L2 cache:            2048K
NUMA node0 CPU(s):   0,1
Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid

Versions of relevant libraries:
[pip3] torch==1.13.1
[conda] Could not collect

cc @seemethere @malfet
DISABLED test_open_device_registration (__main__.TestCppExtensionOpenRgistration)
Labels: module: cpp-extensions, triaged, module: flaky-tests, skipped

Platforms: asan, linux, win, windows, dynamo, mac, macos

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_open_device_registration&suite=TestCppExtensionOpenRgistration) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/undefined).

Over the past 3 hours, it has been determined flaky in 22 workflow(s) with 44 failures and 22 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_open_device_registration`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_cpp_extensions_open_device_registration.py`

cc @malfet @zou3519
This flag not work : torch.backends.cudnn.allow_tf32 = False  
Labels: module: cuda, triaged

### 🐛 Describe the bug

import torch
>>> torch.backends.cuda.matmul.allow_tf32
True
>>> torch.backends.cuda.matmul.allow_tf32=False
>>> torch.backends.cuda.matmul.allow_tf32
True

torch                         1.13.0a0+08820cb



### Versions

torch                         1.13.0a0+08820cb


cc @ngimel
'pip install triton' from pinned hash gives unreliable triton
Labels: module: build, triaged, oncall: pt2

Needs further investigation, but several users have reported that the following comand, which is supposedly fine to use, produces triton builds that sometimes segfault with pytorch+benchmarks.

`pip install "git+https://github.com/openai/triton@7d1a95b04654ff9c216afe08a454ad0822f05370#subdirectory=python"`
(replace 7d1a... with any valid pinned hash)

Instead, the `make triton` command which calls [install_triton_wheel.sh](https://github.com/pytorch/pytorch/blob/main/scripts/install_triton_wheel.sh) installs from our CI's build of triton, and this works reliably.

`pip install --extra-index-url https://download.pytorch.org/whl/nightly/ "pytorch-triton==$(cat .ci/docker/triton_version.txt)+$(head -c 10 .ci/docker/ci_commit_pins/triton.txt)"`

The difference is not the triton hash version as the same pinned hash was used with the working/broken install.

We should figure out if there is a missing or wrong config in the setup script for the regular pip build and make sure it works.

cc @malfet @seemethere @ezyang @soumith @msaroufim @ngimel @bdhirsh
[pt2-functorch] torch.func.functional_call works with func.vmap but breaks for func.grad
Labels: triaged, oncall: pt2, module: functorch

### 🐛 Describe the bug

```python
import torch
from torch import nn
from torchvision.models import resnet18
from torch._dynamo import allow_in_graph
from functools import wraps
from functorch import make_functional_with_buffers, vmap, grad

def traceable(f):
    f = allow_in_graph(f)

    @wraps(f)
    def wrapper(*args, **kwargs):
        return f(*args, **kwargs)

    return wrapper

x = torch.randn(3, 3)

class Net(torch.nn.Module):
    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)

    def forward(self, x):
        return x.sum()

net = Net()

f = lambda param, buffer, x: torch.func.functional_call(net, (param, buffer), x)

p = dict(net.named_parameters())
b = dict(net.named_buffers())

grad(f)(p, b, x)
# RuntimeError: Cannot access data pointer of Tensor that doesn't have storage
torch.compile(traceable(grad(f)))(p, b, x)

vmap(f)(p, b, x)
# Works
torch.compile(traceable(vmap(f)))(p, b, x)
```

But if one uses legacy `make_functional_with_buffers` then both cases work.
```python
import torch
from torch import nn
from torchvision.models import resnet18
from torch._dynamo import allow_in_graph
from functools import wraps
from functorch import make_functional_with_buffers, vmap, grad

def traceable(f):
    f = allow_in_graph(f)

    @wraps(f)
    def wrapper(*args, **kwargs):
        return f(*args, **kwargs)

    return wrapper

x = torch.randn(3, 3)

class Net(torch.nn.Module):
    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)

    def forward(self, x):
        return x.sum()

net = Net()

fnet, p, b = make_functional_with_buffers(net)

f = lambda param, buffer, x: fnet(param, buffer, x)

grad(f)(p, b, x)
torch.compile(traceable(grad(f)))(p, b, x)

vmap(f)(p, b, x)
torch.compile(traceable(vmap(f)))(p, b, x)


```

### Versions

master

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 @zou3519 @Chillee @samdow @janeyx99
Inductor origins still not accurate
Labels: triaged, oncall: pt2, module: inductor

### 🐛 Describe the bug

Given

```
import torch

@torch.compile()
def f(x):
    y = x * 2
    z = torch.cumsum(y, 0)
    return z + 1

f(torch.randn(3, device='cuda'))
```

you end up with

```
def call(args):
    arg0_1, = args
    args.clear()
    with torch.cuda._DeviceGuard(0):
        torch.cuda.set_device(0) # no-op to ensure context
        buf0 = empty_strided((3, ), (1, ), device='cuda', dtype=torch.float32)
        stream0 = get_cuda_stream(0)
        triton_poi_fused_cumsum_mul_0.run(arg0_1, buf0, 3, grid=grid(3), stream=stream0)
        del arg0_1
        buf1 = aten.cumsum(buf0, 0)
        del buf0
        buf2 = buf1
        assert_size_stride(buf2, (3, ), (1, ))
        del buf1
        buf3 = buf2; del buf2  # reuse
        triton_poi_fused_add_1.run(buf3, 3, grid=grid(3), stream=stream0)
        return (buf3, )
```

cumsum should not be attributed to the first kernel, which doesn't actually do cumsum.

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire @Chillee 

### Versions

master
[regression] torch.norm with out dtype bfloat16 cause runtime error
Labels: triaged, module: regression, module: norms and normalization

### 🐛 Describe the bug

As per torch.norm documentation, it can take out tensor. If we set out tensor to be bfloat16 and in input to be float32 then it fails with error:
```
RuntimeError: Expected out tensor to have dtype float, but got c10::BFloat16 instead
```
It used to work for PT 1.13. Use below code to reproduce the issue.

```
torch.norm(torch.tensor([3.]), 2, [0], False, out=torch.tensor([], dtype=torch.bfloat16))
```

### Versions

Name: torch
Version: 2.0.0
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: [packages@pytorch.org](mailto:packages@pytorch.org)
License: BSD-3
Location: /home/jthakur/.pt_2_0/lib/python3.8/site-packages
Requires: filelock, jinja2, networkx, nvidia-cublas-cu11, nvidia-cuda-cupti-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-runtime-cu11, nvidia-cudnn-cu11, nvidia-cufft-cu11, nvidia-curand-cu11, nvidia-cusolver-cu11, nvidia-cusparse-cu11, nvidia-nccl-cu11, nvidia-nvtx-cu11, sympy, triton, typing-extensions
Required-by: torchaudio, torchvision, triton
[Indexing] Incoherent Tensor indexing for nested lists
Labels: triaged, module: numpy, module: advanced indexing, module: edge cases

### 🐛 Describe the bug

The behavior of `pytorch.tensor.__getitem__()` is incoherent when using nested lists:

```python
import torch
t = torch.tensor([1, 2])

t[0].shape  # torch.Size([])
t[[0]].shape  # torch.Size([1])
t[[[0]]].shape  # torch.Size([1]) <--- No dimension added here
t[[[[0]]]].shape  # torch.Size([1, 1]) <--- Subsequent shapes are thus off by 1 dimension
```

That behavior is not observed with tensordict or numpy, where dimensions are added as expected.
```python
import numpy as np
nd = np.array([1, 2])

nd[0].shape  # ()
nd[[0]].shape  # (1,)
nd[[[0]]].shape  # (1, 1)
nd[[[[0]]]].shape  # (1, 1, 1)
```

### Versions

PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.3.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.8 (main, Nov 24 2022, 08:08:27) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-13.3.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==1.24.2
[pip3] torch==2.0.0
[pip3] torchrl==0.1.0+9a85f45
[conda] numpy                     1.24.2                   pypi_0    pypi
[conda] torch                     2.0.0                    pypi_0    pypi
[conda] torchrl                   0.1.0+9a85f45             dev_0    <develop>

cc @mruberry @rgommers @ezyang @gchanan @zou3519
[compile] output does not match eager mode
Labels: high priority, triaged, oncall: pt2, module: functorch, module: pt2 accuracy

### 🐛 Describe the bug

```python
import torch
from torch import nn
from torchvision.models import resnet18
from torch._dynamo import allow_in_graph
from functools import wraps
from functorch import make_functional_with_buffers, vmap, grad

def traceable(f):
    f = allow_in_graph(f)

    @wraps(f)
    def wrapper(*args, **kwargs):
        return f(*args, **kwargs)

    return wrapper

torch.manual_seed(42)
device = 'cpu'  # also fails on CUDA

model = resnet18(pretrained=False, norm_layer=(lambda c: nn.GroupNorm(min(c, 32), c)))
model.to(device)
model.eval()

fnet, params, buffers = make_functional_with_buffers(model)

x = torch.randn(10, 3, 224, 224, device=device)
f = lambda p, b, x : fnet(p, b, x).sum()

# Works for this simpler function
# f = lambda p, b, x: (torch.sin(x) + torch.cos(x) + torch.exp(x)).sum()

f = grad(f)

expected = f(params, buffers, x)
actual = torch.compile(traceable(f))(params, buffers, x)

torch.testing.assert_close(actual, expected)
```

Output
```
AssertionError: Tensor-likes are not close!

Mismatched elements: 9406 / 9408 (100.0%)
Greatest absolute difference: 0.12333643436431885 at index (0, 2, 3, 4) (up to 1e-05 allowed)
Greatest relative difference: 41.227630615234375 at index (5, 0, 0, 6) (up to 1.3e-06 allowed)

```

### Versions

master

cc @ezyang @gchanan @zou3519 @soumith @msaroufim @wconstab @ngimel @bdhirsh @Chillee @samdow @janeyx99
DISABLED test_checkpointing_resets_persistent_refs (__main__.CudaGraphTreeTests)
Labels: triaged, module: flaky-tests, skipped, module: inductor

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_checkpointing_resets_persistent_refs&suite=CudaGraphTreeTests) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/undefined).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 3 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_checkpointing_resets_persistent_refs`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `inductor/test_cudagraph_trees.py` or `inductor/test_cudagraph_trees.py`

cc @soumith @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire