gopikrsmscs/torch-issues huggingface dataset:
Integrate cutlass headers and scripts in pytorch package
Labels: oncall: releng, triaged, topic: binaries, oncall: pt2

### 🐛 Describe the bug

Feature request : https://github.com/pytorch/pytorch/issues/106991
PR: https://github.com/pytorch/pytorch/pull/108015

Require cutlass integration:
Pytorch release: to properly release cutlass together with Pytorch, we need to find a way to pack cutlass into Pytorch package distribution. This includes both C++ header files (third_party/cutlass/include) as well as Python scripts and modules (third_party/cutlass/tools/library/scripts).

OS support - Linux only. Support CUDA 11.8, 12.1.

OSS use case: It's mainly for performance reason. Cutlass is the most efficient open source library on H100 for now (Triton is catching up). The template interface it provides make it easier to do epilogue fusions.
If it's necessary, we could also do some model level benchmark testing, though it would require quite some implementation work.

We could work with NVIDIA to package this into a cutlass python dependency. It also looks like there's some additional Python requirements for these scripts. One option is also always copy-pasting these files into our tree into a subfolder with the correct LICENSE header etc.

Questions:
 - Can we build this as special pypi or conda package so that we don't need to integrate cutlass in each of our binaries ? But depend dynamically on it.  See https://github.com/pytorch/builder/blob/main/conda/pytorch-cuda/meta.yaml for example of integration
 - Need to confirm if we need whole of cutlass/tools/library/, rather then only third_party/cutlass/tools/library/scripts @ipiszy could you please confirm this with a test ?
 - Since this code is property of Nivida do we have a rights to distribute it in this form ? cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @ptrblck 

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @seemethere @malfet @pytorch/pytorch-dev-infra @jansel @ipiszy @albanD @cpuhrsch

### Versions

2.2.0
Pytorch versions without the abi3 flag
Labels: oncall: binaries, triaged, module: python frontend

### 🚀 The feature, motivation and pitch

In some projects I work on, I see not deterministic behavior when I try to import Pytorch with other libraries(for example-tensorrt).
The most famous issue is the imports order, which causes to different behaviors.
I can't share the code with you because of my company policy. But when I tried to figure out what is the problem, I came to conclusion that it might be the abi3 flag which you use(and the other libraries not).

Is there a reason why you don't publish wheel files without this flag?

Can you publish versions without the abi3 flag?
Or explain how I can do that by myself?

Thanks!

### Alternatives

_No response_

### Additional context

_No response_

cc @seemethere @malfet @albanD
Unrecognized attribute: axes for operator ReduceMean during onnx model conversion
Labels: module: onnx, triaged

### 🐛 Describe the bug

I am trying to convert demucs model to onnx , Generally demucs onnx is not supported as of now i have written all preprocessing and postprocessing steps like stft outside model. While converting demucs to onnx i am facing below issue
Using opset version: 18 . Using torchnightly version installed from ```pip3 install --pre torch torchaudio torchvision --index-url https://download.pytorch.org/whl/nightly/cu121```
```
Traceback (most recent call last):
  File "/../lib/python3.9/site-packages/torch/onnx/utils.py", line 1686, in _export
    _C._check_onnx_proto(proto)
RuntimeError: Unrecognized attribute: axes for operator ReduceMean

==> Context: Bad node spec for node. Name: /ReduceMean OpType: ReduceMean

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/../demucs/onnxfile_create_script.py", line 67, in <module>
    torch.onnx.export(model, args=(mix, x), f="htdemucs_ft_onnx.onnx", opset_version=18, do_constant_folding=False)
  File "/home/../lib/python3.9/site-packages/torch/onnx/utils.py", line 516, in export
    _export(
  File "/home/../demucs/lib/python3.9/site-packages/torch/onnx/utils.py", line 1688, in _export
    raise errors.CheckerError(e) from e
torch.onnx.errors.CheckerError: Unrecognized attribute: axes for operator ReduceMean

==> Context: Bad node spec for node. Name: /ReduceMean OpType: ReduceMean
```

### Versions

```
Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] pytorch-triton==2.1.0+e6216047b8
[pip3] torch==2.1.0.dev20230830+cu121
[pip3] torchaudio==2.1.0.dev20230830+cu121
[pip3] torchvision==0.16.0.dev20230830+cu121
[pip3] triton==2.0.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.7.0              hd8887f6_10    conda-forge
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7e14d7c_0    conda-forge
[conda] mkl_fft                   1.3.1            py39h0c7bc48_1    conda-forge
[conda] mkl_random                1.2.2            py39hde0f152_0    conda-forge
[conda] numpy                     1.24.3           py39h14f4228_0  
[conda] numpy-base                1.24.3           py39h31eccc5_0  
[conda] pytorch-mutex             1.0                         cpu    pytorch
[conda] pytorch-triton            2.1.0+e6216047b8          pypi_0    pypi
[conda] torch                     2.1.0.dev20230830+cu121          pypi_0    pypi
[conda] torchaudio                2.1.0.dev20230830+cu121          pypi_0    pypi
[conda] torchvision               0.16.0.dev20230830+cu121          pypi_0    pypi
[conda] triton                    2.0.0                    pypi_0    pypi
```
RuntimeError: NYI: Named tensors are not supported with the tracer
Labels: triaged, open source, topic: not user facing

 #48054 RuntimeError: NYI: Named tensors are not supported with the tracer
 #49538 jit tracer doesn't work with unflatten layer
 #31591 when i try to export a pytorch model to ONNX, got RuntimeError: output of traced region did not have observable data dependence with trace inputs; this probably indicates your program cannot be understood by the tracer.
   - This bug was closed but exists. Multiple comments on it still showing error. This is addressed

Likely fixes the following issues (but untested)

 #63297 Named tensor in tracer
 #2323 [Bug] torch.onnx.errors.UnsupportedOperatorError when convert mask2former to onnx

Fix zero dimensioned tensors when used with jit.trace They are currently assigned an empty set for names {} this is not the same as "no name" so jit.trace bails with
  "NYI: Named tensors are not supported with the tracer"
This happens when I am trying to save a non-trivial model as onnx but the simplest repro I have seen is 48054 above which has been added as test/jit/test_zero_dim_tensor_trace.py

Test plan:
  New unit test added
  Broken scenarios tested locally
  CI

Fixes #48054
@pytorchbot label "topic: not user facing"
DistributedSampler class: Change total_size into num_samples
Labels: oncall: distributed, triaged

https://github.com/pytorch/pytorch/blob/68b518c13e128997a0c7c9ab8ce9508cc4062e3a/torch/utils/data/distributed.py#L118C42-L118C52

This line can be:
 indices = indices[self.rank:self.**num_samples**:self.num_replicas]

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu
[docs] F.interpolate(uint8_input, mode = 'bicubic', ...) overshoot behavior: adjust the note in docs to explain that for uint8 saturating store is done and no manual clamp is needed or mention that bicubic is not supported for uint8 inputs
Labels: module: docs, module: nn, triaged, actionable

### 🐛 Describe the bug

The https://pytorch.org/docs/stable/generated/torch.nn.functional.interpolate.html say that for avoiding overshoot the user must manually do `.clamp(0, 255)`. 
<img width="459" alt="image" src="https://github.com/pytorch/pytorch/assets/1041752/5add9591-f605-4fb4-8275-050518727012">

For me, this makes sense if the input is float32 (or `int32` when it gets supported https://github.com/pytorch/pytorch/issues/5580). But if the input/output is uint8 (which is now finally supported), the overshoot would simply overflow and clamp would not help.

I would propose that for uint8 inputs / outputs F.interpolate should do saturating/clamping cast/store (so that negative values are clamped to 0 and >255 values are clamped to 255)

### Versions

N/A

cc @svekars @carljparker @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
Is there a standard procedure to check the consistency of environment across all nodes in PyTorch DDP training?
Labels: oncall: distributed

### 🚀 The feature, motivation and pitch

In a distributed environment, maintaining consistency across all nodes (e.g., driver versions, NCCL versions) can help reduce some training issues caused by the environment, such as performance or compatibility problems, and make troubleshooting easier. I am wondering if PyTorch provides any standard hooks or other methods to support this kind of check?

In my current setup, I have multiple nodes with different hardware and software configurations, and I want to ensure that the environment is consistent across all nodes before starting the DDP training. Is there any built-in functionality in PyTorch to perform these checks, or do I need to implement a custom solution to compare the environments?

Any suggestions or best practices on how to handle this situation would be greatly appreciated. Thank you in advance for your help!

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu
AdaptiveMaxPool documentation is not detailed
Labels: module: docs, module: nn, triaged, actionable, module: pooling

### 📚 The doc issue

The documentation does not give insights into how the assignments are performed for AdaptiveMaxPool2d (and other AdaptiveMaxPoolXd).
While it is clear what happens in simple cases where the input shape is divisible by the input shape, the behavior when it is not divisible (i.e., where it is actually necessary) is not clarified.
For example, for a 3x3 input and a 2x2 output, it is not obvious that each output is the maximum of 2x2 inputs. (It could also be maxima over 2x2, 2x1, 1x2, and 1x1 input elements.)

My source for the behavior is: https://github.com/pytorch/pytorch/blob/51861cc9b19d9c483598e39932661822a826d3a2/aten/src/ATen/native/AdaptiveMaxPooling2d.cpp 


### Suggest a potential alternative/fix

The following could be added:

For each element of the batch and each channel, given an input shape of $H_{in}\times W_{in}$ and a desired output of shape $H_{out}\times W_{out}$, `AdaptiveMaxPool2d` is defined as follows: 
$$Y[i, j] = \max\left\\{ X[a, b] \ |\  \left\lfloor \frac{i \cdot H_{in}}{H_{out}} \right\rfloor \leq a < \left\lceil \frac{(i+1) \cdot H_{in}}{H_{out}} \right\rceil \ ,\  \left\lfloor \frac{j \cdot W_{in}}{W_{out}} \right\rfloor \leq b < \left\lceil \frac{(j+1) \cdot W_{in}}{W_{out}} \right\rceil   \right\\}$$


cc @svekars @carljparker @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
[inductor] benchmark fusion
Labels: module: inductor, module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #109245
* __->__ #108193



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @anijain2305
[FSDP] incorrect backward prefetch order when using BackwardPrefetch.BACKWARD_POST
Labels: triaged, module: fsdp

### 🐛 Describe the bug

how to reproduce
* pytest test/distributed/fsdp/test_fsdp_backward_prefetch.py
* the unit test wraps nn.Transformer with FSDP

for backward_prefetch = BackwardPrefetch.BACKWARD_POST
* state._exec_order_data.handles_post_forward_order equals forward order: encoder 0...5 -> decoder 0...5 -> root
* post-backward hook (AccumulateGrad) order: decoder 5, 4...0 -> encoder 5...0 -> root
* current prefetch order: decoder 4...0 -> encoder 5...0 -> None -> decoder 5. decoder 5 backward is the 1st forward but it's prefetched at the the last. it means decoder 5 will be block waiting for all-gather without prefetch
* ideal prefetch order: decoder 5...0 -> encoder 5...0 -> None

the default backward_prefetch is BackwardPrefetch.BACKWARD_PRE. if more customers use BACKWARD_POST, we should revisit the priority for fixing

### Versions

[conda] torch                     2.1.0a0+gitd3da6ce           dev_0    <develop>

cc @zhaojuanmao @mrshenli @rohan-varma @awgu @fegin @penguinwu
[Performance] Pass in head_size_og to FlashAttentionV2 
Labels: triaged, module: multi-headed-attention

# Summary
We currently pad the inputs to FLashAttention V2 to the nearest multiple of 8 in order to make sure that inputs are aligned. We are likely doing extra MAD instructions in this case. This change was made in order to make the operator compliant to Autograds aliasing rules. 

A solution would be to pass down the og size from top level sdpa to the flash_api.cpp calls in order to save on some wasted compute when we are padding to a larger multiple.
Enable FlashAttentionV2 on Windows
Labels: triaged, module: multi-headed-attention

# Summary
This PR: https://github.com/pytorch/pytorch/issues/108174 will update the FlashAttention kernel within PyTorch core to V2. Currently this kernel does not support windows. This Issue is used to track support.

See: https://github.com/Dao-AILab/flash-attention/issues/345
FlashAttentionV2 will OOM when building on ci/cd with default settings
Labels: module: cuda, module: ci, triaged

# Summary
Changes were needed to be able to build FlashAttentionV2 on the CI/CD runners that are used when building Cuda with with Arch_list = 8.6.

The default MAX_JOBS will cause the docker container building PyTorch to get killed with exit code 137 from utilizing more than its allocated 16gb of memory. This was resolved by setting MAX_JOBS =2.

## Investigate
This issue is to investigate potential other solutions that maintain build speed.

PR that added this functionality: 
https://github.com/pytorch/pytorch/pull/105602

cc @ptrblck @seemethere @malfet @pytorch/pytorch-dev-infra
TorchInductor Opinfo fixes for rng ops
Labels: ciflow/trunk, topic: not user facing, module: inductor, keep-going

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108170

Tests rng ops both with
- fallback_random=True, assertEqual=True
- fallback_random=False, assertEqual=False


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[inductor] Replace empty_strided with empty in simple cases
Labels: topic: not user facing, module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108169
* #108172
* #108168



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
Automate release only changes in https://github.com/pytorch/pytorch/pull/108053
Labels: oncall: releng, module: ci, triaged


We can use following approach:
Update branch used in composite actions from trunk to release (for example, can be done by running 
``for i in .github/workflows/*.yml; do sed -i -e s#@master#@release/2.0# $i; done``
Example: https://github.com/pytorch/pytorch/commit/51b42d98d696a9a474bc69f9a4c755058809542f

create new script in https://github.com/pytorch/pytorch/tree/main/scripts/release
post_cut_release_branch.sh to generate the above PR automatically


cc @seemethere @malfet @pytorch/pytorch-dev-infra
Problems hit when upgrading the version of HF used in CI
Labels: triaged, oncall: pt2, module: dynamic shapes

@angelayi upgraded the HF version used on CI to overcome a common Export+AOTInductor failure on HF, https://github.com/pytorch/pytorch/pull/107400. However, the upgrading revealed several other issues,

- [ ] test_hf_bert_ddp_inductor failure, `python test/distributed/test_dynamo_distributed.py -k test_hf_bert_ddp_inductor`
- [ ] several benchmarks failures with dynamic shapes ON, also reported in https://github.com/pytorch/pytorch/issues/107200
- [ ] increased graph break count, see https://github.com/pytorch/pytorch/pull/107400/files . Most of the affected benchmarks overlap with those failed at dynamic shapes, so likely caused by the same issue. Performance [drop]( https://hud.pytorch.org/benchmark/compilers?startTime=Tue%2C%2022%20Aug%202023%2015%3A56%3A44%20GMT&stopTime=Tue%2C%2029%20Aug%202023%2015%3A56%3A44%20GMT&granularity=hour&suite=torchbench&mode=training&dtype=amp&lBranch=angelayi/hf_version&lCommit=c08e3fad00a2702ed7a1ee06a3c832d4d00bf74b&rBranch=main&rCommit=138e2895d08a6517c5718b2a0118c1b23ff4664c) because of additional graph breaks.

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
Enable Mypy Checking in torch/_inductor/scheduler.py
Labels: triaged, open source, module: inductor, ciflow/inductor

Fixes #105230 

Summary:

As suggested in https://github.com/pytorch/pytorch/issues/105230 mypy checking is enabled in torch/_inductor/kernel/scheduler.py

After Fix:

mypy --follow-imports=skip torch/_inductor/kernel/scheduler.py Success: no issues found in 1 source file

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
nccl:all_reduce is not profiled correctly
Labels: oncall: distributed, oncall: profiler

### 🐛 Describe the bug

I am reading the source code or PyTorch DDP and using PyTorch profiler to measure the performance of NCCL allreduce operation. I understand the ncclAllReduce is an async call. 
In https://github.com/pytorch/pytorch/blob/6648880aca3914cc7995535fa0813ae580582492/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp#L1710, work is created in collective method(in my case, it's ncclAllReduce), and add a callback to the work at https://github.com/pytorch/pytorch/blob/6648880aca3914cc7995535fa0813ae580582492/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp#L1809. In the callback, it records the function end.
Then at https://github.com/pytorch/pytorch/blob/00eed6f36759323c70c6e05ccb1561367921eee0/torch/csrc/distributed/c10d/reducer.cpp#L1565, the Future waits for the CUDA stream to be completed, then the callback will be triggered and record the end time of nccl:all_reduce. 

Questions:

1. Is my above understanding correct?
2. If I understand the code correctly, why the duration of nccl:all_reduce the profiler captured looks incorrect? E.g., the duration is very short, and it's not the end time of CUDA allreduce kernel.
![image](https://github.com/pytorch/pytorch/assets/97820520/ad7f8f92-ac8d-42ce-9982-c22e536df63e)




### Versions

https://github.com/pytorch/pytorch.git

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @robieta @chaekit @aaronenyeshi @nbcsm @guotuofeng @guyang3532 @gaoteng-git @tiffzhaofb @dzhulgakov @davidberard98
[inductor] Minifier fails on resnet50_quantized_qat
Labels: triaged, oncall: pt2, module: inductor

```
TORCHDYNAMO_REPRO_AFTER=dynamo ./benchmarks/dynamo/torchbench.py --training --performance --no-skip --inductor --only resnet50_quantized_qat
...
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64]] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information

Minifier script written to /data/users/jansel/pytorch/torch_compile_debug/run_2023_08_28_17_07_24_508190-pid_3449892/minifier/minifier_launcher.py. Run this script to find the smallest traced graph which reproduces this error.
```

The generated script contains invalid syntax:
```python
python /data/users/jansel/pytorch/torch_compile_debug/run_2023_08_28_17_07_24_508190-pid_3449892/minifier/minifier_launcher.py
  File "/data/users/jansel/pytorch/torch_compile_debug/run_2023_08_28_17_07_24_508190-pid_3449892/minifier/minifier_launcher.py", line 27
    (activation_post_process): MovingAverageMinMaxObserver(min_val=-5.017982006072998, max_val=4.760481357574463)
                             ^
SyntaxError: invalid syntax
```


```python
class Repro(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.L__mod___activation_post_process_0 = FusedMovingAvgObsFakeQuantize(
  fake_quant_enabled=tensor([1], device='cuda:0'), observer_enabled=tensor([1], device='cuda:0'), scale=tensor([0.0770], device='cuda:0'), zero_point=tensor([65], device='cuda:0', dtype=torch.int32), dtype=torch.quint8, quant_min=0, quant_max=127, qscheme=torch.per_tensor_affine, reduce_range=True
  (activation_post_process): MovingAverageMinMaxObserver(min_val=-5.017982006072998, max_val=4.760481357574463)
)
        self.L__mod___conv1 = ConvBnReLU2d(
  3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False
  (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (weight_fake_quant): FusedMovingAvgObsFakeQuantize(
```



cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[inductor] soft_actor_critic training is slower than eager
Labels: triaged, oncall: pt2, module: inductor

Our [perf dashboard](https://hud.pytorch.org/benchmark/torchbench/inductor_with_cudagraphs?startTime=Mon,%2021%20Aug%202023%2023:38:56%20GMT&stopTime=Mon,%2028%20Aug%202023%2023:38:56%20GMT&granularity=hour&mode=training&dtype=amp&lBranch=main&lCommit=138e2895d08a6517c5718b2a0118c1b23ff4664c&rBranch=main&rCommit=134d415615e39fb4a54420e58a508236a0614610) shows that soft_actor_critic is ~5% slower than eager for AMP training.

Running locally with float32 precision is even worse
```
TORCHINDUCTOR_MAX_AUTOTUNE=1 ./benchmarks/dynamo/torchbench.py --training --performance --no-skip --inductor --only soft_actor_critic
```
with a speedup of `0.82x`.

This is one of the only slowdowns inductor generates.  We should investigate what is going on here and fix it.  Looking at float32 might be easier to debug since the slowdown is more dramatic.




cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
_sampled_addmm_kernel cause 'misaligned address' with new triton pin
Labels: module: sparse, triaged

### 🐛 Describe the bug

This issue tracks the test failure for 
```
python test/test_sparse_csr.py -k test_triton_scaled_dot_product_attention_block_size_16_cuda_bfloat16
```
with new triton commit: 0410652666ddcad54aa4c403276b478a36379b90

Right after calling `_sampled_addmm_kernel` kernel, printing the tensor arguments result in 'misaligend address' error.

Note that test `test_triton_sampled_addmm` passes even though it runs 18 parametrized tests calling `_sampled_addmm_kernel` kernel. So the issue only happens with very specific input to the kernel.

We disable the test to unblock the triton pin upgrade. Use this issue to track it.

### Versions

..

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer
[fx] Show original user stack trace on GraphModule exception if it's available
Labels: fb-exported, release notes: fx

Summary:
Previously we only showed the FX-generated source code, which is not very helpful to end-users.

This PR changes things by showing the original user stack trace if it's available.

We do this by recording a map of lines -> nodes during codegen and using that to figure out which node to show a stack trace for.

This works only if recording stack traces (either you set `record_stack_traces=True`, or are using torch.export/make_fx).

It also doesn't persist across GraphModule pickling, as we have no way of persisting the node references.

In cases where we don't have stack traces available, or we don't have the line -> node mapping, things should degrade gracefully to the previous behavior.

Test Plan: added a unit test

Differential Revision: D48758566
DISABLED test_multilayer_var_cpu (__main__.CpuTests)
Labels: triaged, module: macos, skipped

Platforms: mac, macos

This test was disabled because it is failing on main branch ([recent examples](http://torch-ci.com/failure/inductor%2Ftest_torchinductor.py%3A%3ACpuTests%3A%3Atest_multilayer_var_cpu)).

Here is an example https://hud.pytorch.org/pytorch/pytorch/commit/60bb02a907aa1f5e6f2150d50ea28a48afaa8e95

cc @malfet @albanD @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
[inductor] minifier fails on moco
Labels: triaged, oncall: pt2, module: dynamic shapes, module: inductor

Run without minifier, the failure seems to be tensor being assigned to symint compile error:
```python
CUDA_LAUNCH_BLOCKING=1 ./benchmarks/dynamo/torchbench.py --inference --performance --no-skip --inductor --freezing --only moco
...
torch._dynamo.exc.BackendCompilerFailed: backend='compile_fn' raised:
TypeError: can't assign a SymInt to a torch.cuda.LongTensor

While executing %setitem_1 : [num_users=0] = call_function[target=operator.setitem](args = (%l__self___queue_ptr, 0, %mod), kwargs = {})
Original traceback:
  File "/data/users/jansel/torchbenchmark/torchbenchmark/models/moco/moco/builder.py", line 66, in <resume in _dequeue_and_enqueue>
    self.queue_ptr[0] = ptr
```


If I enable the minifier, the error somehow changes to an IMA at runtime and the compile error goes away:
```python
TORCHDYNAMO_REPRO_AFTER=dynamo CUDA_LAUNCH_BLOCKING=1 ./benchmarks/dynamo/torchbench.py --inference --performance --no-skip --inductor --freezing --only moco
...
  File "/tmp/torchinductor_jansel/4j/c4jliihk5stxxkbqkwmftdi3lzst3kortckftv6e3hikrpuh5wyo.py", line 665, in call
    triton_poi_fused_convolution_0.run(arg162_1, buf0, 96, 50176, grid=grid(96, 50176), stream=stream0)
  File "/data/users/jansel/pytorch/torch/_inductor/triton_heuristics.py", line 426, in run
    return launcher(
  File "<string>", line 6, in launcher
torch._dynamo.exc.BackendCompilerFailed: backend='compile_fn' raised:
RuntimeError: Triton Error [CUDA]: an illegal memory access was encountered

While executing %submod_0 : [num_users=2] = call_module[target=submod_0](args = (%l_im_q_,), kwargs = {})
Original traceback:
None

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information

Minifier script written to /data/users/jansel/pytorch/torch_compile_debug/run_2023_08_28_13_50_54_266476-pid_245453/minifier/minifier_launcher.py. Run this script to find the smallest traced graph which reproduces this error.
```


If I run the generated script, the failure can't be reproed:
```python
python /data/users/jansel/pytorch/torch_compile_debug/run_2023_08_28_13_50_54_266476-pid_245453/minifier/minifier_launcher.py
...
RuntimeError: Input graph did not fail the tester
```

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[Optimizer Perf] Improve speed of _init_group to c++
Labels: module: performance, module: optimizer, triaged, needs research

### 🚀 The feature, motivation and pitch

It has been noticed on many models that[ _init_group](https://github.com/pytorch/pytorch/blob/main/torch/optim/adamw.py#L90-L146) takes 33% of the whole `optimizer.step()` on **every step**  for popular optimizers like adam, adamw, etc. since it has to loop through each single param for each step

## Proposal, do this in c++

just like how `_group_tensors_by_device_and_dtype` in https://github.com/pytorch/pytorch/pull/100007 was moved in c++, lets move _init_group there too

cc: @janeyx99 @crcrpar

cc: @clchan @ezhang887

### Alternatives

_No response_

### Additional context

_No response_

cc @vincentqb @jbschlosser @albanD @janeyx99 @crcrpar
Aliased Input/Output Requirement in `aot_export_joint_simple`
Labels: triaged, oncall: pt2, module: aotdispatch

### 🐛 Describe the bug

When compiling a simple module with the function [`aot_export_joint_simple`](https://github.com/pytorch/pytorch/blob/054f3f1d8f9eb63ef8437991eba5b8f2aeee920f/torch/_functorch/aot_autograd.py#L4084), the following error is encountered:
```python
Traceback (most recent call last):
  File "~/test.py", line 16, in <module>
    gm = aot_export_joint_simple(Aliased(), [torch.rand((5, 5))], trace_joint=False)
  File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/aot_autograd.py", line 4134, in aot_export_joint_simple
    raise RuntimeError(f"aot_export_joint_simple does not support outputs that alias inputs. {str(metadata)}")
RuntimeError: aot_export_joint_simple does not support outputs that alias inputs. ViewAndMutationMeta(input_info=[InputAliasInfo(is_leaf=True, mutates_data=False, mutates_metadata=False)], output_info=[OutputAliasInfo(output_type=<OutputType.alias_of_input: 2>, raw_type=<class 'torch.Tensor'>, base_idx=0, dynamic_dims=set())], requires_grad_info=[False], num_intermediate_bases=0, keep_input_mutations=False, traced_tangents=[], num_symints_saved_for_bw=None)
```
The model can be found below:
```python
import torch
from torch._functorch.aot_autograd import aot_export_joint_simple

class Aliased(torch.nn.Module):
    def forward(self, x):
        return (x[None],)

gm = aot_export_joint_simple(Aliased(), [torch.rand((5, 5))], trace_joint=False)
```

If we comment out these lines:
https://github.com/pytorch/pytorch/blob/054f3f1d8f9eb63ef8437991eba5b8f2aeee920f/torch/_functorch/aot_autograd.py#L4133-L4134
Then, the module compiles and returns a graph in which the inputs and outputs are valid and no longer aliased, as so:
```python
graph():
    %arg0_1 : [num_users=1] = placeholder[target=arg0_1]
    %unsqueeze : [num_users=1] = call_function[target=torch.ops.aten.unsqueeze.default](args = (%arg0_1, 0), kwargs = {})
    return (unsqueeze,)
```
Would it be possible to have the input/output alias check in this function be controlled by a flag, since it does not seem required for the output graph to be valid?

Additionally, are there any suggestions for utilities to de-alias inputs from outputs?

### Versions

PyTorch Version: `2.1.0.dev20230825+cu121`

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
testing out_dtype_int_mm
Labels: release notes: fx, module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108059

Summary:

Test Plan:
python /home/cdhernandez/local/pytorch/test/test_out_dtype_op.py -k
"int_mm" &> log.log
Reviewers:

Subscribers:

Tasks:

Tags:

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[POC] Add `frozen_offload`

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108052
* #107784
* #106080
* #106068

This is a proof-of-concept for an internal use case request. `frozen_offload` only applies to a module with all frozen parameters. In default, the parameters are offloaded to CPU (as views into a single contiguous tensor for copy efficiency), and for forward computation, the parameters are moved to GPU.

![Screenshot 2023-08-28 at 10 10 01 AM](https://github.com/pytorch/pytorch/assets/31054793/866d1245-f1a2-44c9-ad41-03759acc854c)
Enhanced Available Backend Discovery and Selection in PyTorch 2
Labels: triaged, enhancement, module: python frontend

### 🚀 The feature, motivation and pitch

New utility functions are proposed to make backend selection more intuitive and efficient in PyTorch 2:

Avoid this type of constructs:
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
DEVICE = torch.device("cuda" if torch.cuda.is_available() else ("mps" if torch.has_mps else "cpu"))

By introducing for example:
torch.backends.get_available() -> List[str]: Returns a sorted list of available backends based on a set of predefined compatibility and performance metrics.

torch.backends.select_optimal() -> str: Automatically selects the most compatible and efficient backend for the current system configuration. (The first one from the sorted get_available() list.)


The feature aims to categorize and sort available computational backends, such as CUDA, MPS, and CPU, based on their computational power or other suitable criteria.

### Alternatives

_No response_

### Additional context

_No response_

cc @albanD
Undefined Symobl: pybind11::detail::type_caster<at::Tensor, void>::load(pybind11::handle, bool)
Labels: needs reproduction, triaged, module: pybind

### 🐛 Describe the bug

This bug report is a derivation of what was reported on the pybind/pybind11 GitHub repo:
[pybind issue #4696](https://github.com/pybind/pybind11/issues/4696)

The traceback I got was:
Traceback (most recent call last):
File "run.py", line 25, in 
import cfvpy.tasks
File "/home/usrname/rebel-main/cfvpy/tasks.py", line 15, in 
import cfvpy.selfplay
File "/home/usrname/rebel-main/cfvpy/selfplay.py", line 27, in 
import cfvpy.rela
ImportError: /home/usrname/rebel-main/cfvpy/rela.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN8pybind116detail11type_casterIN2at6TensorEvE4loadENS_6handleEb

To reproduce, please follow the steps as outlined by me in a comment of the pybind issue above:
[reproduction procedure](https://github.com/pybind/pybind11/issues/4696#issuecomment-1678574353)

And one of pybind's contributors, rwgk, suggested that this might be a bug/issue of PyTorch,
[rwgk's comment](https://github.com/pybind/pybind11/issues/4696#issuecomment-1679013574)



### Versions

PyTorch: 1.4.1
pybind: [a1b71df](https://github.com/pybind/pybind11/tree/a1b71df137e015d44f7e31f7b6d4807253fb7871)
Script to compare measured (trace) runtimes with estimated runtimes
Labels: fb-exported, topic: not user facing, module: inductor, ciflow/inductor

Reviewed By: xuzhao9

Differential Revision: D48523883



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
DISABLED test_autocast_flash_attention (__main__.ActivationCheckpointingViaTagsTests)
Labels: module: rocm, triaged, module: flaky-tests, skipped, module: dynamo

Platforms: rocm

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/failure/test_autocast_flash_attention) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/16234790976).

Over the past 72 hours, it has flakily failed in 9 workflow(s).

**Debugging instructions (after clicking on the recent samples link):**
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Grep for `test_autocast_flash_attention`


Test file path: `dynamo/test_activation_checkpointing.py`

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov
[FSDP] Ignored modules on meta device seem to be initialized on CUDA device
Labels: oncall: distributed, triaged, actionable, module: fsdp

### 🐛 Describe the bug

When passing in a module as `ignored_modules`, should we also ensure FSDP does not initialize it via `to_empty` + `reset_parameters`? If `ignored_modules` contract is that FSDP ignores the module entirely, this might be what's expected especially if users want to apply different initialization / sharding to the ignored modules.

One way this could cause issues in practice is if I have a very large embedding on a meta device, wrap with FSDP and ignore it, and want to initialize + shard the embedding separately after FSDP wrap. In this case, seems like the full Embedding would be allocated by FSDP.

The following test should reproduce

```
@skip_if_lt_x_gpu(2)
    def test_fsdp_ignored_module_meta(self):
        class CPUGPUModule(nn.Module):
            def __init__(self):
                super().__init__()
                self.a = nn.Linear(1, 1)
                self.b = nn.Linear(1, 1)

        with torch.device("meta"):
            m= CPUGPUModule()
        m = FSDP(m, device_id=self.rank, ignored_modules=[m.a], use_orig_params=True)
        print(f"RV: {next(m.a.parameters()).device}")
```

### Versions

main

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin
Intra-graph communication reordering pass on Inductor scheduler IR (based on #100762)
Labels: fb-exported, module: inductor, ciflow/inductor

Differential Revision: D47657156



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @chenyang78 @kadeng @muchulee8 @aakhundov
`upsample_bilinear2d_backward_out_cuda` is nondeterministic
Labels: triaged, module: determinism

### 🐛 Describe the bug

torch.use_deterministic_algorithms(True)
nn.Upsample(scale_factor=scale_factor, mode='bilinear')

RuntimeError: upsample_bilinear2d_backward_out_cuda does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application.

### Versions

nn.Upsample(scale_factor=scale_factor, mode='bilinear')

cc @mruberry @kurtamohler
fix: adam(w) ignore stride mismatch when dim is size 1
Labels: module: optimizer, triaged, open source, release notes: foreach_frontend

Fixes #106951

when dim is of size 1, stride mismatch does not matter

As discussed in #78050 strides are inconsistent and may sometimes change when dim 1 during autograd.

Thus we fix the fused_adam(w) check to account for this case.

cc: @clchan @ezhang887

cc for review @janeyx99 @albanD 


cc @vincentqb @jbschlosser @albanD @janeyx99 @crcrpar
DISABLED test_predispatch_with_for_out_dtype_nested_dynamic_shapes (__main__.DynamicShapesExportTests)
Labels: triaged, module: flaky-tests, skipped, module: dynamo

Platforms: asan, linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_predispatch_with_for_out_dtype_nested_dynamic_shapes&suite=DynamicShapesExportTests) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/16222572004).

Over the past 3 hours, it has been determined flaky in 54 workflow(s) with 162 failures and 54 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_predispatch_with_for_out_dtype_nested_dynamic_shapes`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `dynamo/test_dynamic_shapes.py`

cc @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov
enable in-place buffer mutation
Labels: ciflow/trunk, topic: not user facing, module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #107972

Differential Revision: [D48693776](https://our.internmc.facebook.com/intern/diff/D48693776/)

cc @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov
Batching rule for aten::_scaled_dot_product_attention_math not yet implemented
Labels: triaged, actionable, module: vmap, oncall: transformer/mha, module: functorch

I got the following warning:
`UserWarning: There is a performance drop because we have not yet implemented the batching rule for aten::_scaled_dot_product_attention_math. Please file us an issue on GitHub so that we can prioritize its implementation. (Triggered internally at ..\aten\src\ATen\functorch\BatchedFallback.cpp:84.)`

Since you asked for it ("If you see any of these cases, please let us know by opening an issue at our GitHub!"), here is the issue!

cc @zou3519 @jbschlosser @bhosmer @cpuhrsch @erichan1 @drisspg @Chillee @samdow @kshitij12345 @janeyx99
[PyTorch] torch.empty_permuted: rename param name from 'physical_layout' to 'dim_order'
Labels: fb-exported, module: inductor, ciflow/inductor

Summary:
Alignes well with the new `tensor.dim_order()` fn.
Since this is arg and not kwarg, and same dtype, this change shouldn't be BC breaking.

Test Plan: CI

Differential Revision: D48176693



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
aten.lift throws error in dynamo backends -> RuntimeError: !at::functionalization::impl::isFunctionalTensor(self)  INTERNAL ASSERT FAILED at "../aten/src/ATen/FunctionalizeFallbackKernel.cpp":167
Labels: triaged, module: functionalization

### 🐛 Describe the bug

Minimal Example: 
```python
import torch

def foo(x):
    return torch.ops.aten.lift(x)

opt = torch.compile(foo, backend="inductor")
opt(torch.randn(10))
```

Using any backend (other than vanilla `eager` mode) in Dynamo seems to trigger this assertion error. The offending assertion is [here](https://github.com/pytorch/pytorch/blob/3a3cf0e09d475df9237c95ebd14debf650e0c038/aten/src/ATen/FunctionalizeFallbackKernel.cpp#L176) and the `isFunctionalTensor` check is [here](https://github.com/pytorch/pytorch/blob/3a3cf0e09d475df9237c95ebd14debf650e0c038/aten/src/ATen/FunctionalTensorWrapper.cpp#L575). It seems that the tensor is not functionalized properly? I'm a newbie to dynamo so I'm not sure exactly what the issue is but this seems to affect all the dynamo backends - even though backends like `inductor` actually attempt to [lower](https://github.com/pytorch/pytorch/blob/3a3cf0e09d475df9237c95ebd14debf650e0c038/torch/_inductor/lowering.py#L647) this op into a no-op first.

Dynamo newbie so I'm not sure if this even an intended use case, but would like to know what the issue is here.

cc @bdhirsh @ezyang @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @brucekimrokcmu @stellaraccident




### Versions

```
PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.0 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: version 3.26.4
Libc version: N/A

Python version: 3.10.12 (main, Jul  5 2023, 15:02:25) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-13.0-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M2 Max

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==2.0.1
[pip3] torchaudio==2.0.2
[pip3] torchdata==0.6.1
[pip3] torchtext==0.15.2
[pip3] torchvision==0.15.2
[conda] cpuonly                   2.0                           0    pytorch-nightly
[conda] numpy                     1.24.3                   pypi_0    pypi
[conda] pytorch-mutex             1.0                         cpu    pytorch-nightly
[conda] torch                     2.0.1                    pypi_0    pypi
[conda] torchaudio                2.0.2                    pypi_0    pypi
[conda] torchdata                 0.6.1                    pypi_0    pypi
[conda] torchtext                 0.15.2                   pypi_0    pypi
[conda] torchvision               0.15.2                   pypi_0    pypi

```
Improve Error Message in MultiMarginLoss for Inconsistent Target Size
Labels: triaged, open source, ciflow/inductor

As the suggestion in #106251, improve the error message of MultiMarginLoss for inconsistent target size using the format below:

```MultiMarginLoss: The size of input tensor ({input size}) must match the size of target tensor ({target size}) at non-singleton dimension {dimension}```

cc @GwiHwan-Go @ezyang @mikaylagawarecki
add torch.float16 support for xla amp
Labels: triaged, module: xla, module: bfloat16, module: amp (automated mixed precision)

### 🚀 The feature, motivation and pitch

https://github.com/pytorch/pytorch/blob/main/torch/amp/autocast_mode.py#L308

The "bfloat16 only" limitation is only applied to google TPU but not other backends like GPU and other accelerators.

And XLA itself is kind of a device abstraction, so for the long run, maybe these checks should be moved to lower level.


### Alternatives

_No response_

### Additional context

_No response_

cc @bdhirsh @mcarilli @ptrblck @leslie-fang-intel @jgong5
DISABLED test_redundant_clone_for_layout_convert_cuda (__main__.FreezingCudaTests)
Labels: module: rocm, triaged, module: flaky-tests, skipped, module: inductor

Platforms: rocm

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/failure/test_redundant_clone_for_layout_convert_cuda) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/16199575184).

Over the past 72 hours, it has flakily failed in 6 workflow(s).

**Debugging instructions (after clicking on the recent samples link):**
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Grep for `test_redundant_clone_for_layout_convert_cuda`


Test file path: `inductor/test_inductor_freezing.py`

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov