gopikrsmscs/torch-issues huggingface dataset:
DISABLED test_backend_match_guard_multi_threads (__main__.MiscTests)
Labels: triaged, module: flaky-tests, module: macos, skipped, oncall: pt2, module: dynamo

Platforms: mac, macos

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_backend_match_guard_multi_threads&suite=MiscTests) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/16801698712).

Over the past 3 hours, it has been determined flaky in 3 workflow(s) with 12 failures and 3 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_backend_match_guard_multi_threads`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `dynamo/test_misc.py`

cc @malfet @albanD @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Add a multiprocess CI job to torchbench dynamo runner
Labels: release notes: releng, module: dynamo, ciflow/inductor

Currently no torchbench CI job runs python benchmarks/dynamo/torchbench.py with `--multiprocess` flag, so we don't run CI test for distributed torchbench models e..g simple_gpt (https://github.com/pytorch/benchmark/pull/1867)

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
DISABLED test_nondeterministic_alert_histc_cuda (__main__.TestTorchDeviceTypeCUDA)
Labels: module: tests, triaged, module: flaky-tests, skipped, oncall: pt2

Platforms: inductor

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_nondeterministic_alert_histc_cuda&suite=TestTorchDeviceTypeCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/16792643971).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 2 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_nondeterministic_alert_histc_cuda`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_torch.py`

cc @mruberry @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
Create static analysis tool to improve ONNX export success
Labels: module: onnx, triaged, onnx-triaged, release notes: onnx

### 🚀 The feature, motivation and pitch

Although the ONNX exportes strives to export the model without code changes on user side, some limitations might arise and changes on user script become inevitable

One way to ease the changes on user side is to create a static analysis tool that identifies unsupported patterns and recommend code changes, possibly even making the changes for the user

Integrations with IDEs, such as Visual Studio Code is a plus

Supporting is a must `torch.onnx.dynamo_export` but supporting `torch.onnx.export` woul;d be a great plus

Tools to consider during planning: https://github.com/pytorch/test-infra/tree/main/tools/torchfix and https://libcst.readthedocs.io/en/latest/

### Alternatives

_No response_

### Additional context

_No response_
aten::squeeze exported to ONNX as an `If` node
Labels: module: onnx, triaged

### 🐛 Describe the bug

Hi,

I am exporting to ONNX llama from transformers implementation, and don't understand why `If` nodes are inserted for the export of this `squeeze` operator, where the first dims are always of shape `1` and `1`: https://github.com/huggingface/transformers/blob/7c63e6fc8c34dcf8b0121eaee776f41ccf3b1137/src/transformers/models/llama/modeling_llama.py#L182

It appears we are going into this path: https://github.com/pytorch/pytorch/blob/b6a1d3fb97ca8eeccf15a4c495fdd1af4b197f88/torch/onnx/symbolic_opset11.py#L937

because the symbolic helper `_get_tensor_sizes` give us [here](https://github.com/pytorch/pytorch/blob/b6a1d3fb97ca8eeccf15a4c495fdd1af4b197f88/torch/onnx/symbolic_helper.py#L578) `x_type.varyingSizes() = [None, None, None, None]` .

Is there a way to hint that the shapes are constant hard-coded to `1`? The intermediate captured graph is (the aten::sub is not in the original code base - added for debugging):

```
  %4407 : Float(1, 1, 17, 4, strides=[68, 68, 4, 1], requires_grad=0, device=cpu) = aten::sub(%cos.1, %4405, %4406), scope: transformers.models.llama.modeling_llama.LlamaForCausalLM::/transformers.models.llama.modeling_llama.LlamaModel::model/transformers.models.llama.modeling_llama.LlamaDecoderLayer::layers.0/transformers.models.llama.modeling_llama.LlamaAttention::self_attn # /home/fxmarty/hf_internship/transformers/src/transformers/models/llama/modeling_llama.py:183:0
  %4408 : int = prim::Constant[value=1](), scope: transformers.models.llama.modeling_llama.LlamaForCausalLM::/transformers.models.llama.modeling_llama.LlamaModel::model/transformers.models.llama.modeling_llama.LlamaDecoderLayer::layers.0/transformers.models.llama.modeling_llama.LlamaAttention::self_attn # /home/fxmarty/hf_internship/transformers/src/transformers/models/llama/modeling_llama.py:185:0
  %4409 : Float(1, 17, 4, strides=[68, 4, 1], requires_grad=0, device=cpu) = aten::squeeze(%4407, %4408), scope: transformers.models.llama.modeling_llama.LlamaForCausalLM::/transformers.models.llama.modeling_llama.LlamaModel::model/transformers.models.llama.modeling_llama.LlamaDecoderLayer::layers.0/transformers.models.llama.modeling_llama.LlamaAttention::self_attn # /home/fxmarty/hf_internship/transformers/src/transformers/models/llama/modeling_llama.py:185:0
```

A workaround is to use `cos = cos[0, 0]` - wondering if there is anything better.

Thank you!

cc @justinchuby 

### Versions

torch 2.0.1, opset_version = 12
DISABLED test_nondeterministic_alert_bincount_cuda (__main__.TestTorchDeviceTypeCUDA)
Labels: triaged, module: flaky-tests, skipped

Platforms: inductor

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_nondeterministic_alert_bincount_cuda&suite=TestTorchDeviceTypeCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/16782561380).

Over the past 3 hours, it has been determined flaky in 4 workflow(s) with 4 failures and 4 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_nondeterministic_alert_bincount_cuda`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_torch.py`

ResponseTimeoutError: Response timeout for 5000ms, GET https://raw.githubusercontent.com/pytorch/pytorch/main/test/test_torch.py 200 (connected: true, keepalive socket: false, socketHandledRequests: 1, socketHandledResponses: 1)
headers: {"connection":"keep-alive","content-length":"422792","cache-control":"max-age=300","content-security-policy":"default-src 'none'; style-src 'unsafe-inline'; sandbox","content-type":"text/plain; charset=utf-8","etag":"\"e774cf397307c4c11e0a581b2258b54d36c079b283546f13beeb2d80ed18b6cf\"","strict-transport-security":"max-age=31536000","x-content-type-options":"nosniff","x-frame-options":"deny","x-xss-protection":"1; mode=block","x-github-request-id":"F6E4:0CE2:26B37:2FF4E:65030054","accept-ranges":"bytes","date":"Thu, 14 Sep 2023 12:45:23 GMT","via":"1.1 varnish","x-served-by":"cache-sjc10051-SJC","x-cache":"MISS","x-cache-hits":"0","x-timer":"S1694695523.836439,VS0,VE286","vary":"Authorization,Accept-Encoding,Origin","access-control-allow-origin":"*","cross-origin-resource-policy":"cross-origin","x-fastly-request-id":"981f341f188ae8c6947a1d84967501ca2cc1c0a4","expires":"Thu, 14 Sep 2023 12:50:23 GMT","source-age":"0"}

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
PyTorch 2.1 smoke test requirements 
Labels: triaged

PyTorch-2.1 is a big release with a lots of new feature, so we need to make sure that:
- [ ] CUDA
   - [x] pypi binaries with slimmed dependencies are usable in standard AWS containers (amazonlinux:2 regression in 1.13)
   - [x] pypi binaries with slimmed dependencies are usable with stock  Ubuntu-20.04: https://github.com/pytorch/pytorch/issues/91067 . Test: https://github.com/pytorch/builder/actions/runs/6190494544/job/16806853909#step:11:2278
  - [ ] PyTorch+CUDA11.8 and CUDA 12.1 has a working FFT on 4090
  - [x] Check cuda 1.12.1 update issue: https://github.com/pytorch/pytorch/issues/94772 with small wheels
 - [ ] `torch.compile`
    - [x] Basic test works (for example see test mentioned in https://github.com/openai/triton/pull/1176 ) in PyTorch docker container
    - [ ] `torch.compile` produces a binary which can be used on 3090
    - [x] `torch.compile` raises an error if used on Windows. Test: https://github.com/pytorch/builder/actions/runs/6171808682/job/16752011341#step:9:13394
    - [x] `torch.compile` works on 3.11 : Test: https://github.com/pytorch/builder/actions/runs/6223171830/job/16889860866
  - MPS
    - [x] Resnet is usable out of the box (i.e. https://github.com/pytorch/builder/blob/main/test/smoke_test/smoke_test.py passes for MPS device). Test: https://github.com/pytorch/builder/actions/runs/6171808682/job/16752006835#step:9:1349


### Versions

2.1
Adding index function for lists
Labels: triaged, open source, module: dynamo, ciflow/inductor, release notes: dynamo

Fixes #109031


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
add _amp_foreach_non_finite_check_and_unscale_cpu_ and _amp_update_scale_cpu_ kernels on CPU
Labels: module: cpu, open source, ciflow/trunk, ciflow/periodic, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #109994
* #109993
* __->__ #109281



cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
DISABLED test_nondeterministic_alert_MaxUnpool3d_cuda_float64 (__main__.TestTorchDeviceTypeCUDA)
Labels: module: tests, triaged, module: flaky-tests, skipped, oncall: pt2

Platforms: inductor

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_nondeterministic_alert_MaxUnpool3d_cuda_float64&suite=TestTorchDeviceTypeCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/16776175972).

Over the past 3 hours, it has been determined flaky in 11 workflow(s) with 11 failures and 11 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_nondeterministic_alert_MaxUnpool3d_cuda_float64`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_torch.py`

cc @mruberry @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
DISABLED test_nondeterministic_alert_MaxUnpool3d_cuda_float32 (__main__.TestTorchDeviceTypeCUDA)
Labels: module: tests, triaged, module: flaky-tests, skipped, oncall: pt2

Platforms: inductor

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_nondeterministic_alert_MaxUnpool3d_cuda_float32&suite=TestTorchDeviceTypeCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/16768147456).

Over the past 3 hours, it has been determined flaky in 5 workflow(s) with 5 failures and 5 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_nondeterministic_alert_MaxUnpool3d_cuda_float32`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_torch.py`

cc @mruberry @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
Add a unittest for ModuleWrapPolicy callable
Labels: oncall: distributed, good first issue, triaged, pt_distributed_rampup

### 🚀 The feature, motivation and pitch

We made ModuleWrapPolicy a callable in https://github.com/pytorch/pytorch/pull/109117, and should add an appropriate unittest. 

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu
[profiler] Show shapes for lists of tensors in chrome traces
Labels: oncall: profiler

### 🚀 The feature, motivation and pitch

When profiling with record_shapes=True, chrome traces show shapes for tensors; but not for lists of tensors. It would be helpful if we also saw shape info for lists of tensors.

Suggestions:
* Build a local script that you can use for testing. It can be basic - just run a few basic pytorch ops, and then run with the profiler with the `record_shapes=True` option turned on. Then export the chrome trace, and view it in `chrome://tracing`. Example script https://gist.github.com/davidberard98/a9711961a78f50b1b0eff530322848df - note that `torch.cat` takes a list of tensors, so it's a good op for testing here. Verify that when you click on the op, it shows some shape information.
* See https://github.com/pytorch/pytorch/pull/100593 for a similar change (focus on the changes in profiler_kineto.cpp). Note that the referenced PR adds support for both collecting and displaying the information; in this case, collection is already done; you'll just need to display the info
* To avoid making traces too much larger, you should probably limit to, say, the first 30 elements if list of tensors has >30 elements.
* For now you can probably just display the information in the chrome trace, and not add it into the KinetoEvent. (You could do that too, if you want)
* Add a test

### Alternatives

_No response_

### Additional context

_No response_

cc @robieta @chaekit @aaronenyeshi @nbcsm @guotuofeng @guyang3532 @gaoteng-git @tiffzhaofb @dzhulgakov
Add .item() and .tolist() support in Dynamo/Inductor without graph break
Labels: release notes: fx, module: inductor, ciflow/inductor

Main changes:
1. Allow DynamicScalar to flow around in the system (by implementing `ir.DynamicScalar` and not throw error in `size_hint()` or `evaluate_expr()` when returned value has unbacked symint)
2. Make DynamicScalar object synonymous to its underlying Symbol object, to make downstream SymPy reasoning in various part of the system easier (NOTE: this is a semi-hacky change, and needs discussion on whether it's okay to do)
3. Change various inequality checks to use the SymPy equivalent version, to recognize the fact that now there is unbacked symint involved in those comparisons
4. Make the downstream kernel depend on the buffer that represents the symint, so that the symint buffer would be not DCE’ed away
5. Pass needed unbacked symints into Triton kernel
6. For Triton size hint, replace unbacked symints with default value 32 (NOTE: this is the only place we do this kind of direct replacement to integer value)
7. Remove the `sym_constrain_range_for_size` decomposition empty stub, and implement the `sym_constrain_range_for_size` inductor lowering (this is needed for user side `constrain_as_size` annotation to work)
8. Gate these changes behind `torch._dynamo.config.capture_scalar_outputs` flag

Will break this down into several PRs if we are good with the general approach.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @chenyang78 @kadeng @muchulee8 @aakhundov
[dynamo] Disable DDPOptimizer or error out if DDPOptimizer + static_graph is detected
Labels: triaged, module: ddp, oncall: pt2

### 🐛 Describe the bug

AFAIK, DDPOptimizer doesn't work with DDP when static_graph=True.

Until we have a fix / better solution, we could probably either warn, error out, or automatically disable DDPOptimizer if we detect static_graph=True. That would help with debugging why DDPOptimizer is erroring out.

### Versions

N/A

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
[FSDP] Simplify `_fully_sharded_module_to_handle`
Labels: triaged, module: fsdp

It looks like `_fully_sharded_module_to_handle` contains at most 1 element, so there is no reason for it to be a `Dict`.

This can also be simplified to only insert the state on the fully sharded module if not already inserted (by the `None` check):
https://github.com/pytorch/pytorch/blob/dbddf1816a847382a9e110a1ed2dfb245495df41/torch/distributed/_composable/fully_shard.py#L115-L124

cc @zhaojuanmao @mrshenli @rohan-varma @fegin @penguinwu
Remove det_singular OpInfo
Labels: open source

Fixes #93045


singular linalg_det test cases are flaky. Consensus seems to be we can remove associated OpInfo according to this:
https://github.com/pytorch/pytorch/issues/93045#issuecomment-1477674083
@lezcano 

This change is a duplicate of a different stale PR: https://github.com/pytorch/pytorch/pull/102581
metric table
Labels: module: inductor, module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #109245
* #108193

In dynamo/inductor, sometimes it helps to gather metrics/statistics for each model in different levels like model level, graph level, kernel level or pair of fusion nodes level. This kind of thing will be very easy to do with Scuba, but we only have scuba in fbcode. This PR build metric tables to solve part of the problem.

Q: why not log to stdout/err direclty
A: sometimes we need more structured data. E.g., it would be helpful to gather all the stats in a CSV and then do post-processing (like calculating a geomean etc.). Also metric table will tag each row with the model name which is helpful.

Q: what's the difference with speedup_indcutor.csv
A: speedup_indcutor.csv is a special case that gather statistics on model level: i.e., we have one row for each model. But recording statistics on finer grain level like graph etc. is also helpful.

Example use cases:
- As a followup on the bechmark fusion PR, I want to gather all the 'slow' fusion and analyze them. With the metric table, I can easily log slow fusion for each model into a csv file. Here is the log gathered for huggingface:
 https://gist.github.com/shunting314/964e73cc98368b301414ec7b7ad4c702 .
- To help understand the effect of 'loop ordering after fusion' PR, it would be helpful to gather stats like how many fusions happens for each graph. Previously we log the metric to stderr directly. But logging these metrics in a structural way is useful.
- gather number of registers, register spills, shared memory usage for each kernel in each model with runnable kernel code logged.


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
AOTAutograd should put keep mutations in the graph during training
Labels: triaged, oncall: pt2, module: aotdispatch

Tracking issue for future work in aot autograd.

Today, if you compile a model that has input mutations, and you're compiling an inference graph (because e.g. you're running your model under `no_grad()`), then AOTAutograd will do the following:

(1) functionalize the graph; `x.mul_(2)` becomes `x_updated = x.mul(2)`
(2) detect that there were input mutations, which it represents as extra `x.copy_(x_updated)` calls
(3) Put those copy_() nodes directly in the graph so that inductor can fuse them

However, this is not true in the training case. Instead, AOTAutograd will:

(1) return `x_updated` as an additional graph output
(2) run an opaque epilogue (in eager mode), that manually performs the mutation: `x.copy_(x_updated)`

Historically, getting input mutations into the graph for inference was important because of optimizers: optimizers are (usually) run under no_grad so we get an inference graphs, and captured graphs of optimizers have a large number of input mutations, due to each parameter (a graph input) being mutated. So we made sure that when you compile optimizer code as an inference graph, we keep the input mutations in the graph for inductor to fuse.

It would still be useful to keep input mutations in the graph for the training case too, though. One example is batch norm: today, inductor can't actually see and speed up the buffer mutations that occur from batch norm during training, because they are not kept in the graph.

The main difficult in getting this to work is that we need to make sure that when AOTAutograd invokes the partitioner to partition out a separate fw/bw graph, the "correct" thing needs to happen (if there are copy_() nodes in the joint graph due to input mutations, those nodes should always show up at the very end of the partitioned forward graph).

cc @ezyang @msaroufim @wconstab @anijain2305
AOTAutograd should track view chains so it can replay them, instead of using as_strided.
Labels: triaged, oncall: pt2, module: aotdispatch

This is a tracker issue for doing the longer term thing described in https://github.com/pytorch/pytorch/issues/109053.

The point of this is mainly perf: As mentioned in this internal issue, calling as_strided() means that as_strided_backward() will show up in the backward graph, which is known to be slow (internal post: https://fb.workplace.com/groups/147416452503510/posts/1211736186131257/)

cc @ezyang @msaroufim @wconstab @anijain2305
Report name of defining class along side function name in Dynamo logs
Labels: module: logging, triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

Here's a common log message I see:

```
[rank0]:[2023-09-13 11:54:28,851] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line <listcomp> /data/users/ezyang/b/torchrec/torchrec/distributed/dist_data.py:173 (inline depth: 15)
```

This message is annoying because all it says is `<listcomp>` but what I really wanted to know was what class (and function) I was in. In principle, we should be able to figure this, not from the code object (which doesn't really know what class it was defined in) but from the line number.

A shitty implementation that would do better might look like this: when processing `dist_data.py`, read and parse the Python file, giving a stream of tokens. For every line that precedes an increase of indentation, check if it is `class XXX` or `def XXX`; if so, associate all nested lines with `XXX`. This can be done recursively so you get `XXX.YYY`. The line to class/meth table can be cached so we only have to do this once per file.

cc @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @williamwen42 

### Versions

main
DISABLED test_nondeterministic_alert_MaxUnpool3d_cuda_float16 (__main__.TestTorchDeviceTypeCUDA)
Labels: module: tests, triaged, module: flaky-tests, skipped, oncall: pt2

Platforms: inductor

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_nondeterministic_alert_MaxUnpool3d_cuda_float16&suite=TestTorchDeviceTypeCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/16755371891).

Over the past 3 hours, it has been determined flaky in 3 workflow(s) with 3 failures and 3 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_nondeterministic_alert_MaxUnpool3d_cuda_float16`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_torch.py`

cc @mruberry @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
inductor/test_max_autotune having timeout issues
Labels: module: tests, triaged, oncall: pt2, module: inductor

These tests sometimes take <1min, and other times cause the entire test file to timeout, meaning they are taking upwards of 20 minutes.  Sometimes they succeed on reruns which has been reducing the impact, but if it happens to more than just 1 or 2 tests, then the job will fail since it will run out of rerun chances.

They seem to only impact `linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default)` tests

Visible on hud https://hud.pytorch.org/hud/pytorch/pytorch/bfa8429c6a2ee9e0f4050066304743f6b13f14f7/1?per_page=100 (check show unstable)

The tests were introduced in https://github.com/pytorch/pytorch/pull/108015/files

Unlikely related to parallelization, the serial mem leak check job also had problems https://github.com/pytorch/pytorch/actions/runs/6169966074/job/16746129137, but I submitted https://github.com/pytorch/pytorch/pull/109209 just to be sure.

Here is a list of tests I have noticed do this:
```
inductor/test_max_autotune.py::TestDoBench::test_max_autotune_cutlass_backend_addmm_dynamic_False_max_autotune_gemm_backends_CUTLASS
inductor/test_max_autotune.py::TestDoBench::test_max_autotune_cutlass_backend_addmm_dynamic_False_max_autotune_gemm_backends_ATen, Triton, CUTLASS
inductor/test_max_autotune.py::TestDoBench::test_max_autotune_cutlass_backend_regular_mm_dynamic_False_max_autotune_gemm_backends_CUTLASS
inductor/test_max_autotune.py::TestDoBench::test_max_autotune_cutlass_backend_regular_mm_dynamic_False_max_autotune_gemm_backends_ATen, Triton, CUTLASS
inductor/test_max_autotune.py::TestDoBench::test_max_autotune_cutlass_backend_mm_bias_dynamic_False_max_autotune_gemm_backends_CUTLASS
inductor/test_max_autotune.py::TestDoBench::test_max_autotune_cutlass_backend_mm_bias_dynamic_False_max_autotune_gemm_backends_ATen, Triton, CUTLASS
```

If this doesn't get fixed, I'm going to need to revert the change or disable all these tests.

Example log: https://ossci-raw-job-status.s3.amazonaws.com/log/16755850617
ctrl f for `KeyboardInterrupt` and the relevant logs should be above it


cc @mruberry @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
Apply release only changes to core
Labels: topic: not user facing

Utility script to run after branch cut have been completed.
Execute: ``RELEASE_VERSION=2.1 apply-release-changes.sh``
Similar to: https://github.com/pytorch/audio/pull/3590

Test PR: https://github.com/pytorch/pytorch/pull/109210

Automate generation of PRs: 
https://github.com/pytorch/pytorch/pull/108053
https://github.com/pytorch/pytorch/pull/108688
https://github.com/pytorch/pytorch/pull/108064
TST: add a numpy reproducibility test
Labels: open source, topic: not user facing, module: dynamo

Setting a global `torch._dynamo.config.use_numpy_random_stream` should make compiled numpy programs using torch.dynamo use NumPy's PRNG (at an expense of having a graph break, most likely).

ATM, this only adds a test which passes but is going to start failing once gh-109109 lands, so will need to reevaluate once that one lands.


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Pytorch ROCM windows builds
Labels: module: windows, module: rocm, triaged

### 🚀 The feature, motivation and pitch

There are more guides showing up for ROCM on Windows such as this cuda program which needed cublas dependencies compiled with AMDs equivalent HIPblas:
https://github.com/YellowRoseCx/koboldcpp-rocm/releases/tag/Windows-v1.43-ROCm

As of July 27th, AMD officially supports HIP SDK on windows:
https://www.amd.com/en/developer/rocm-hub/hip-sdk.html

It would be very good for Pytorch Windows to function with a greater variety of AMD devices.

### Alternatives

_No response_

### Additional context

_No response_

cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
DISABLED test_nondeterministic_alert_MaxUnpool2d_cuda_float64 (__main__.TestTorchDeviceTypeCUDA)
Labels: module: tests, triaged, module: flaky-tests, skipped, oncall: pt2

Platforms: inductor

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_nondeterministic_alert_MaxUnpool2d_cuda_float64&suite=TestTorchDeviceTypeCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/16744750754).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 2 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_nondeterministic_alert_MaxUnpool2d_cuda_float64`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_torch.py`

cc @mruberry @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
`torch.mean` not supported for `torch.sparse_coo_tensor`, but `torch.sum` is supported (`scipy.sparse.coo_matrix` does support both `mean` and `sum`)
Labels: module: sparse, triaged, enhancement

### 🚀 The feature, motivation and pitch

I guess, torch.mean can easily be supported at least by doing torch.sum and then dividing by the aggregated dim size?

This can be used for estimating e.g. average edge weight in a graph or similar

```python
import torch

a = torch.eye(10)

print(a.to_sparse().sum(dim = -1))
# tensor(indices=tensor([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]]),
#       values=tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]),
#       size=(10,), nnz=10, layout=torch.sparse_coo)

print(a.to_sparse().mean(dim = -1))
# NotImplementedError: Could not run 'aten::mean.dim' with arguments from the 'SparseCPU' backend. ...

```

### Alternatives

_No response_

### Additional context

_No response_

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer
DISABLED test_nondeterministic_alert_MaxUnpool2d_cuda_float32 (__main__.TestTorchDeviceTypeCUDA)
Labels: triaged, module: flaky-tests, skipped

Platforms: inductor

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_nondeterministic_alert_MaxUnpool2d_cuda_float32&suite=TestTorchDeviceTypeCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/16736765453).

Over the past 3 hours, it has been determined flaky in 7 workflow(s) with 7 failures and 7 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_nondeterministic_alert_MaxUnpool2d_cuda_float32`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_torch.py`

ResponseTimeoutError: Response timeout for 5000ms, GET https://raw.githubusercontent.com/pytorch/pytorch/main/test/test_torch.py 200 (connected: true, keepalive socket: false, socketHandledRequests: 1, socketHandledResponses: 1)
headers: {"connection":"keep-alive","content-length":"422792","cache-control":"max-age=300","content-security-policy":"default-src 'none'; style-src 'unsafe-inline'; sandbox","content-type":"text/plain; charset=utf-8","etag":"\"e774cf397307c4c11e0a581b2258b54d36c079b283546f13beeb2d80ed18b6cf\"","strict-transport-security":"max-age=31536000","x-content-type-options":"nosniff","x-frame-options":"deny","x-xss-protection":"1; mode=block","x-github-request-id":"9A3A:380B:798740:912FAB:65015924","accept-ranges":"bytes","date":"Wed, 13 Sep 2023 06:39:33 GMT","via":"1.1 varnish","x-served-by":"cache-sjc1000140-SJC","x-cache":"HIT","x-cache-hits":"1","x-timer":"S1694587173.107265,VS0,VE176","vary":"Authorization,Accept-Encoding,Origin","access-control-allow-origin":"*","cross-origin-resource-policy":"cross-origin","x-fastly-request-id":"13152540781d8e5f8c129c621c20215635673f7e","expires":"Wed, 13 Sep 2023 06:44:33 GMT","source-age":"0"}

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
FSDP vs. MiCS
Labels: oncall: distributed, module: docs, triaged, module: fsdp

### 📚 The doc issue

I have read the documentation and paper about PyTorch FSDP (https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/). I know there are two different type model sharding strategies: one is full sharding, and another is hybrid sharding. Also, i have read the MiCS in the DeepSpeed (paper: https://www.vldb.org/pvldb/vol16/p37-zhang.pdf code: https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/runtime/zero/mics.py). 
What is the difference between the PyTorch FSDP's Hybrid sharding and MiCS ? It seems that both of them first perform a reduce scatter of the gradients in the sharding Group and then the AllReduce operation in the replication group. The documentation makes me a bit confused.

### Suggest a potential alternative/fix

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @svekars @carljparker @fegin
NAN appears in the backward results of masked.cumprod on macos
Labels: needs reproduction, module: autograd, triaged, module: NaNs and Infs, module: half, module: mps

### 🐛 Describe the bug

When we add fp16 support for cumprod on CPU, there is a CI failure on macos for masked.cumprod.  I don't have a MacOS machine to reproduce this issue, so I print the results in the UT defined in test_mps.py. https://github.com/pytorch/pytorch/actions/runs/6166735824/job/16736964541. 
`mps_grad_inputs has nan:  tensor(True, device='mps:0')` NAN appears in the backward results of masked.cumprod on macos.

### Versions

latest main branch.

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @kulinseth @malfet @DenisVieriu97 @razarmehr @abhudev
DISABLED test_nondeterministic_alert_MaxUnpool2d_cuda_float16 (__main__.TestTorchDeviceTypeCUDA)
Labels: module: tests, triaged, module: flaky-tests, skipped, oncall: pt2

Platforms: inductor

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_nondeterministic_alert_MaxUnpool2d_cuda_float16&suite=TestTorchDeviceTypeCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/16731933745).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 2 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_nondeterministic_alert_MaxUnpool2d_cuda_float16`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_torch.py`

cc @mruberry @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
[pytorch][PR] [pytorch-vulkan] add aten::randn_like & aten::normal_
Labels: fb-exported, module: vulkan, release notes: vulkan, ciflow/periodic

Summary:
**COPIED FROM D48814024 (having issue on Pytorch sync).**

Implemented aten::normal_ shader and used it to create aten::randn_like.
Ops defintions:
https://pytorch.org/docs/stable/generated/torch.randn_like.html
https://pytorch.org/docs/stable/generated/torch.Tensor.normal_.html

Test Plan:
```
[ttingchulin@53491.od /data/sandcastle/boxes/fbsource (randn)]$ LD_LIBRARY_PATH=third-party/swiftshader/lib/linux-x64/ buck run fbcode/mode/dev-nosan //xplat/caffe2:pt_vulkan_api_test_bin  -- --gtest_filter="*<test>*" eg.  -- --gtest_filter="*randn_like*"

[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from VulkanAPITest
[ RUN      ] VulkanAPITest.randn_like
[       OK ] VulkanAPITest.randn_like (230 ms)
[ RUN      ] VulkanAPITest.randn_like_large
[       OK ] VulkanAPITest.randn_like_large (570 ms)
[----------] 2 tests from VulkanAPITest (801 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (801 ms total)
[  PASSED  ] 2 tests.

[ttingchulin@53491.od /data/sandcastle/boxes/fbsource (randn)]$ LD_LIBRARY_PATH=third-party/swiftshader/lib/linux-x64/ buck run fbcode/mode/dev-nosan //xplat/caffe2:pt_vulkan_api_test_bin  -- --gtest_filter="*<test>*" eg.  -- --gtest_filter="*normal_*"
[==========] Running 3 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 3 tests from VulkanAPITest
[ RUN      ] VulkanAPITest.normal_
[       OK ] VulkanAPITest.normal_ (222 ms)
[ RUN      ] VulkanAPITest.normal_large
[       OK ] VulkanAPITest.normal_large (136 ms)
[ RUN      ] VulkanAPITest.normal_error
[       OK ] VulkanAPITest.normal_error (37 ms)
[----------] 3 tests from VulkanAPITest (396 ms total)

[----------] Global test environment tear-down
[==========] 3 tests f.
```

Differential Revision: D49208229
CPU Publish: Fix Assign device error, when module has multiple devices
Labels: fb-exported, release notes: quantization, release notes: AO frontend

Summary:
Fix Assign device error, when module has multiple devices


If fc_fp16_quantization enabled for CPU model.
And module REMOTE_OTHER has multiple devices: {device(type='meta'), device(type='cpu')}

We fail on this assertion:
https://www.internalfb.com/code/fbsource/[7ad02231251739135fda7567c5d5323b5132e0ae]/fbcode/caffe2/torch/ao/quantization/fx/utils.py?lines=232


Since CPU models work on CPU devices, added a condition before the assertion.
In case, we have CPU in module list of devices. Set device as CPU.

Please see debug details:
https://docs.google.com/document/d/1pMPCeJyMPA15NhFc2uAyNDkS9azR40uaNyOP0DIgHjU/edit

Test Plan:
AIMP_DISAGG_CPU=true buck run mode/opt -c python.package_style=inplace -c fbcode.enable_gpu_sections=true lego/scripts:lego_cli -- run-locally --model_entity_id 959168967 --config_version 28 --publish_context OFFLINE_PUBLISH --lego_pipeline aiplatform.modelstore.model_generation.lego.lego_pipeline_builder.gmpp_lego_pipeline --gmpp_config '{"gmpp_pipeline_descriptor": "aiplatform.modelstore.model_generation.v1.ads_pipelines.aimp_pyper_pipeline.model_generation_pipeline", "worker_process_number":12, "worker_thread_per_process_number": 6, "use_work_assignment": true}' 2>&1 | tee /tmp/gmpp_lc.txt


Snapshot:
https://www.internalfb.com/manifold/explorer/ads_storage_fblearner/tree/user/facebook/fblearner/predictor/959168967/47

Differential Revision: D49110166
[dynamo] `torch.no_grad` doesn't preserve the name of the wrapped function (eager mode does)
Labels: triaged, module: dynamo

### 🐛 Describe the bug

```python
import torch

def cool_name(x):
    return x.sin()

def fn(x):
    return torch.no_grad(cool_name)

x = torch.ones([])
result = fn(x)
print(result.__name__)
result = torch.compile(fn, backend="eager", fullgraph=True)(x)
print(result.__name__)
```

Output:
```
cool_name
set_grad_enabled
```

### Versions

main

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
[ONNX] Provide an option to not generate `report_dynamo_export.sarif`
Labels: module: onnx, triaged

### 🚀 The feature, motivation and pitch

When bulk converting or running tests, it would be nice to have an option to not create `report_dynamo_export.sarif` because they get overwritten anyways.

cc @BowenBao 

### Alternatives

_No response_

### Additional context

_No response_
FSDP should have tests for partial state_dict and optim state_dict loading
Labels: triaged, module: fsdp, module: distributed_checkpoint

### 🚀 The feature, motivation and pitch

See the context in https://github.com/pytorch/pytorch/pull/109116. It is common for users to load pre-trained state_dict and train the new parameters. FSDP should support the use cases that state_dict and optim state_dict are partial. We need test cases for the use cases.

### Alternatives

_No response_

### Additional context

_No response_

cc @zhaojuanmao @mrshenli @rohan-varma @awgu @penguinwu
[inductor] Parameterize ir.Scan on combine_fn
Labels: open source, module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #110911
* __->__ #109132
* #106581
* #109601

This replaces `tl.cumsum` and `tl.cumprod` with calls to `tl.associative_scan`
where the combine function is generated from inductor IR.

So before we had:
```python
@triton.jit
def triton_(in_ptr0, out_ptr0, xnumel, rnumel, XBLOCK : tl.constexpr):
    xnumel = 20
    rnumel = 30
    RBLOCK: tl.constexpr = 32
    xoffset = tl.program_id(0) * XBLOCK
    xindex = xoffset + tl.arange(0, XBLOCK)[:, None]
    xmask = xindex < xnumel
    rindex = tl.arange(0, RBLOCK)[None, :]
    rmask = rindex < rnumel
    r1 = rindex
    x0 = xindex
    tmp0 = tl.load(in_ptr0 + (r1 + (30*x0)), rmask & xmask, other=0).to(tl.float32)
    tmp1 = tl.broadcast_to(tmp0, [XBLOCK, RBLOCK])
    tmp2 = tl.where(rmask & xmask, tmp1, 0)
    tmp3 = tl.cumsum(tmp2, 1)
    tl.store(out_ptr0 + (r1 + (30*x0)), tmp3, rmask & xmask)
```

Now we have:
```python
@triton.jit
def _triton_helper_fn0(arg0, arg1):
    tmp0 = tmp0 + tmp1
    return tmp0

@triton.jit
def triton_(in_ptr0, out_ptr0, xnumel, rnumel, XBLOCK : tl.constexpr):
    xnumel = 20
    rnumel = 30
    RBLOCK: tl.constexpr = 32
    xoffset = tl.program_id(0) * XBLOCK
    xindex = xoffset + tl.arange(0, XBLOCK)[:, None]
    xmask = xindex < xnumel
    rindex = tl.arange(0, RBLOCK)[None, :]
    rmask = rindex < rnumel
    r1 = rindex
    x0 = xindex
    tmp0 = tl.load(in_ptr0 + (r1 + (30*x0)), rmask & xmask, other=0).to(tl.float32)
    tmp1 = tl.broadcast_to(tmp0, [XBLOCK, RBLOCK])
    tmp2 = tl.where(rmask & xmask, tmp1, 0)
    tmp3 = tl.associative_scan(tmp2, 1, _triton_helper_fn0)
    tl.store(out_ptr0 + (r1 + (30*x0)), tmp3, rmask & xmask)
```

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
Introduce 'backend' concept to torch.export.export API
Labels: module: onnx, feature, triaged, topic: new features, module: export

### 🚀 The feature, motivation and pitch

Currently, ONNX exporter is exposed through `torch.onnx.export`, which was the only “export” API on PyTorch repo until PyTorch 2.0 introduced Dynamo and its great backends.

With the introduction of `torch.export` on PyTorch 2.0, there are 2 `export` APIs, namely `torch.onnx.export` and `torch.export.export`, which might confuse users, especially the existing ONNX users.

We propose a way to allow `torch.export.export` to support multiple backends, similar in principle to the `torch.compile` API, which allow backends to produce, for example, either a `GraphModule` with the default `backend=dynamo` or an `onnx.ModelProto` when `backend="onnx"` is specified, unifying both FX and ONNX export APIs into one.

Backend implementation are separate as it is today and users could register extra backends through `register_backend` or list the available ones through `list_backends`

Below is an example on how a model would be exported with the new API

```python
import torch

def f(x, y):
    return x[0] + y

inp = ([torch.ones(1, 3)], torch.ones(1, 3))

exported_program = torch.export.export(f, inp, backend="dynamo", options={})  # ``backend`` and ``options`` are optionals
output = exported_program(*inp)
```

The PR https://github.com/pytorch/pytorch/pull/109649 roughly implements the backend concept

The ONNX exporter could leverage dynamo under the hood and be exported as below:

```python
import torch

def func(x):
    y = x + 1
    z = y.relu()
    return (y, z)
export_options = torch.onnx.ExportOptions()  # TODO: Maybe we can flatten ExportOption in favor of a dict as dynamo?
exported_onnx = torch.export.export(func, (torch.randn(1, 1, 2),), backend="onnx", options={"export_options":  export_options},
)
```

The PR https://github.com/pytorch/pytorch/pull/109650 adds onnx as a backend using the new API. 

### Additional context

This idea was briefly brainstormed with @msaroufim @kit1980 and @malfet who referred @suo as an important stakeholder.
DISABLED test_nondeterministic_alert_MaxUnpool1d_cuda_float64 (__main__.TestTorchDeviceTypeCUDA)
Labels: module: tests, triaged, module: flaky-tests, skipped, oncall: pt2

Platforms: inductor

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_nondeterministic_alert_MaxUnpool1d_cuda_float64&suite=TestTorchDeviceTypeCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/16719710151).

Over the past 3 hours, it has been determined flaky in 3 workflow(s) with 3 failures and 3 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_nondeterministic_alert_MaxUnpool1d_cuda_float64`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_torch.py`

cc @mruberry @ezyang @msaroufim @wconstab @bdhirsh @anijain2305