gopikrsmscs/torch-issues huggingface dataset:
[optimize_ddp] moco - NameError: name 's2' is not defined 
Labels: oncall: distributed, triaged, module: ddp, oncall: pt2, module: dynamic shapes

### 🐛 Describe the bug

In https://github.com/pytorch/pytorch/issues/101939#issuecomment-1699726230 @jansel reported that when he turns on `--dynamic-shapes` on moco it fails with

```
  File "/tmp/torchinductor_jansel/dd/cdd3xq74ulaveo5x7yion4c5db2hdp4ri3ivhi5typqx3gud4fge.py", line 599, in call
    assert_size_stride(primals_103, (s0, 512, 1 + (((-1) + s2) // 8), 1 + (((-1) + s2) // 8)), (512 + (512*((((-1) + s2) // 8)*(((-1) + s2) // 8))) + (1024*(((-1) + s2) // 8)), 1 + ((((-1) + s2) // 8)*(((-1) + s2) // 8)) + (2*(((-1) + s2) // 8)), 1 + (((-1) + s2) // 8), 1))
NameError: name 's2' is not define
```

I have confirmed that when `optimize_ddp = False`, this error goes away.

Success log: https://gist.github.com/ezyang/3dd898ba65d3fd6451d9b3e01611ebb7
Fail log: https://gist.github.com/ezyang/76d31fb7d0e04e89b8707c946c3e46ff

I have a theory for what is going on here. In dynamic shapes, we require the invariant that all symbolic symbols be explicit inputs to the graph in question. This is required to ensure that you never accidentally pass in an argument like `f32[s0 + s1]` in which case it is impossible to infer what the true s0/s1 variables is. In particular, if you don't pass in the symbol as an explicit argument, Inductor won't bind it.

`optimize_ddp` involves splitting graphs. I doubt that it maintains this invariant when generating graphs, which is the cause of this problem.

Because we are sunsetting `optimize_ddp` soon, I am not sure if it is worth directly fixing this.

### Versions

main

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @msaroufim @wconstab @bdhirsh @anijain2305
Inconsistent, platform-dependent torch.ones_like behavior on metatensors
Labels: triaged, module: meta tensors

### 🐛 Describe the bug

Calling `torch.ones_like` in Python and providing a metatensor results in a tensor with the correct shape but unpredictable values.

```
import torch
print(torch.__version__)
x = torch.empty(12, device="meta")
print(x)
t = torch.ones_like(x, device="cpu")
print(t)
```

On Google Colab:
```
2.0.1+cu118
tensor(..., device='meta', size=(12,))
tensor([ 2.4151e-38,  4.3132e-41, -4.6492e+36,  3.3216e-41,  1.5909e-38,
         4.3132e-41, -7.1350e+31,  4.3131e-41,  1.5046e-38,  4.3132e-41,
        -7.1691e+31,  4.3131e-41])
```

On OSX (Macbook M1 Pro):
```
2.0.1
tensor(..., device='meta', size=(12,))
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
```

And interestingly:
```
2.2.0a0+fb
tensor(..., device='meta', size=(12,))
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
```

Not sure if this means this issue has already been fixed in OSS, but I figured it was worth raising regardless.



### Versions

PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.5.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.6 (default, May  7 2023, 23:32:44)  [Clang 14.0.3 (clang-1403.0.22.14.1)] (64-bit runtime)
Python platform: macOS-13.5.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpydoc==1.5.0
[pip3] torch==2.0.1
[conda] Could not collect

cc @ezyang @eellison @bdhirsh
Ubuntu vs Windows: torch.cuda.OutOfMemoryError only happens on Ubuntu
Labels: module: memory usage, triaged

### 🐛 Describe the bug

I am trying to run inference on the CodeLlama-7b-Instruct model from Facebook.
I originally got the example running on my Windows laptop, with an Nvidia T550 GPU with 4GB of vram and 32GB of ram. It was slow, but it worked without crashing. What is worth mentioning is that everything but the model running on the integrated graphics, so the entire 4GB of vram was available to the model. Almost the entire 4GB was used by the model, and the rest was loaded onto RAM, with around 18gb of usage.
Then, I did the exact same thing but on an Ubuntu desktop. It has an Nvidia Quadro T2000 Mobile, with also 4GB of vram and 32GB of ram. However, it crashed with the error message below. Since this desktop doesn't have integrated graphics and idles at ~0.5GB usage of the already very little 4GB of vram, I assumed that 3.5GB of VRAM is just not enough.
Now, I tried the exact same model, with the exact same example on a Ubuntu server, with an Nvidia RTX 2080 Ti with 11GB of vram and 64GB of ram. The same error occurred.
One last thing I tried, is plugging in a different disk with Windows on it on the same server with the same hardware, and it ran on there. Much faster than the laptop running Windows as well.

So what is the difference that makes it so I can run the model on Windows but not on Ubuntu?

I found #44380, which implies that it shouldn't even be running on the 11GB card let alone the 4GB card since falling back to system memory isn't supported.

The error message:
https://gist.github.com/MartijnVerberk/3c7a6922cd775ebaf4a442b91de77b52#file-torch-cuda-outofmemoryerror-txt

My ram usage while loading the model:
https://gist.github.com/MartijnVerberk/ab2c0dc6334a24b729ba83e7e41390f8#file-ram_usage-txt

### Versions
https://gist.github.com/MartijnVerberk/03af9369aafa89b3abfba5c2b3b13e76#file-collect_env-txt
DISABLED test_nondeterministic_alert_MaxUnpool1d_cuda_float16 (__main__.TestTorchDeviceTypeCUDA)
Labels: module: tests, triaged, module: flaky-tests, skipped, oncall: pt2

Platforms: inductor

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_nondeterministic_alert_MaxUnpool1d_cuda_float16&suite=TestTorchDeviceTypeCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/16605009409).

Over the past 3 hours, it has been determined flaky in 3 workflow(s) with 3 failures and 3 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_nondeterministic_alert_MaxUnpool1d_cuda_float16`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_torch.py`

cc @mruberry @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
Fix the max pool kernel with channels_last memory layout
Labels: module: cpu, open source, ciflow/trunk, release notes: memory format

Fixes #109586

In the kernel for the `channels_last` memory layout pytorch doesn't set the initial value correctly for integer inputs (setting the it to zero rather than the actual minimum value). This causes us to get the wrong value out for max pool if all the inputs are negative.

This fixes the onnx test (for int8) because the onnx test checks that the value returned by onnx is the same as the value pytorch computes and it is now computed correctly in pytorch.

ps: This is not related to ONNX. An ONNX test was fixed as the result of fixing a pytorch kernel

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
about nccl not work
Labels: module: build, triaged, module: nccl

### 🐛 Describe the bug

When I used the nccl framework for distributed training of multiple machines, the problem was located in the nccl component did not work properly. I downloaded and installed the conforming version of NCCL according to the guidelines on the official website, but it still did not work when I ran it again.

I would like to ask:

1. I found that the pytorch repository already has nccl built-in, but there seems to be no relevant documentation on the official website. Will this built-in nccl conflict with the nvidia official website?

2. What should I do if I want to change the built-in nccl
![image](https://github.com/pytorch/pytorch/assets/75578545/21bbe386-7faf-4d5a-92bf-f88c5fd40b8a)



### Versions

cuda：12.1
python: 3.8
NCCL (from NVIDIA):  2.18.3
NCCL (from Pytorch): 2.14.3

cc @malfet @seemethere
WIP
Labels: module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108845

WIP


WIP

cc @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
[dynamo][guard refactor] C++ Guard data structure
Labels: module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #110590
* #110735
* #110589
* #110265
* __->__ #108839

**This PR does not change any existing guards flow in Dynamo.**

This PR is introducing new C++ level data structure to holds guard. Goal is to have necessary python extensions to conceptually represent any source and any guard using these data structures. 

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
CompileId in Dynamo log messages should include restart analysis count
Labels: module: logging, triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

Instead of `[0/0]` on both analysis 1 and analysis 2, if we restart analysis we should add another number, e.g., `[0/0@1]` or something. Probably best to elide the number if not needed. This makes it visually obvious when analysis was restarted.

### Versions

main

cc @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Export should never unspec NN module
Labels: triaged, oncall: pt2, module: dynamo, module: export

### 🐛 Describe the bug

See https://github.com/pytorch/pytorch/issues/108805#issuecomment-1711004337

When an NN module gets unspecialized, it forces all parameters to become inputs. This is never desirable for export, as input matching will now fail in this regime. Instead, we should prevent unspecialization from happening and/or immediately error if we would have to unspec a module.

If you decide not to fix this, you need to make the error message better; it seems my input matching error message isn't sufficient to catch this case.

### Versions

main

cc @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
torch._export.pass_base.ExportPassBaseError: Unsupported target type: <function sym_min at 0x7ff25dd10670>
Labels: triaged, oncall: pt2, module: dynamic shapes, module: export

### 🐛 Describe the bug

sym_min/sym_max appear not to be supported by torch.export.export (but they are supported by torch._dynamo.export)

```
  File "/data/users/ezyang/b/pytorch/torch/fx/interpreter.py", line 138, in run
    self.env[node] = self.run_node(node)
  File "/data/users/ezyang/b/pytorch/torch/_export/pass_base.py", line 232, in run_node
    return super().run_node(n)
  File "/data/users/ezyang/b/pytorch/torch/fx/interpreter.py", line 195, in run_node
    return getattr(self, n.op)(n.target, args, kwargs)
  File "/data/users/ezyang/b/pytorch/torch/_export/pass_base.py", line 209, in call_function
    raise ExportPassBaseError(f"Unsupported target type: {target}")
torch._export.pass_base.ExportPassBaseError: Unsupported target type: <function sym_min at 0x7ff25dd10670>

While executing %sym_min : [num_users=1] = call_function[target=torch.sym_min](args = (%sym_size, 1000), kwargs = {})
Original traceback:
  File "/data/users/ezyang/b/torchvision/torchvision/models/detection/retinanet.py", line 660, in forward
    detections = self.postprocess_detections(split_head_outputs, split_anchors, images.image_sizes)
  File "/data/users/ezyang/b/torchvision/torchvision/models/detection/retinanet.py", line 536, in postprocess_detections
    scores_per_level, idxs = scores_per_level.topk(num_topk)
```

cc @msaroufim @wconstab @bdhirsh @anijain2305 @avikchaudhuri 

### Versions

main
torch._dynamo.export produces object that is not pickleable
Labels: module: serialization, triaged, oncall: pt2, module: export

### 🐛 Describe the bug

If you do try to save it you get

```
Traceback (most recent call last):
  File "/data/users/ezyang/b/pytorch/wn.py", line 14, in <module>
    torch.save(exported_model, "retina_retina.pt")
  File "/data/users/ezyang/b/pytorch/torch/serialization.py", line 619, in save
    _save(obj, opened_zipfile, pickle_module, pickle_protocol, _disable_byteorder_record)
  File "/data/users/ezyang/b/pytorch/torch/serialization.py", line 831, in _save
    pickler.dump(obj)
TypeError: cannot pickle 'weakref.ReferenceType' object
```

I guess this isn't supposed to matter too much because we're supported to use torch.export.export.

### Versions

main

cc @mruberry @mikaylagawarecki @msaroufim @wconstab @bdhirsh @anijain2305
DISABLED test_lstm_packed (__main__.CPUReproTests)
Labels: triaged, skipped, module: inductor

Platforms: asan

Although this is already marked as a slow test, this is timing out on slow job itself, i.e. https://hud.pytorch.org/pytorch/pytorch/commit/c75aec90d3e901abd604d075c5695ae561608605.  I need to disable the test until we figure out how to address it.  Is this too slow even for the current timeout value of 30m?

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @clee2000
[PT2] Make masked_scatter_backward traceable without unbacked symints
Labels: open source, module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108804
* #108803

`masked_scatter_backward` creates a temorary result from `masked_select` which has an unbacked size,
however it is immediately padded to a known size from the original `source` shape.

This merges those two into a single ATen operator `masked_select_static` so the tracing sees only
tensors with known sizes in the graph.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
Dynamo Swallowing Exception In Lambda
Labels: triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

Steps to reproduce:

- remove `@skipIfTorchInductor("AssertionError: RuntimeError not raised by <lambda>")` from `test_index_add_correctness` in `test_torch.py`
- run PYTORCH_TEST_WITH_INDUCTOR=1 python test/test_torch.py -k test_index_add_correctness, will get error:

```
FAIL: test_index_add_correctness (__main__.TestTorch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_torch.py", line 6046, in test_index_add_correctness
    self.assertRaises(RuntimeError, lambda: result.index_add_(dim, index, source))
  File "/scratch/eellison/work/pytorch/torch/testing/_internal/common_utils.py", line 3321, in assertRaises
    return super().assertRaises(expected_exception, *args, **kwargs)
AssertionError: RuntimeError not raised by <lambda>
```

Now, if you rewrite the test to just invoke the code which is supposed to throw:

```
diff --git a/test/test_torch.py b/test/test_torch.py 
index ffcf3bfcf07..de340b1500d 100644
--- a/test/test_torch.py
+++ b/test/test_torch.py
@@ -6044,9 +6044,7 @@ class TestTorch(TestCase):
                 result = torch.zeros(1, 512, 256, dtype=dtype)
                 source = torch.ones(1, 512, 256, dtype=dtype)
                 index = torch.ones(257).to(dtype=torch.long)
-                self.assertRaises(RuntimeError, lambda: result.index_add_(dim, index, source))
-                index = (torch.ones(256) * 257).to(dtype=torch.long)
-                self.assertRaises(RuntimeError, lambda: result.index_add_(dim, index, source))
+                result.index_add_(dim, index, source)
```

And re-run the test and an exception is thrown:
> RuntimeError: index_add_(): Number of indices (257) should be equal to source.size(dim): (256), for dim: 2

I think somehow the exception in the lambda is being swallowed within dynamo. I'm not sure of root cause. You can verify that this code snippet throws in a stand alone script:

```
import torch

@torch.compile()
def foo():
    dtype = torch.float
    dim = -1
    result = torch.zeros(1, 512, 256, dtype=dtype)
    source = torch.ones(1, 512, 256, dtype=dtype)
    index = torch.ones(257).to(dtype=torch.long)
    result.index_add_(dim, index, source)
    return result

foo()
```



### Versions

master

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Back out "[Dynamo x FSDP] Add support for params, buffers, submodules on FSDPManagedNNModuleVariable (#107923)"
Labels: fb-exported, module: dynamo, ciflow/inductor

Summary:
Original commit changeset: 33650f7cb0fb

Original Phabricator Diff: D48833682

Test Plan: See T162942232 for how we figured out that this diff caused significant numeric difference.

Reviewed By: voznesenskym, richqyz

Differential Revision: D49068818




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
[C++ Frontend] Simple Changes for Cleaner Options
Labels: module: cpp, triaged

### 🚀 The feature, motivation and pitch

The current implementation of many options in the high-level C++ frontend is rather clunky, due to the desire to replicate python's named arguments. I will be using `torch::data::DataLoaderOptions` as an example, but the issue applies to almost all functions. Here is an example from [the official tutorial docs](https://pytorch.org/tutorials/advanced/cpp_frontend.html): 
```c++
auto data_loader = torch::data::make_data_loader(
    std::move(dataset),
    torch::data::DataLoaderOptions().batch_size(kBatchSize).workers(2));
```
The use of naming the entire type, `torch::data::DataLoaderOptions`, makes the code unnecessary long and intimidating. It is more apparent when only on one line:
```c++
auto data_loader = torch::data::make_data_loader(std::move(dataset), torch::data::DataLoaderOptions().batch_size(kBatchSize).workers(2));
```
C++20's [designated initializers] would make this easier, as it allows braced initialization of data structs by naming individual specified members, leaving others default initialized. It is significantly cleaner and more concise, while also better aligning with the python API:
```c++
auto data_loader = torch::data::make_data_loader(std::move(dataset), { .batch_size = kBatchSize, .workers = 2 });
```
`DataLoaderOptions` and other options do not allow for aggregate initialization since it relies on the [`TORCH_ARG`](https://github.com/pytorch/pytorch/blob/main/torch/csrc/api/include/torch/arg.h) macro, which creates the args as private members, and since some have a user-defined constructor (e.g. `DataLoaderOptions(size_t batch_size)`). From what I can tell, creating them as private members has no advantage and is only to conform to a traditional OOP factory paradigm, so this change should be trivial. The other change of removing the constructor may be less favorable, but I believe the benefit to significantly outweigh the costs. Breaking backwards compatibility is a big deal, but even on older language standards, the fix for existing codebases is simply changing `torch::data::DataLoaderOptions(batch_size)` to `torch::data::DataLoaderOptions().batch_size(batch_size)`, which also leads to more readable code.

### Alternatives

_No response_

### Additional context

_No response_

cc @jbschlosser
[state_dict][2/N] Implement the flattening and unflattening of optimizer state_dict feature

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108790
* #105902

This PR implements the flattening optimizer state_dict ability for `distributed.state_dict()`. `distributed.load_state_dict()` will detect whether the input optimizer state_dict is flattened or not and performs the unflattening process if necessary.

Differential Revision: [D49064111](https://our.internmc.facebook.com/intern/diff/D49064111/)
Collect more env variables in `collect_env.py`
Labels: triaged, needs research

### 🚀 The feature, motivation and pitch

`torch.utils.collect_env` is a script that collects certain information about a user's OS, setup and hardware. The current bug report template asks every bug reporter to include this script's output so that PyTorch maintainers can track down bugs more easily. 

While the script already collects a lot of information, there are multiple environment variables controlling `torch`'s behavior that are not included in the output. Some examples:

- `TORCH_ALLOW_TF32_CUBLAS_OVERRIDE`
- `TORCH_LINALG_PREFER_CUSOLVER`
- `TORCH_CUDNN_USE_HEURISTIC_MODE_B`
- `TORCH_NCCL_USE_COMM_NONBLOCKING`
- `CUBLAS_WORKSPACE_CONFIG`
- `NVIDIA_TF32_OVERRIDE`
- and many more ...

My proposal is to add all environment variables starting with either `TORCH_`, `PYTORCH_`, `CUBLAS_` or `NVIDIA_` to `collect_env`'s output. This might be helpful to reproduce and track down bugs.

Possible downsides: Environment variables might contain personal information, such as the user's name. It should, however, be possible to minimize this risk by stripping/replacing certain parts based on some simple heuristics.

### Alternatives

_No response_

### Additional context

I can work on a PR myself, if this is considered a useful addition.
RuntimeError: tried to get Double out of SymFloat
Labels: triaged, oncall: pt2, module: dynamic shapes

### 🐛 Describe the bug

This test case

```
    def test_double_from_sym_float(self):
        def f(x, factor):
            return torch.sym_int(x.size(0) * factor)

        make_fx(f, tracing_mode="symbolic")(torch.randn(5, 4), torch.tensor(2.0))
```

fails with

```
======================================================================
ERROR: test_double_from_sym_float (__main__.TestSymbolicTracing)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/users/ezyang/b/pytorch/torch/testing/_internal/common_utils.py", line 2388, in wrapper
    method(*args, **kwargs)
  File "/data/users/ezyang/b/pytorch/test/test_proxy_tensor.py", line 1062, in test_double_from_sym_float
    make_fx(f, tracing_mode="symbolic")(torch.randn(5, 4), torch.tensor(2.0))
  File "/data/users/ezyang/b/pytorch/torch/fx/experimental/proxy_tensor.py", line 838, in wrapped
    t = dispatch_trace(wrap_key(func, args, fx_tracer, pre_dispatch), tracer=fx_tracer, concrete_args=tuple(phs))
  File "/data/users/ezyang/b/pytorch/torch/_compile.py", line 24, in inner
    return torch._dynamo.disable(fn, recursive)(*args, **kwargs)
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/eval_frame.py", line 338, in _fn
    return fn(*args, **kwargs)
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/external_utils.py", line 17, in inner
    return fn(*args, **kwargs)
  File "/data/users/ezyang/b/pytorch/torch/fx/experimental/proxy_tensor.py", line 461, in dispatch_trace
    graph = tracer.trace(root, concrete_args)
  File "/data/users/ezyang/b/pytorch/torch/fx/_symbolic_trace.py", line 817, in trace
    (self.create_arg(fn(*args)),),
  File "/data/users/ezyang/b/pytorch/torch/fx/experimental/proxy_tensor.py", line 497, in wrapped
    out = f(*tensors)
  File "/data/users/ezyang/b/pytorch/test/test_proxy_tensor.py", line 1060, in f
    return torch.sym_int(x.size(0) * factor)
  File "/data/users/ezyang/b/pytorch/torch/__init__.py", line 420, in sym_int
    return py_int(a)  # type: ignore[operator]
RuntimeError: tried to get Double out of SymFloat
```

A workaround is to write factor.item() instead of factor. There is probably a mistake in how we implement the Scalar extraction object, possibly on x.size(0) itself.

Test case extracted torchvision retinanet: https://github.com/pytorch/pytorch/issues/108337

### Versions

main

cc @msaroufim @wconstab @bdhirsh @anijain2305
SDPA with nested backend: expose a way to avoid recomputing data layout information
Labels: triaged, oncall: transformer/mha

### 🚀 The feature, motivation and pitch

Hi, `torch.nn.functional.scaled_dot_product_attention` has a very nice torch.nested backend that allows to use e.g. flash attention with variable length inputs. In general, this is useful when doing training (fine-tuning) with padding, or during batched inference.

However, profiling SDPA and looking at the code, it happens that some overhead exists for the torch.nested backend, notably https://github.com/pytorch/pytorch/blob/49e964cad6c6f0241e08adb619bec637763888a8/aten/src/ATen/native/nested/cuda/NestedTensorTransformerFunctions.cpp#L228 which computes cumulative sequence length and maximum sequence length, especially this [to](https://github.com/pytorch/pytorch/blob/49e964cad6c6f0241e08adb619bec637763888a8/aten/src/ATen/native/nested/cuda/NestedTensorTransformerFunctions.cpp#L257C14-L257C14) operation. However, those are usually constants over different layers.

![image](https://github.com/pytorch/pytorch/assets/9808326/eed6e1bf-3611-4ebe-b102-82bac1fdbb4a)
(with code borrowed from the nice path in https://github.com/pytorch/pytorch/issues/105913, here max_seqlen=4096, pad percentage=0.4, batch_size=16, head_dim=128, num_heads=32)

As computing those stats only once [is recommended](https://github.com/Dao-AILab/flash-attention/issues/11#issuecomment-1156681278) for flash attention, it would be neat to be able to indeed compute them only once using PyTorch interface.

Related: https://github.com/huggingface/transformers/pull/25598

### Alternatives

/

### Additional context

/

cc @jbschlosser @bhosmer @cpuhrsch @erichan1 @drisspg
[ghexport] reintroduce old way of non diff train way of finding oss base
Labels: topic: not user facing

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108769
* #108768

When we introduced diff train to ghexport, it made it such that it did not work with non difftrain prs. Here we reintroduce that logic erased from D39008711

Differential Revision: [D49042620](https://our.internmc.facebook.com/intern/diff/D49042620/)
[ghexport] dynamically get github repo id
Labels: topic: not user facing

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #108769
* __->__ #108768

Get github repo id from graphql api such that it's not hardcoded to pytorch

Differential Revision: [D49042635](https://our.internmc.facebook.com/intern/diff/D49042635/)
[Quantization] Add "quantization_tag" as metadata to fx proxy
Labels: release notes: quantization

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108764

Summary:
In order to make sure that quantization_tag is preserved through second
stage export, this PR adds it as a special metadata that should be
preserved.

Since quantization in export path will work on top of pre dispatch
graph, subsequent post dispatch op decomposition, will decompose ops
that quant workflow tagged. In order to make sure that the patterns
identified by quantizer, remains identifiable, even after decompositions
are applied, we must preserve "quantization_tag".

This enables backend delegates, that quantized a model for specific
backend, to be able to identify "quantized" patterns.

Test Plan:
metadata porting tests

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D49056259](https://our.internmc.facebook.com/intern/diff/D49056259)
[temp] enabled benchmark fusion
Labels: module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108755
* #100331
* #108193



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
updated test cases to use MultithreadTestCase
Labels: triaged, open source, topic: not user facing

[![workerB](https://img.shields.io/endpoint?url=https%3A%2F%2Fworkerb.linearb.io%2Fv2%2Fbadge%2Fprivate%2FU2FsdGVkX1TEFpJB0utNppyMMjGuiCEcbqTi9eRSI%2Fcollaboration.svg%3FcacheSeconds%3D60)](https://workerb.linearb.io/v2/badge/collaboration-page?magicLinkId=VSV8ekm)
Fixes #108744
[dtensor] Add debug tool to visualize sharding
Labels: triaged, module: dtensor

Add a debug tool for sharding visualization, put the API in `https://github.com/pytorch/pytorch/tree/main/torch/distributed/_tensor/debug`, where we can define a API

`visualize_sharding(dtensor)`, where it takes in a dtensor, and print out a nice figure about the sharding layout for 1d - 2d tensor, this would greatly help debugging the sharding layout of a particular dtensor. 

One can be creative about what python package to use for sharding visualization in terminal, one example could be https://pypi.org/project/tabulate/
switch more test cases to use MultithreadTestCase
Labels: good first issue, triaged, module: dtensor

MultithreadTestCase allow us to run less resouce by spawning threads instead of processes, which could make distributed tests run faster. We have the following test files still not using MultithreadTestCase, and we should switch those test case to use it.

[ ] https://github.com/pytorch/pytorch/blob/main/test/distributed/_tensor/test_math_ops.py
[ ] https://github.com/pytorch/pytorch/blob/main/test/distributed/_tensor/test_matrix_ops.py
[ ] https://github.com/pytorch/pytorch/blob/main/test/distributed/_tensor/test_tensor_ops.py
[ ] https://github.com/pytorch/pytorch/blob/main/test/distributed/_tensor/test_embedding_ops.py

Example test case that already uses multithreaded test case, see https://github.com/pytorch/pytorch/blob/main/test/distributed/_tensor/test_pointwise_ops.py#L75

one just need to extend the `DTensorOpTestBase` for the above test files, should be relatively simple
DISABLED test_complex_half_reference_testing_fft_hfft2_cuda_complex32 (__main__.TestCommonCUDA)
Labels: triaged, module: flaky-tests, skipped, module: primTorch

Platforms: inductor

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_complex_half_reference_testing_fft_hfft2_cuda_complex32&suite=TestCommonCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/16564758724).

Over the past 3 hours, it has been determined flaky in 4 workflow(s) with 8 failures and 4 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_complex_half_reference_testing_fft_hfft2_cuda_complex32`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_ops.py`

cc @ezyang @mruberry @Lezcano @peterbell10 @msaroufim @wconstab @bdhirsh @anijain2305
[dtensor] enable tensor metadata check across ranks when run_check=True
Labels: triaged, module: dtensor

https://github.com/pytorch/pytorch/blob/main/torch/distributed/_tensor/api.py#L106

When calling `DTenor.from_local`, by default we need to sync the tensor metadata across ranks to ensure user passed in the same  type of tensor across ranks, this is a safety check and can be turn off by `run_check=False`, we should implement this feature by using `dist.allgather_object` to all gather the tensor metadata, and make sure each rank have the same metadata.
[Decomposition] sum
Labels: fb-exported, ciflow/inductor

Summary:
Decomp already exists; include it in core_aten_decompositions

https://www.internalfb.com/code/fbsource/[e69bf00ff87a55c9a30bd7905881661ff05fa211]/xplat/caffe2/torch/_refs/__init__.py?lines=2228

Differential Revision: D49042180
[Decomposition] rand_like
Labels: fb-exported, module: inductor, ciflow/inductor

Summary: Move decomp from _inductor and include it in core_aten_decompositions

Differential Revision: D48940164




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[Decomposition] lift_fresh
Labels: fb-exported, ciflow/inductor

Summary:
Decomp already exists. Include it in core_aten_decompositions

https://www.internalfb.com/code/fbsource/[b15dc20207e33abb49621994196f2ee063724d2a]/fbcode/caffe2/torch/_decomp/decompositions.py?lines=1777

Differential Revision: D48871716
Support benchmark fusion for TemplateKernel
Labels: triaged, module: inductor, module: dynamo

### 🚀 The feature, motivation and pitch

A followup for https://github.com/pytorch/pytorch/pull/108193 

### Alternatives

_No response_

### Additional context

_No response_

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[TGIF Inplace] [xlv2][1/n]  Expose a couple APIs from inline_container that will be used for chunk read
Labels: caffe2, fb-exported

Summary: Expose APIs needed for chunk read. Tested together with stacked diff.

Test Plan:
unit test
```
buck test //caffe2/caffe2/serialize:inline_container_test  --  --run-disabled --print-passing-details
```

Integration test is done together with stacked diff.

Differential Revision: D48544397


// Temporarily adding to unblock shipIt:
@diff-train-skip-merge
doctr_reco_predictor: ERROR:common:call_function groupby in skip_files Builtin groupby
Labels: triaged, oncall: pt2, module: dynamo, module: export

This is because we dont support itertools.groupby

Repro
~~~
python benchmarks/dynamo/torchbench.py --bfloat16 --accuracy --inference --device cuda  --export-aot-inductor  --only doctr_reco_predictor
~~~

1) Come up with a repro
2) Search for itertools.accumulate impl in Dynamo. Investigate if gropuby can be implemented.
3) Add your repro test case. Use fullgraph=True to ensure no graph break.

cc @ezyang @msaroufim @wconstab @bdhirsh @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Adding Maximal Update Parametrization (µP) to torch.nn.init
Labels: module: nn, triaged, needs research

### 🚀 The feature, motivation and pitch

Using muP as the default parameter initializer as it represents a unique point in the parametrization space.

### Alternatives

Status Quo

as suggested in https://github.com/pytorch/pytorch/issues/102477#issuecomment-1574125554

### Additional context

https://github.com/microsoft/mup
https://arxiv.org/abs/2011.14522

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
Simplify symbolize choice

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108677

Previously we were using a weak symbol, but in certain internal builds it didn't get overridden correctly. This just puts the conditional compilatin right in the unwind.cpp file.

Differential Revision: [D49020023](https://our.internmc.facebook.com/intern/diff/D49020023/)
RuntimeError when calling conv_transpose2d with groups
Labels: needs reproduction, module: nn, triaged

### 🐛 Describe the bug

When calling conv_transpose2d with groups > 1, I run into the following error:
```
    r = torch.nn.functional.conv_transpose2d(a, wts, stride=stride, padding=padding, dilation=dilation,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: could not construct a memory descriptor using a format tag
```

Full code snippet:
```
import torch

device = "cpu"

a = torch.randn((9, 42, 1, 5), device=device, dtype=torch.float32)
wts = torch.randn((42, 1, 1, 1), device=device, dtype=torch.float32)
stride = [1, 1]
padding = [0, 0]
dilation = [4, 3]
groups = 2

r = torch.nn.functional.conv_transpose2d(a, wts, stride=stride, padding=padding, dilation=dilation, groups=groups)
print(r)
```



### Versions
```
PyTorch version: 2.0.0.post101
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.11.5 | packaged by conda-forge | (main, Aug 27 2023, 03:34:09) [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-5.4.0-42-generic-x86_64-with-glibc2.31
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
...

Versions of relevant libraries:
[pip3] numpy==1.25.2
[pip3] torch==2.0.0.post101
[conda] mkl                       2022.2.1         h84fe81f_16997    conda-forge
[conda] numpy                     1.25.2                   pypi_0    pypi
[conda] pytorch                   2.0.0           cpu_mkl_py311had667d7_101    conda-forge
[conda] torch                     2.0.0.post101            pypi_0    pypi
```

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
use reduced_precision_reduction flags in Triton matmul
Labels: triaged, open source, topic: not user facing, module: inductor, module: dynamo, ciflow/inductor

Fixes #108621


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
Back out "Faster gc_count update for CUDACachingAllocator"
Labels: fb-exported

Summary:
Original commit changeset: 1d04ae368fd8

Original Phabricator Diff: D48481557

Test Plan: llm inference service can encounter a segfault underload. it no longer does after backing out the diff.

Reviewed By: houseroad

Differential Revision: D49003404
[inductor] Triton matmul templates should use reduced_precision_reduction flags
Labels: feature, good first issue, triaged, module: half, oncall: pt2, module: inductor, matrix multiplication

In eager mode [we have flags](https://pytorch.org/docs/stable/notes/cuda.html#reduced-precision-reduction-in-fp16-gemms) for using fp16 accumulators in matmuls.

Currently, our Triton matmul templates ignore this flag and always accumulate in float32.  This will make them slower, so we may be leaving some perf on the table in max-autotune mode.

The implementation of this just involves updating the `acc_type` function to read the correct flags:
https://github.com/pytorch/pytorch/blob/c8e72a4a5c6398ad38d41ceee9775f8d4544225c/torch/_inductor/kernel/mm_common.py#L134-L137

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[codemod] Del `(object)` from 10 inc caffe2/fb/nas_profiler/lookups/xtensa_lookup.py
Labels: fb-exported, topic: not user facing

Summary: Python3 makes the use of `(object)` in class inheritance unnecessary. Let's modernize our code by eliminating this.

Test Plan: Sandcastle

Reviewed By: meyering

Differential Revision: D48957985
[WIP] Test threaded multi compile
Labels: ciflow/trunk, release notes: releng, module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108610



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[export] Fix getattr node issues with custom obj
Labels: module: export

When we recreate a graph module with a custom object, we will run into a SyntaxError due to the python codegen not understanding what to do with the custom in-memory objects. To solve this, we bypass the issue through creating an empty GraphModule and manually set the graph (https://fburl.com/code/5pgtpju8).

However, this runs into an issue when there are attributes on the graph module. torch.fx.GraphModule initialization only copies over the attributes if there is a get_attr node in the graph (https://www.internalfb.com/code/fbsource/[3f79c0a1c045]/fbcode/caffe2/torch/fx/graph_module.py?lines=360-363). Since we don't initialize the graph module with a graph when there's a custom object in the graph, these attributes will never get copied over to the newly created graph module.

Fixes an issue from tensorrt team.
torchrun fails to run on Windows 11
Labels: module: windows, triaged

### 🐛 Describe the bug

On Windows 11, just running vanilla "torchrun train.py" gives the error "failed to create process." - with no other info (no stack trace).  

```
torch                              2.0.1
python                             3.9.0

```

### Versions

PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Enterprise
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.0 (default, Nov 15 2020, 08:30:55) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22621-SP0
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=3504
DeviceID=CPU0
Family=207
L2CacheSize=12288
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=3504
Name=Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz
ProcessorType=3
Revision=21767

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] pytorch-ranger==0.1.1
[pip3] torch==2.0.1
[pip3] torch-optimizer==0.3.0
[pip3] torchaudio==2.0.2
[pip3] torchdata==0.6.1
[pip3] torchtext==0.15.2
[pip3] torchvision==0.15.2
[conda] blas                      1.0                         mkl
[conda] mkl                       2023.1.0         h6b88ed4_46357
[conda] mkl-service               2.4.0            py39h2bbff1b_1
[conda] mkl_fft                   1.3.6            py39hf11a4ad_1
[conda] mkl_random                1.2.2            py39hf11a4ad_1
[conda] numpy                     1.23.5                   pypi_0    pypi
[conda] pytorch                   2.0.1           py3.9_cuda11.7_cudnn8_0    pytorch
[conda] pytorch-cuda              11.7                 h16d0643_5    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] pytorch-ranger            0.1.1                    pypi_0    pypi
[conda] torch-optimizer           0.3.0                    pypi_0    pypi
[conda] torchaudio                2.0.2                    pypi_0    pypi
[conda] torchdata                 0.6.1                    pypi_0    pypi
[conda] torchtext                 0.15.2                   pypi_0    pypi
[conda] torchvision               0.15.2                   pypi_0    pypi

cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite
[POC] Avoid `recordStream` for `_allgather_base`
Labels: release notes: distributed (c10d)

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #108591
* __->__ #108590

This is needed for implementing FSDP memory management without ever using `recordStream` (without setting `TORCH_NCCL_AVOID_RECORD_STREAMS=1`). Another option is to always avoid `recordStream` if `async_op=False`.
TorchInductor workers use "fork" which doesn't work in a multithreaded environment
Labels: triaged, module: multithreading, oncall: pt2, module: inductor

### 🐛 Describe the bug

TorchInductor workers use "fork" as the default method for spawning processes: https://github.com/pytorch/pytorch/blob/ff38c0e2f9cae35378553c38ccf7188007fed938/torch/_inductor/codecache.py#L1349

"fork" in general is [broken](https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf) and should not be used in modern systems where multithreading is used commonly. Even [python](https://discuss.python.org/t/switching-default-multiprocessing-context-to-spawn-on-posix-as-well/21868) plans to switch to using "spawn" as the default multiprocessing spawn method.

Creating this issue to follow up from the discussion [here](https://github.com/pytorch/pytorch/pull/87411#issuecomment-1699795308). The usage of "fork" creates non-deterministic situations where application code deadlocks or crashes in a multi-threaded environment. We should switch the default here to be "spawn" or at least give users an option to switch to using "spawn" instead.

This means we need to set `num_workers=0` which slows down compilation times significantly. First compile usually takes about 20mins with `num_workers=16`. However, with `num_workers=0` this goes up to 90mins.

### Versions

main branch

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov