gopikrsmscs/torch-issues huggingface dataset:
Torchscript with dynamic quantization produces inconsistent model outputs
Labels: oncall: jit, oncall: quantization, triaged

### 🐛 Describe the bug

Hello, I've been experimenting with torchscript and dynamic quantization and often have the issue that results of the models that are dynamically quantized are not consistent between Python and Java.
To reproduce the issue I created a fork of the python java-demo: https://github.com/westphal-jan/java-demo.

To setup you need to download libtorch and set its the location in `build.gradle` (https://github.com/westphal-jan/java-demo/blob/master/build.gradle#L16)
Download Link: https://download.pytorch.org/libtorch/cpu/libtorch-shared-with-deps-1.13.1%2Bcpu.zip

I created a simple dummy model with one linear layer and export it unquantized and quantized here: https://github.com/westphal-jan/java-demo/blob/master/create_dummy_models.py
(The code can also be run using the dependencies defined in https://github.com/westphal-jan/java-demo/blob/master/requirements.txt but I also commited the dummy models)

Python:
```
Unquantized model:
[[-2.758167028427124, 2.0038578510284424, -4.114053726196289, -1.2928203344345093, 1.4940322637557983]]
Quantized model:
[[-2.747678756713867, 1.9912285804748535, -4.110795021057129, -1.2891944646835327, 1.4982664585113525]]
```

You can run the java code with `./gradlew run`.
Java:
```
Unquantized model:
data: [-2.758167, 2.0038579, -4.1140537, -1.2928203, 1.4940323]
[W qlinear_dynamic.cpp:239] Warning: Currently, qnnpack incorrectly ignores reduce_range when it is set to true; this may change in a future release. (function apply_dynamic_impl)
Quantized model:
data: [-2.7473624, 1.9966378, -4.110954, -1.283469, 1.4918814]
```

As you can see the output of the unquantized model is perfectly consistent while the output of the dynamically quantized model is slightly inconsistent. It might seem insignificant but with larger models like a transformer it becomes more obvious (differences usually already in the first decimal place). Am I misunderstanding something conceptually? I thought as the code is compiled down to C it should be the same even when using dynamic quantization.

Note: I made sure that Python and Java use the same version of Torch `1.13.1` which is the latest published mvn version (https://mvnrepository.com/artifact/org.pytorch/pytorch_java_only)

### Versions

1.13.1

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @Xia-Weiwen @leslie-fang-intel
View ops on fake tensors can dispatch `detach`es to backend kernels
Labels: triaged, module: fakeTensor

### 🐛 Describe the bug

When calling an op on a fake tensor, a new fake tensor is made for the result via a call to `detach` ([src](https://github.com/pytorch/pytorch/blob/4e204ff87b1fbbe2f1820704e45559a6f448ce17/torch/csrc/autograd/python_variable.cpp#L587)).

When the op called is a view op (& I think it also needs to be one without a meta Python decomposition?), the `detach` is dispatched to the backend kernel (eg. `CPU`), instead of to `Meta`.

This usually isn't an issue as `detach` is declared only `CompositExplicitAutograd`, but for out-of-tree backends that _do_ intercept `detach`, they'll end up getting an op dispatched with a `Meta` tensor—this becomes an issue with Dynamo.

# Reproducer

```python
import torch

t = torch.randn(5, 5)
with torch._subclasses.FakeTensorMode() as mode:
    ft = mode.from_tensor(t)
    res = ft.view(25)
    print(res)
```

Running this with `TORCH_SHOW_DISPATCH_TRACE=1` gives:

```
[call] op=[aten::randn], key=[BackendSelect]
  [redispatch] op=[aten::randn], key=[CPU]
   [call] op=[aten::empty.memory_format], key=[BackendSelect]
    [redispatch] op=[aten::empty.memory_format], key=[CPU]
   [call] op=[aten::normal_], key=[CPU]
 [call] op=[aten::empty_strided], key=[BackendSelect]
  [redispatch] op=[aten::empty_strided], key=[Meta]
 [call] op=[aten::detach], key=[AutogradMeta]
  [redispatch] op=[aten::detach], key=[ADInplaceOrView]
   [redispatch] op=[aten::detach], key=[Meta]
 [call] op=[aten::view], key=[PythonTLSSnapshot]
  [redispatchBoxed] op=[aten::view], key=[AutogradCPU]
   [redispatch] op=[aten::view], key=[ADInplaceOrView]
    [redispatch] op=[aten::view], key=[Python]
     [callBoxed] op=[aten::view], key=[Meta]
     [call] op=[aten::detach], key=[CPU]    <<<<<===== The issue
FakeTensor(..., size=(25,))
```

Running in a debugger, in `THPVariable_make_subclass` (called from `FakeTensor.__new__`), printing `toString(r.tensor(1).key_set())` gives `DispatchKeySet(CPU, ADInplaceOrView, AutogradCPU, AutocastCPU)`.

### Versions

Collecting environment information...
PyTorch version: 2.1.0a0+git4ff3108
Is debug build: True
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.25.0
[pip3] torch==2.1.0a0+git9593b3a
[conda] numpy                     1.25.0                   pypi_0    pypi
[conda] torch                     2.1.0a0+git9593b3a          pypi_0    pypi
Add memory managemenet information for Apple silicon mps backend
Labels: triaged, enhancement, module: mps

### 🚀 The feature, motivation and pitch

Torch has useful memory management information for CUDA devices such as 
- `torch.cuda.mem_get_info`
- `torch.cuda.memory_summary`
- `torch.cuda.max_memory_allocated`
- etc...

which are useful for debugging purposes. Nevertheless, these functionalities are very limited for Apple silicon mps backend. Is it possible to include more memory management information in `torch.mps`? I think these would be the most relevant memory management information functions for mps devices that are missing:

- memory summary
- available/free GPU memory
- maximum allocated memory

### Alternatives

_No response_

### Additional context

_No response_

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
[inductor] Updated upsample_bilinear2d decomposition
Labels: open source, module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #104248
* __->__ #104182
* #104181

Description:
- Updated upsample_bilinear2d decomposition
  - added support for uint8 dtype support
  - code improvements
- Added uint8 dtype tests

The goal of this PR is performance improvements on CPU by reducing the number of generated blocks (3 blocks -> 1 block):

```
[------------------------- Interpolate bilinear, AA=false, cpu -------------------------]
                                                                   |  Eager   |  Compiled
1 threads: ------------------------------------------------------------------------------
- PR
      Input (3, 345, 456), torch.float32, torch.contiguous_format  |  1435.3  |    942.2 
      Input (3, 345, 456), torch.float32, torch.channels_last      |   984.3  |    944.0

      Input (3, 345, 456), torch.uint8, torch.contiguous_format    |   682.4  |   1366.9 
      Input (3, 345, 456), torch.uint8, torch.channels_last        |   194.0  |   1387.4

- main/nightly
      Input (3, 345, 456), torch.float32, torch.contiguous_format  |  1474.4  |   1234.5 
      Input (3, 345, 456), torch.float32, torch.channels_last      |  1001.2  |   1371.3 

      Input (3, 345, 456), torch.uint8, torch.contiguous_format    |   596.2  |   1287.0 
      Input (3, 345, 456), torch.uint8, torch.channels_last        |   195.6  |   1402.4

Times are in microseconds (us).
```

No regression for CUDA
```

[------------------------ Interpolate bilinear, AA=false, cuda ------------------------]
                                                                   |  Eager  |  Compiled
1 threads: -----------------------------------------------------------------------------
- PR
      Input (3, 345, 456), torch.float32, torch.contiguous_format  |   11.7  |    40.6  
      Input (3, 345, 456), torch.float32, torch.channels_last      |   10.9  |    41.6  

- main/nightly
      Input (3, 345, 456), torch.float32, torch.contiguous_format  |   12.2  |    42.6  
      Input (3, 345, 456), torch.float32, torch.channels_last      |   11.5  |    44.5  

Times are in microseconds (us).

```

[Source](https://gist.github.com/vfdev-5/eac35472981c09ea4898f0059011dae7)


- [main/nighlty C++ generated code](https://gist.github.com/vfdev-5/eac35472981c09ea4898f0059011dae7#file-main-generated-code-cpp) (3 blocks) vs [PR C++ generated code](https://gist.github.com/vfdev-5/eac35472981c09ea4898f0059011dae7#file-pr-generated-code-cpp) (1 block)

- [main/nighlty triron generated code](https://gist.github.com/vfdev-5/eac35472981c09ea4898f0059011dae7#file-main-triton-code-py), [PR triron generated code](https://gist.github.com/vfdev-5/eac35472981c09ea4898f0059011dae7#file-pr-triton-code-py)



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78
No document for parameter `load_debug_files` in `torch::jit::load` in C++ API
Labels: module: docs, triaged, actionable

### 📚 The doc issue

Thanks a lot for your work.

I found that there is no document for parameter `load_debug_files` in `torch::jit::load` in C++ API.

This parameter can be found in various overloads of the function, however, none of them specifies what `load_debug_files` is, and whether it affects the loaded pre-trained model when setting to false.

Ref: 
* https://pytorch.org/cppdocs/api/function_namespacetorch_1_1jit_1ac9b78087c3a653b13868cf7e6f8ed171.html#exhale-function-namespacetorch-1-1jit-1ac9b78087c3a653b13868cf7e6f8ed171
* https://github.com/pytorch/pytorch/blob/3c34a00d1b94aa4ceeede498ab878f1c5a25afb1/torch/csrc/jit/serialization/import.h#L100

I encounter this when I deployed a pre-trained model by libtorch.  For my own model, I found that `torch::jit::load` works no matter whether `load_debug_files` is set to true or false, and the inference results are also correct.

So can I safely set `load_debug_files` to false for __all models in release mode__, i.e. no "debug symbols" are included, if my understanding is correct. 

Thanks.

P.S.
I dug a little bit, and noticed that `load_debug_files` will enforce `PyTorchStreamReader` to read some "Records", however, I didn't find a definition for "Records".
https://github.com/pytorch/pytorch/blob/3c34a00d1b94aa4ceeede498ab878f1c5a25afb1/caffe2/serialize/inline_container.cc#L306

### Suggest a potential alternative/fix

_No response_

cc @svekars @carljparker
DDP enhancement 
Labels: oncall: distributed

### 🚀 The feature, motivation and pitch

The DDP feature is super cool and efficient. However, it is hardware agnostic and there is no specification regarding the machines used as backend. Making this feature hardware aware can help a lot to optimize executions.

With some tips it is possible to get better performance in this area. I added some line of code based on this idea. It is something which does not go too deep in the code structure and can be exposed to the user as argument in torchrun. The result I obtained are interesting in term of speedup and training time reduction.

I will be happy to share my experience and my code with the community, looking for your feedbacks.

Rgrds

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Network does not return any thing, not even None and breaks loops
Labels: needs reproduction, module: windows, triaged, module: third_party

### 🐛 Describe the bug

I just installed torch via `pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117`.

I have created a network 

```python
class NeuralNetwork(nn.Module):

    def __init__(self):
        super(NeuralNetwork, self).__init__()

    def forward(self, x):
        return x

```
and a training loop

```python

net = NeuralNetwork()
criterion = CrossEntropyLoss()
N_EPOCHS = 10

for e in range(N_EPOCHS):
    print(e)
    for x, y in dataLoader: # `dataLoader` is a DataLoader object
        logit = net(x)

        loss = criterion(logit, y)
        optimizer.zero_grad()

        loss.backward()

        optimizer.step()
```

Running the code only prints "0" but no error is thrown.
I have narrowed it down to it being the `net(x)` call.

If I do a short test

```python
net = NeuralNetwork()#.to(device)
x = torch.rand(1, 28, 28)
net(x)
```
then the last `net(x)` does return anything.

Doing 

```python
a = net(x)
print(a)
```
results in `no variable named "a"`


### Versions

PyTorch version: 2.0.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.11.4 (tags/v3.11.4:d2340ef, Jun  7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22621-SP0
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA RTX A3000 Laptop GPU
Nvidia driver version: 531.14
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=2611
DeviceID=CPU0
Family=207
L2CacheSize=10240
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2611
Name=11th Gen Intel(R) Core(TM) i9-11950H @ 2.60GHz
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] torch==2.0.1+cu117
[pip3] torchaudio==2.0.2+cu117
[pip3] torchvision==0.15.2+cu117
[conda] Could not collect


cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite
add dist hooks support for custom device
Labels: triaged, open source, release notes: distributed (ddp)

Fixes https://github.com/pytorch/pytorch/issues/104389
1. Now for distributed hooks, there are some hard-code like `cuda`, we want to support these hooks for custom device( privateuse1 backend), so we use the abstract device-module to run some funcs.
2. In `torch/nn/parallel/distributed.py`, I want to define a variable `self.device_module = getattr(torch, self.device_type, None)` so that we can reuse it, but it will cause an error in serialization, `TypeError: cannot pickle 'module' object`. So we call the func by `getattr` when needs.
make_fx: torch.where scalar promotion burns in device
Labels: triaged, module: aotdispatch

```
import torch
from torch.fx.experimental.proxy_tensor import make_fx

def f(x):
    return torch.where(x > 0.5, x, 1.0)
g = make_fx(f)(torch.rand(2, 3))
print(g)
```

yields
```
def forward(self, x_1):
    gt = torch.ops.aten.gt.Scalar(x_1, 0.5)
    scalar_tensor = torch.ops.aten.scalar_tensor.default(1.0, dtype = torch.float32, layout = torch.strided, device = device(type='cpu'))
    where = torch.ops.aten.where.self(gt, x_1, scalar_tensor);  gt = x_1 = scalar_tensor = None
    return where
```

Note the hard-coded cpu device, which we are getting as we trace the promotion of `1.0` from float -> tensor.

We just spend almost 2 weeks debugging IMAs in distributed training that were root caused from this behavior during export :( . Now that we have root caused it, it's trivial to detect device burn-in and ask people to change the model. But in the long run we'd like to solve this entirely. Any thoughts on how to do so?

cc @ezyang, @bdhirsh
[ONNX] Support symbolic tracing without using external `FakeTensorMode` on public API
Labels: module: onnx, triaged, enhancement, release notes: onnx

### 🚀 The feature, motivation and pitch

https://github.com/pytorch/pytorch/pull/103865 assumes that if symbolic tracing is to be used, the user must provide the `FakeTensorMode` instance used to fakefy the model input and weight.

An alternative design would be converting model input and weight into faketensors inside the `torch.onnx.dynamo_export` to eliminate the need of a `fake_tensor`.

### Alternatives

_No response_

### Additional context

_No response_
add fsdp checkpoint support for custom device
Labels: module: cpu, triaged, open source, Stale, ciflow/trunk, release notes: distributed (sharded), ciflow/inductor

Fixes https://github.com/pytorch/pytorch/issues/104390
Now for distributed checkpointing, there are some hard-code like cuda, we want to support checkpointing for custom device( privateuse1 backend), so we use the abstract device-module to run some funcs.

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
Python Crashes When Importing Torch With C API
Labels: needs reproduction, module: windows, triaged, module: third_party

### 🐛 Describe the bug

EDIT: Please read my comment to @malfet below. It isn't the case that Python is crashing. Rather, my C++ file seems to hang midway through importing torch when I call `PyImport_ImportModule`.

I'm trying to embed Python into a C++ based application using the [Python/C API][1]. If I don't include an embedded Python installation inside my projects folder, then the program uses my local Python install by default. Using the local install doesn't result in any errors and torch is able to import successfully. However, if I include a [Windows embeddable package (64-bit)][2] version of Python in my projects folder, then Python crashes after calling `PyImport_ImportModule("importsFile")` in my C++ file.

Inside of `importsFile.py`I have the following code:

```py
import numpy as np
import matplotlib
import torch
```

If I comment out `import torch`, then the other modules are able to import without error. However, if I leave `import torch` uncommented, then Python crashes.

I'm not sure if this is a bug on Python's side or PyTorch's side. Python only seems to crash when importing torch specifically, so it seems to me like it should be on PyTorch's side, but I may be wrong about this.

Is this a known issue?
How can I fix it?


  [1]: https://docs.python.org/3.9/c-api/import.html
  [2]: https://www.python.org/downloads/release/python-397/

### Versions

PyTorch version: 2.0.1+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Education
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19041-SP0
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3070 Ti
Nvidia driver version: 535.98
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=3701
DeviceID=CPU0
Family=107
L2CacheSize=6144
L2CacheSpeed=
Manufacturer=AuthenticAMD
MaxClockSpeed=3701
Name=AMD Ryzen 9 5900X 12-Core Processor
ProcessorType=3
Revision=8448

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] torch==2.0.1+cu118
[pip3] torchaudio==2.0.2+cu118
[pip3] torchvision==0.15.2+cu118
[conda] Could not collect

cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite
Re-enable `test_typing`
Labels: module: typing, module: ci, triaged

### 🚀 The feature, motivation and pitch

Further to the discussion in #103376 and OH, we need some tests for static type checking with `reveal_type` on `Tensor` methods.

However, `test_typing` is currently in the blocklist and is broken, confirmed in https://github.com/pytorch/pytorch/issues/103376#issuecomment-1595010512 (cc @ezyang @malfet @rgommers @xuzhao9 @gramster @seemethere @pytorch/pytorch-dev-infra @huydhn).

My plan is to to re-enable it, and then in #103376 I can add some more reveal tests.

### Alternatives

Create a new test for #103376, but it will include a lot of code duplication with the existing `test_typing`.

### Additional context

`test_typing` is added to the blocklist when the blocklist is introduced in #64246 c5d80e41b0271163aa1900c002dbf5d430c4b953. I'm not sure why it was blocked. If it's because it was broken, I can fix it (actually very easy); if it's because it was slow (can take 5~10 mins w/o cache), we will anyway need to spend that time for the reveal test; if it's other reasons, please tell me. (cc @malfet as the commit author).
DISABLED test_graph_breaks (__main__.LoggingTests)
Labels: triaged, module: flaky-tests, skipped, module: dynamo

Platforms: linux, rocm, mac

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_graph_breaks&suite=LoggingTests) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/14493989183).

Over the past 3 hours, it has been determined flaky in 6 workflow(s) with 18 failures and 6 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_graph_breaks`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `dynamo/test_logging.py`

cc @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @chenyang78
RuntimeError: _ivalue_ INTERNAL ASSERT FAILED
Labels: oncall: jit

### 🐛 Describe the bug

A sudden Error in using script model in cpp

`RuntimeError: _ivalue_ INTERNAL ASSERT FAILED at "/home/loc/softwares/pytorch_v2.0.0/torch/csrc/jit/api/object.h":37, please report a bug to PyTorch. `

I built my torch from source "2.0.0a0+gitc263bd4 " 
 libtorch " libtorch-cxx11-abi-shared-with-deps-2.0.0+cu118" 
cuda 11.8 on Ubuntu 22.04

Sample Code

`    torch::Tensor example_input = torch::randn({1, 3, 224, 224});

    // forward
    auto desc = model_.forward({example_input}).toTensor();
    std::cout << "des: " << desc.sizes() << std::endl;

    // return descriptors
    return desc;
`

Note

I m binding my class to python and including :

`#include "torch/torch.h"
#include <torch/script.h>
#include <torch/extension.h>`



### Versions

python=3.8.16
cuda=11.8
pytorch=2.0.0 (source)
torchvision=0.15.2


cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Regressions with torch.compile + amp + ddp with recent nightly builds
Labels: needs reproduction, oncall: distributed, module: cuda, triaged, module: ddp, oncall: pt2

              Saw similar regressions with torch.compile + amp + ddp with nightly builds from 06/19/23 and 06/20/23. Wasn't seeing this issue before. This is on both 4x A100s (40GB) and 8xH100s

_Originally posted by @nanand2 in https://github.com/pytorch/pytorch/issues/101838#issuecomment-1600056539_
            

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @ptrblck @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
[WIP]Add mutliple CUDA streams support to TorchInductor
Labels: topic: not user facing, module: inductor, module: dynamo, ciflow/inductor

It is still a very early stage PR. As for now, it could successfully run some models with `--disable-cudagraphs`.

This PR is supposed to enable TorchInductor to generate output_code.py that utilizes multiple CUDA streams. 

TODO:
- [ ] ~merge the stream context lines when the adjcent lines share the same stream.~
- [ ] use a better stream_pool_pop algorithem.
- [x] correct the wrapper code generation.
- [x] process 'nop' node.
- [x] check the correctness.
- [x] add config.multiple_stream checks.
- [ ] remove debug lines.
- [ ] solve all other TODOs in the code
- [x] finish cpp wrapper version
- [x] fix most accuracy issues
cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @anijain2305
Tracking issue for optimizer graph not being an inference graph
Labels: triaged, tracker, oncall: pt2, module: aotdispatch

### 🐛 Describe the bug

The optimizer is incorrectly classified as a training graph in aot. The main reason this occurs is that some parameters have `requires_grad=True` *and* torchdynamo calls set_grad_enabled(True|False) while simultaneously putting a call_function node in the graph which performs the global state change. 

As a result, when dynamo traces a no_grad context manager, it will trace the *exit* of the context manager *before* calling the user compiler. As a result, by the time we reach aot, grad is enabled, and the graph is treated as a training graph.

Why this matters:
Functionalization appends an epilogue which aggregates all mutation updates of tensors into a single code block consisting of a bunch of copies. In inference mode, these copies are visible to inductor while in training mode, these copies are not visible to inductor. If the copies are not visible to inductor, inductor will allocate tensors for every single parameter in the graph in order for them to get sent to the epilogue, which is atrocious for perf. 

Solutions:
1. graph break before grad is re-enabled
2. in functionalization make the epilogue visible in training mode
3. with compiled backwards, the entire graph is an inference graph by default

1 is two lines of code, so that approach is the easiest. 
3 is getting implemented currently, so once it's ready we can remove the graph breaks.


### Error logs

N/A

### Minified repro

_No response_

### Versions

N/a

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
"TorchDynamo Deeper Dive" doc is missing some part
Labels: triaged, topic: docs

In https://pytorch.org/docs/main/compile/deep-dive.html#torchdynamo-deeper-dive, the first sentence is

> TorchDynamo operates just-in-time and specializes graphs based on dynamic properties. For example, **the first graph above** has the following guards:

but there's no graph above, so it's not clear what the rest is referring to.

CC @jansel as you seem to be the author
Torch.compile tutorial shows incorrect triton kernel
Labels: triaged, topic: docs

This tutorial https://pytorch.org/docs/main/compile/get-started.html claims that `cos(x) + sin(y)` in

```py
def fn(x, y):
    a = torch.cos(x).cuda()
    b = torch.sin(y).cuda()
    return a + b
```

gets compiled into and fused into a triton kernel that computes `sin(sin(x))`:

```py
    tmp0 = tl.load(in_ptr0 + (x0), xmask)
    tmp1 = tl.sin(tmp0)
    tmp2 = tl.sin(tmp1)
    tl.store(out_ptr0 + (x0 + tl.zeros([XBLOCK], tl.int32)), tmp2, xmask)
```


I assume the triton kernel in the tutorial is wrong / outdated?
Improved error checking for custom Function when saving intermediates
Labels: module: double backwards, module: autograd, triaged, actionable

Currently, if users create a custom Function, save intermediate tensors (tensors that are neither inputs nor outputs to the custom function) for backward, and use them in the backward formula to compute gradients, the gradients produced in double backward would be silently incorrect.

We should produce a error in this case instead. One way to do this is to attach an ErrorNode to all non-input/non-output tensors being saved.

https://github.com/pytorch/pytorch/issues/103726#issuecomment-1599015774

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @Lezcano @Varal7
Reproducibility documentation should be updated
Labels: module: docs, triaged, module: numerical-reproducibility

### 📚 The doc issue
When loading the weights, the model and the optimizer are loaded. Besides, the sampled data in dataloader are guaranteed to be the same. It should be considered in https://pytorch.org/docs/stable/notes/randomness.html.

### Suggest a potential alternative/fix
```python
import torch
import numpy
import random
print(torch.__version__)  # 1.13.0+cu116

def set_random_seed(seed, deterministic=True):
    """Set random seed.

    Args:
        seed (int): Seed to be used.
        deterministic (bool): Whether to set the deterministic option for
            CUDNN backend, i.e., set `torch.backends.cudnn.deterministic`
            to True and `torch.backends.cudnn.benchmark` to False.
            Default: False.
    """
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    if deterministic:
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = False

if __name__ == '__main__':

    n = 100
    set_random_seed(1)

    seed = 1#int(torch.empty((), dtype=torch.int64).random_().item())
    generator = torch.Generator()

    # Function 'torch.Generator()' is not limited by the global seed
    # because there is another seed offset inside it to ensure that each epoch is different
    generator.manual_seed(seed)

    def simple_generator():
        # When the epoch is increasing, torch.randperm also generates a seed offset inside,
        # which can be controlled by the generator.
        # Note that it cannot be set the same for each epoch

        # When loading the weights, the model and the optimizer are loaded.
        # Besides, the sampled data in dataloader are guaranteed to be the same.
        # See Line: 36-39
        yield torch.randperm(n, generator=generator).tolist()


    state = generator.get_state()

    for i in range(3):
        my_gen = simple_generator()
        # It will be make sure each epoch is the same, but don't do this,
        # only when restoring the model to ensure that the sampled data are the same.
        # It should be considered in https://pytorch.org/docs/stable/notes/randomness.html
        print(generator.set_state(state))
        print(generator.get_state().sum())
        # print(torch.get_rng_state().sum())
        print(next(my_gen))

    # It is related to the instance, not the global.
    print(torch.Generator().get_state().sum())
```

cc @svekars @carljparker
SummaryWriter.add_embedding not working for RGBA images
Labels: triaged, enhancement, module: tensorboard

### 🐛 Describe the bug

The SummaryWriter.add_embedding does not work for images with transparency (RGBA), although tensorboard's embedding projector does support this.

```
import numpy as np
import torch
from torch.utils.tensorboard import SummaryWriter

writer = SummaryWriter('embedding_logs')

N_samples = 6
N_features = 24
N_channels = 3
dim_img =(128,128)

embeddings = np.random.random((N_samples,N_features))
imgs1 = np.random.random(size=(N_samples,N_channels,dim_img[0],dim_img[1]))
print(imgs1.shape) # (6, 3, 128, 128)

# Working
writer.add_embedding(
    embeddings, 
    label_img = torch.tensor(imgs1),
    global_step='RGB'
)

mask = np.random.randint(low=0, high=1, size=(N_samples,1,dim_img[0],dim_img[1]))
imgs2 = np.concatenate((imgs1, mask,), axis=1)
print(imgs2.shape) # (6, 4, 128, 128)

# Not Working
writer.add_embedding(
    embeddings, 
    label_img = torch.tensor(imgs2),
    global_step='RGBA'
)
```

Error Message:
```
Traceback (most recent call last):
  File "/Users/alex5753/Documents/Python/pytorch/test_embedding_projector.py", line 28, in <module>
    writer.add_embedding(
  File "/Users/alex5753/Documents/Python/pytorch/.env_torch/lib/python3.10/site-packages/torch/utils/tensorboard/writer.py", line 949, in add_embedding
    make_sprite(label_img, save_path)
  File "/Users/alex5753/Documents/Python/pytorch/.env_torch/lib/python3.10/site-packages/torch/utils/tensorboard/_embedding.py", line 45, in make_sprite
    arranged_img_CHW = make_grid(make_np(label_img), ncols=nrow)
  File "/Users/alex5753/Documents/Python/pytorch/.env_torch/lib/python3.10/site-packages/torch/utils/tensorboard/_utils.py", line 74, in make_grid
    assert I.ndim == 4 and I.shape[1] == 3
AssertionError
```

The error occur because the functionality in `torch/utils/tensorboard/_embedding.py, make_sprite()` and `torch/utils/tensorboard/_utils.py, make_grid` have been hard coded to only support three image channels (RGB). Can probably be fixed by reading the amount of channels from the image directly.


### Versions

Collecting environment information...
PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.4 (x86_64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.8 (v3.10.8:aaaf517424, Oct 11 2022, 10:14:40) [Clang 13.0.0 (clang-1300.0.29.30)] (64-bit runtime)
Python platform: macOS-13.4-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz

Versions of relevant libraries:
[pip3] numpy==1.25.0
[pip3] torch==2.0.1
[pip3] torchvision==0.15.2
[conda] Could not collect
[PT2.0][compile] torch._dynamo.config.log_level does not exist
Labels: module: logging, triaged, oncall: pt2

### 🐛 Describe the bug

setting torch._dynamo.config.log_level=logging.DEBUG causes error in nightly build 20230622
while this link mentiones that we can have it. https://pytorch.org/docs/master/compile/troubleshooting.html  

### Error logs

torch_executor.py:21: in <module>
    torch._dynamo.config.log_level=logging.DEBUG
/home/jthakur/.pt_2_0/lib/python3.8/site-packages/torch/_dynamo/config_utils.py:71: in __setattr__
    raise AttributeError(f"{self.__name__}.{name} does not exist")
E   AttributeError: torch._dynamo.config.log_level does not exist

### Minified repro

_No response_

### Versions

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] torch==2.1.0.dev20230621+cpu
[pip3] torchaudio==2.1.0.dev20230621+cpu
[pip3] torchvision==0.16.0.dev20230621+cpu
[pip3] triton==2.0.0
[conda] Could not collect

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
DISABLED test_backward_ddp_outside_uneven_inputs (__main__.TensorPipeDdpUnderDistAutogradTest)
Labels: oncall: distributed, module: flaky-tests, skipped

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_backward_ddp_outside_uneven_inputs&suite=TensorPipeDdpUnderDistAutogradTest) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/14449736644).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 2 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_backward_ddp_outside_uneven_inputs`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `distributed/rpc/test_tensorpipe_agent.py`

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Several Torchbench models don't run with float16 or bfloat16 in the inference eager mode
Labels: triaged, module: bfloat16, module: half, module: benchmark, oncall: pt2

Repro:
Apply the following patch,
```
diff --git a/benchmarks/dynamo/torchbench.py b/benchmarks/dynamo/torchbench.py
index 30406d5345d..6161f02690c 100755
--- a/benchmarks/dynamo/torchbench.py
+++ b/benchmarks/dynamo/torchbench.py
@@ -213,7 +213,7 @@ MAX_BATCH_SIZE_FOR_ACCURACY_CHECK = {
 FORCE_AMP_FOR_FP16_BF16_MODELS = {
     "doctr_det_predictor",
     "doctr_reco_predictor",
-    "Super_SloMo",
+    #"Super_SloMo",
     "tts_angular",
 }
```
and then run
```
benchmarks/dynamo/torchbench.py --inductor --bfloat16 --accuracy --inference --device cuda  --only Super_SloMo
```

Error:
```
Eager model failed to run
Traceback (most recent call last):
  File "/scratch/binbao/work/pytorch/benchmarks/dynamo/common.py", line 1320, in validate_model
    self.model_iter_fn(model, example_inputs)
  File "/scratch/binbao/work/pytorch/benchmarks/dynamo/torchbench.py", line 436, in forward_pass
    return mod(*inputs)
  File "/scratch/binbao/work/pytorch/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/scratch/binbao/work/pytorch/torch/nn/modules/module.py", line 1511, in _call_impl
    return forward_call(*args, **kwargs)
  File "/scratch/binbao/work/torchbench/torchbenchmark/models/Super_SloMo/model_wrapper.py", line 41, in forward
    g_I0_F_t_0 = self.trainFlowBackWarp(I0, F_t_0)
  File "/scratch/binbao/work/pytorch/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/scratch/binbao/work/pytorch/torch/nn/modules/module.py", line 1511, in _call_impl
    return forward_call(*args, **kwargs)
  File "/scratch/binbao/work/torchbench/torchbenchmark/models/Super_SloMo/slomo_model.py", line 286, in forward
    imgOut = torch.nn.functional.grid_sample(img, grid)
  File "/scratch/binbao/work/pytorch/torch/nn/functional.py", line 4295, in grid_sample
    return torch.grid_sampler(input, grid, mode_enum, padding_mode_enum, align_corners)
RuntimeError: "grid_sampler_2d_cuda" not implemented for 'BFloat16'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/scratch/binbao/work/pytorch/benchmarks/dynamo/common.py", line 2672, in run
    ) = runner.load_model(
  File "/scratch/binbao/work/pytorch/benchmarks/dynamo/torchbench.py", line 384, in load_model
    self.validate_model(model, example_inputs)
  File "/scratch/binbao/work/pytorch/benchmarks/dynamo/common.py", line 1322, in validate_model
    raise NotImplementedError("Eager model failed to run") from e
NotImplementedError: Eager model failed to run

WARNING:root:Super_SloMo failed to load
```

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
[dynamo] Update Unsupported to raise from fake tensor exceptions
Labels: Stale, module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #104360
* #103676
* __->__ #103987



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @anijain2305 @ipiszy
gfx906 ROCM print black images all ai torch: 2.0.1+rocm5.4.2/rocm5.5 only works with torch=1.13.0+rocm5.2
Labels: needs reproduction, module: rocm, triaged

### 🐛 Describe the bug

I am not gona typed here all things but I am gona summary before I gave link of I opened on other issues or discussions on github
I tried my gfx906 Radeon VII card with webui and invoke ai its working with torch==1.13.0+rocm5.2 but with torch==2.0.1+rocm5.4.2 I just got problem as black render. But it work with lots people but in my case I couldn't work it in my case.
I think this could be related this. Normal this printing in terminal with torch==1.13.0+rocm5.2 
MIOpen(HIP): Warning [SQLiteBase] Missing system database file: gfx906_60.kdb Performance may degrade. Please follow instructions to install: https://github.com/ROCmSoftwarePlatform/MIOpen#installing-miopen-kernels-package
But with new torch won't.

Here my history of this.
https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/9206
https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/10873

At this point I am stuck with out dated pytorch.
Also my card still in rocm support list but my card 
https://rocm.docs.amd.com/en/latest/release/gpu_os_support.html

Also Rocm version installed ROCm 5.5.1 in my Ubuntu 22.04.2 LTS

### Versions

python: 3.10.6 
working version:
pip install torch==1.13.0+rocm5.2 torchvision==0.14.0+rocm5.2 torchaudio==0.13.0 --extra-index-url https://download.pytorch.org/whl/rocm5.2
Not working
pip install torch==2.0.1+rocm5.4.2 torchvision==0.15.2+rocm5.4.2 --index-url https://download.pytorch.org/whl/rocm5.4.2
pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm5.5

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
[ONNX] Isolate TorchScript-based code-base from Dynamo-based ONNX exporter for easier deprecation
Labels: module: onnx, triaged, enhancement

### 🐛 Describe the bug

Currently, many files and helpers are being shared between both torchscript-based and dynamo-based ONNX exporter.

With future deprecation of TorchScript based exporter and the rise of Dynamo-based exporter in mind, we could start separating both exporter into different folders/files to make deprecation easy.

That should also make the code-base more organized and compact

### Versions

pytorch main branch
```[tasklist]
### Tasks
- [ ] https://github.com/pytorch/pytorch/pull/103942/
```
How to unwrap after auto_wrap in FSDP?
Labels: oncall: distributed, triaged, module: fsdp

I am currently fine-tuning a LLM (LLaMA) and would like to retrieve the gradients of each weight (parameter) after every gradient update. However, I notice that weights are (auto) wrapped into stuff like “_fsdp_wrapped_module._flat_param” during training. I need to map these wrapped weights to the original LLaMA architecture such as “self_attn.v_proj”. Any code examples?

I guess “summon_full_params()” might be the function that I look for, but I am not sure if that is correct. I also have difficulty using this function. Thanks a lot for any help!

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
CODEOWNERS file has errors due to non existent people being referred to
Labels: module: docs, triaged

If you have a question or would like help and support, please ask at our
[forums](https://discuss.pytorch.org/).

If you are submitting a feature request, please preface the title with [feature request].
If you are submitting a bug report, please fill in the following details.

## Issue description

https://github.com/pytorch/pytorch/blob/main/CODEOWNERS displays with errors
```

Unknown owner on line 25: make sure @z-a-f exists and has write access to the repository
…ten/src/ATen/native/ao_sparse @z-a-f @salilsdesai @kimishpatel @digantdesai @jianyuh
Unknown owner on line 26: make sure @z-a-f exists and has write access to the repository
…/native/quantized @jerryzh168 @z-a-f @salilsdesai @kimishpatel @digantdesai @jianyuh
Unknown owner on line 27: make sure @z-a-f exists and has write access to the repository
…ive/quantized/cpu @jerryzh168 @z-a-f @salilsdesai @kimishpatel @digantdesai @jianyuh
Unknown owner on line 31: make sure @z-a-f exists and has write access to the repository
/test/ao/ @jerryzh168 @z-a-f @hdcharles
Unknown owner on line 32: make sure @z-a-f exists and has write access to the repository
…est/quantization/ @jerryzh168 @z-a-f
Unknown owner on line 34: make sure @z-a-f exists and has write access to the repository
ao/sparisty/ @z-a-f @hdcharles
Unknown owner on line 38: make sure @z-a-f exists and has write access to the repository
nn/quantizable/ @jerryzh168 @z-a-f
Unknown owner on line 114: make sure @robieta exists and has write access to the repository
torch/csrc/autograd/profiler* @robieta
Unknown owner on line 115: make sure @robieta exists and has write access to the repository
torch/autograd/profiler* @robieta
Unknown owner on line 116: make sure @robieta exists and has write access to the repository
torch/csrc/profiler/ @robieta
Unknown owner on line 117: make sure @robieta exists and has write access to the repository
torch/profiler/ @robieta
Unknown owner on line 123: make sure @NivekT exists and has write access to the repository
torch/utils/data/ @NivekT @ejguan
```
This should be reviewed to either remove users or ensure that they exist.

I can submit a change to remove the reference to users in this file, if that is the right approach for all of them..  



cc @svekars @carljparker
Need the full "Release Compatibility Matrix" of torch
Labels: oncall: binaries, oncall: releng, triaged

### 📚 The doc issue

Hi, We always tries some old project which may lead us install old version of pytorch. But there is no full-version compatibility matrix of pytorch.  It will be about the compatibility limitations between python version, pytorch version, and CUDA.
NOT THAT  https://github.com/pytorch/pytorch/wiki/PyTorch-Versions
BUT LIKE THAT  https://github.com/pytorch/pytorch/blob/main/RELEASE.md  


### Suggest a potential alternative/fix

_No response_

cc @seemethere @malfet
How to modify gradients of an FSDP model?
Labels: oncall: distributed, module: fsdp

### 📚 The doc issue

I've initially posted the question on [forum](https://discuss.pytorch.org/t/modify-gradients-of-an-fsdp-model/182159) 7 days ago, but crossposting here as well for better visibility since I couldn't get any answers there. 

Hi everyone,
I have an FSDP model which has zeros in some of the `torch.nn.Linear.weight` parameters. During the training I would like to keep those parameters fixed to zeros, and to zero-out their gradients during backward as well. The specific use-case is: I am loading a pruned model and I want to fine-tune it with FSDP while keeping the pruning mask fixed. 

To achieve this I need to do two things: 
1) multiply parameters with the mask before the forward pass (so that all pruned weights remain pruned), 
2) multiply gradients of pruned parameters after the backward pass (so that gradients of pruned weights are zeros)
 
In the standard DDP training I would achieve this by:
1) registering forward pre-hook on `torch.nn.Linear` modules and multiplying weights with the mask before each forward pass,
2) registering a hook on the parameter `torch.nn.Linear.weight` and multiplying its gradient with the mask. 

For example:
```python
def keep_param_pruned(mask, module, input):
    with torch.no_grad():
        module.weight.data.mul_(mask.to(module.weight.device))

def keep_grad_pruned(mask, grad):
    return grad.mul_(mask.to(grad.device))

for n, m in model.named_modules():
    if isinstance(m, torch.nn.Linear):
        mask = m.weight > threshold
        m.register_forward_pre_hook(partial(keep_param_pruned, mask))
        m.weight.register_hook(partial(keep_grad_pruned, mask))
```

However, I am struggling to modify this idea to work with FSDP.  Any suggestions/ideas on what I am doing wrong or if there is a simpler way to achieve this without playing with hooks?

### Suggest a potential alternative/fix

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu