gopikrsmscs/torch-issues huggingface dataset:
Illegal hardware instruction following Real Time Inference on Raspberry Pi 4 tutorial
Labels: oncall: quantization, triaged, module: arm

### 🐛 Describe the bug

Following the [tutorial](https://pytorch.org/tutorials/intermediate/realtime_rpi.html) the program stops with 

`[1]    182488 illegal hardware instruction (core dumped)`

See minimal reproduction code below and output:

```
import torch
from torchvision import models

torch.backends.quantized.engine = "qnnpack"

print("This prints")
net = models.quantization.mobilenet_v2( pretrained=True, quantize=True)  # Illegal instruction occurs here
print("This doens't print")
net = torch.jit.script(net)
```

The code runs fine with torch 1.10.2

Output:
```
$ python torchy.py                                                                                                                   

/home/max/.pyenv/versions/pytorch/lib/python3.9/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 
  warn(f"Failed to load image Python extension: {e}")
A
/home/max/.pyenv/versions/pytorch/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/home/max/.pyenv/versions/pytorch/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=MobileNet_V2_QuantizedWeights.IMAGENET1K_QNNPACK_V1`. You can also use `weights=MobileNet_V2_QuantizedWeights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
[1]    182488 illegal hardware instruction (core dumped)  python torchy.py

```

### Versions

PyTorch version: 1.13.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (aarch64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.9.15 (main, Dec  7 2022, 13:19:38)  [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-1012-raspi-aarch64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==1.13.0
[pip3] torchaudio==0.13.0
[pip3] torchvision==0.14.0
[conda] Could not collect

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel @malfet
Illegal hardware instruction using torch.nn.Conv2d on aarch64 (Raspberry Pi 4)
Labels: module: crash, triaged

### 🐛 Describe the bug

The following code returns an 
`[1]    179220 illegal hardware instruction (core dumped)` error when executed on a Rapsberry Pi (Python 3.9,  torch 1.13.0). 
It runs successfully on torch 1.10.2

This is a minimal reproduction setup. 

```
#! /usr/bin/python
import torch

m = torch.nn.Conv2d(16, 33, 3, stride=2)
input = torch.randn(20, 16, 50, 100)
output = m(input) # Illegal instruction happens here
```

$ python repro.py
                                                                                                                    [1]    181810 illegal hardware instruction (core dumped)  python repro.py

### Versions

PyTorch version: 1.13.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (aarch64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.9.15 (main, Dec  7 2022, 13:19:38)  [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-1012-raspi-aarch64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==1.13.0
[pip3] torchaudio==0.13.0
[pip3] torchvision==0.14.0
[conda] Could not collect
Can torchrun have a shell completion?
Labels: oncall: distributed

### 🚀 The feature, motivation and pitch

Like

- <https://github.com/andreafrancia/trash-cli/blob/master/setup.cfg#L31-L33>
- <https://github.com/inducer/pudb/blob/main/setup.py#L24>
- <https://github.com/piccolomo/plotext/blob/master/setup.py#L25>
- <https://github.com/wookayin/gpustat/blob/master/setup.py#L116>

```
❯ torchrun -<TAB>
option
--help               show this help message and exit
-h                   show this help message and exit
--log_dir            Base directory to use for log files (e.g. /var/log/torch/elastic). The same directory is re-used for multiple runs (a unique job-level sub-directory is created with rdzv_id as the prefix).
--master_addr        Address of the master node (rank 0). It should be either the IP address or the hostname of rank 0. For single node multi-proc training the --master_addr can simply be 127.0.0.1; IPv6 should have the pattern `[0:0:0:0:0:0:0:1]`.
--master_port        Port on the master node (rank 0) to be used for communication during distributed training.
--max_restarts       Maximum number of worker group restarts before failing.
-m                   Change each process to interpret the launch script as a Python module, executing with the same behavior as 'python -m'.
--module             Change each process to interpret the launch script as a Python module, executing with the same behavior as 'python -m'.
--monitor_interval   Interval, in seconds, to monitor the state of workers.
--nnodes             Number of nodes, or the range of nodes in form <minimum_nodes>:<maximum_nodes>.
--node_rank          Rank of the node for multi-node distributed training.
...
```

### Alternatives

```
shtab -s zsh torch.distributed.run.get_args_parser --prog torchrun| sudo tee /usr/share/zsh/site-functions/_torchrun
```

See <https://docs.iterative.ai/shtab>

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Functionalization on inplace_views should properly reflect autograd metadata
Labels: triaged, module: functionalization, oncall: pt2

There's a bug in AOT Autograd today:
```
from functorch.compile import aot_function, nop

def f(x):
    out = x * 2
    out.unsqueeze_(0)
    return out

# works
f(torch.ones(2, requires_grad=True)).mul_(2)

compiled_f = aot_function(f, fw_compiler=nop)

# Fails
compiled_f(torch.ones(2, requires_grad=True)).mul_(2)
# prints:
RuntimeError: Output 0 of CompiledFunctionBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function.
```

The problem is that:

(1) In the eager code, `out` is not a (differentiable) view. `out._is_view() == False`

(2) In the compiled code, functionalization runs and converts the program into:
```
def f_functionalized(x):
    out = x * 2
    out_updated = x.unsqueeze(0)
    # out_updated is a view! out_updated._is_view() == True
    return out_updated
```

When we return an output that has `._is_view()` from an `autograd.Function.forward()`, we are not allowed to mutate it.

From talking with @ezyang, a reasonable fix would be inside of functionalization: functionalization should respect the autograd view metadata when handling inplace_view ops. One way to do that is:

(1) For every inplace view op (e.g. `transpose_()`), and a corresponding `_unsafe_transpose()` op , that autograd doesn't know is an aliasing operator

(2) When functionalization sees `x.transpose_()`, it should decide to dispatch into either `.transpose()` or `._unsafe_tranpose()`, depending on whether `x` is currently a differentiable view.

cc @ezyang @soumith @msaroufim @wconstab @ngimel
Tensor indexing and slicing documentation should explicitly state that indexing follows numpy semantics and link to the numpy indexing documentation.
Labels: module: docs, triaged, module: numpy

### 📚 The doc issue

It's my understanding that indexing a tensor with `[]`/`Tensor.__getitem__`/`Tensor.__setitem__` works like numpy, but this is not mentioned in the [Indexing, Slicing, Joining, Mutating Ops](https://pytorch.org/docs/stable/torch.html#indexing-slicing-joining) docs.


Related discussion: https://discuss.pytorch.org/t/selecting-from-a-2d-tensor-with-rows-of-column-indexes/167717

### Suggest a potential alternative/fix

Add to the [tensor indexing](https://pytorch.org/docs/stable/torch.html#indexing-slicing-joining) docs that this works like numpy, and link to the numpy indexing docs. Maybe https://numpy.org/doc/stable/user/basics.indexing.html or https://numpy.org/doc/stable/user/basics.indexing.html#advanced-indexing. 

cc @svekars @carljparker @mruberry @rgommers
Dynamo and cond with free variables creates malformed graph
Labels: triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

repro

```
diff --git a/test/dynamo/test_export.py b/test/dynamo/test_export.py
index b0640f6511..8fb8754c51 100644
--- a/test/dynamo/test_export.py
+++ b/test/dynamo/test_export.py
@@ -1443,8 +1443,10 @@ class ExportTests(torch._dynamo.test_case.TestCase):
                 self.linear = torch.nn.Linear(3, 3)
 
             def forward(self, pred, x):
+                y = x * 2
+
                 def true_fn(val):
-                    return self.linear(val) * torch.tensor(2)
+                    return self.linear(val) * torch.tensor(2) * y
 
                 def false_fn(val):
                     return self.linear(val) * torch.tensor(-1)
```

the true graph ends up being

```
def forward(self, x):
    self_linear = self.self_linear(x);  x = None
    tensor = torch.tensor(2)
    mul = self_linear * tensor;  self_linear = tensor = None
    mul_1 = mul * mul;  mul = mul = None
    return mul_1
```

which is so bad

### Versions

master

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh @mlazos @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
Saving a scripted module to a buffer does not work.
Labels: oncall: jit

### 🐛 Describe the bug

A scripted pytorch module can not be saved to a buffer. It will instead save the module to a file with the name of the buffer-object.
According to the [documentation](https://pytorch.org/docs/stable/generated/torch.jit.ScriptModule.html#torch.jit.ScriptModule.save), the following code should have saved the model to the buffer. Instead the model is written to a file with the name of the buffer's __str__ representation. The buffer remains empty.
I expected the buffer to not be empty and that no file would be made on the filesystem.

```import torch
import io

class Model(torch.nn.Module):
    def forward(self, x): 
        return x+1 

model = Model().cuda() 
traced = torch.jit.script(model, torch.ones(1, 3, 128, 128, device='cuda'))

with io.BytesIO() as buf:
    traced.save(buf)
```

### Versions



```
Collecting environment information...
PyTorch version: 1.13.0
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.15 (main, Nov 24 2022, 14:31:59)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-53-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 520.61.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.4
[pip3] torch==1.13.0
[pip3] torchaudio==0.13.0
[pip3] torchvision==0.14.0
[conda] blas                      1.0                         mkl  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.23.4           py39h14f4228_0  
[conda] numpy-base                1.23.4           py39h31eccc5_0  
[conda] pytorch                   1.13.0          py3.9_cuda11.6_cudnn8.3.2_0    pytorch
[conda] pytorch-cuda              11.6                 h867d48c_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.13.0               py39_cu116    pytorch
[conda] torchvision               0.14.0               py39_cu116    pytorch
```

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Torch 1.13 Onnx Scope name not correct!
Labels: module: onnx, triaged

### 🐛 Describe the bug

In torch 1.13.0, Onnx Scope name was introduced which replaces the old Onnx node format (`Conv_0`, `Relu_1`, ...) into a new format that captures the pytorch submodule name. But the actual onnx node name may not always correspond to a submodule and instead needs some further post-processing.

Example:
```python
import io
import onnx
import torch
from torchvision.models import resnet18

buffer = io.BytesIO()
torch.onnx.export(resnet18(), torch.randn(1, 3, 224, 224), buffer)
buffer.seek(0, 0)
onnx_model = onnx.load(buffer)
for node in onnx_model.graph.node:
    print(node.name)
```

Output:
```
...
/conv1/Conv
/relu/Relu
/maxpool/MaxPool
/layer1/layer1.0/conv1/Conv
/layer1/layer1.0/relu/Relu
/layer1/layer1.0/conv2/Conv
...
/layer4/layer4.1/relu_1/Relu
/avgpool/GlobalAveragePool
/Flatten
/fc/Gemm
```

As we can see here for `ResNet18`, there is no submodule Conv named `layer1.layer1.0.conv1` in the model but rather `layer1.0.conv1`. So, its onnx node name should be `/layer1/0/conv1/Conv`.

Sidenote: Ideally it would have been more helpful to have the submodule name as the onnx node name i.e. `layer1.0.conv1` instead of `/layer1/layer1.0/conv1/Conv`.

### Versions

```
Collecting environment information...
PyTorch version: 1.13.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.0.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.202)
CMake version: version 3.22.1
Libc version: N/A

Python version: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:14)  [Clang 12.0.1 ] (64-bit runtime)
Python platform: macOS-13.0.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.971
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.1
[pip3] pytorch-sphinx-theme==0.0.19
[pip3] torch==1.13.0
[pip3] torchaudio==0.13.0.dev20221003
[pip3] torchpack==0.3.1
[pip3] torchprofile==0.0.4
[pip3] torchvision==0.14.0
[conda] cpuonly                   2.0                           0    pytorch-nightly
[conda] numpy                     1.23.1           py38h42add53_0  
[conda] numpy-base                1.23.1           py38hadd41eb_0  
[conda] pytorch-mutex             1.0                         cpu    pytorch-nightly
[conda] pytorch-sphinx-theme      0.0.19                   pypi_0    pypi
[conda] torch                     1.13.0                   pypi_0    pypi
[conda] torchaudio                0.13.0.dev20221003        py38_cpu    pytorch-nightly
[conda] torchpack                 0.3.1                    pypi_0    pypi
[conda] torchprofile              0.0.4                    pypi_0    pypi
[conda] torchvision               0.14.0                   pypi_0    pypi
```
A few functions in fbgemm_utils.cpp are defined in global namespace
Labels: module: cpp, oncall: quantization, triaged

### 🐛 Describe the bug

This file defines a few functions in the global namespace, e.g. these:

https://github.com/pytorch/pytorch/blob/e0f681aa85e4466a4096fcb8c3465ed662820760/aten/src/ATen/native/quantized/cpu/fbgemm_utils.cpp#L29-L30

And more in the same file.

That looks wrong.

### Versions

latest on github

cc @jbschlosser @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel
IValue(c10::List<IValue>) constructor is confusing and undocumented
Labels: module: internals, triaged

### 📚 The doc issue

Constructing `IValue` from `std::vector<IValue>` is not possible at the moment, or otherwise esoteric and undocumented.

For collections of tensors, one can simply create a vector<Tensor> and pass them in to an overloaded IValue constructor. For instance, assuming `data` is a C-style Tensor array whose length is described by `num_datums`, one can write:

```
std::vector<torch::Tensor> inputs;
for (int i = 0; i < num_datums; i++) inputs.push_back(*data[i]);
*output = new torch::IValue(inputs);
```

My expectation is that the same would be true for vector<IValue>, i.e., assuming data is now a C-style IValue array, I would expect to be able to write:

```
std::vector<torch::IValue> inputs;
for (int i = 0; i < num_datums; i++) inputs.push_back(*data[i]);
*output = new torch::IValue(inputs);
```

However, this results in a compilation error:

```
static_assert(!std::is_same<T, IValue>::value, "This constructor is not valid for List<IValue>. Please use c10::impl::GenericList(elementType) instead.");
```

It is likewise impossible to replace vector with any form of `c10::List` or `c10::GenericList` for the exact same reason. A basic stack allocation of either of those structures, such as

```
c10::impl::GenericList inputs;
```

yields the exact same compilation error.

What is the expected way to create an IValue with a list of other IValue instances?

### Suggest a potential alternative/fix

_No response_

cc @ezyang @bhosmer @smessmer @ljk53 @bdhirsh
FX graph mode quant: backendconfig configuration missing for torch.nn.GRU
Labels: oncall: quantization, triaged

### 🐛 Describe the bug

`torch.nn.GRU` should be in https://pytorch.org/docs/1.13/quantization-backend-configuration.html, for dynamic quantization but it is not.  We should add it.  Note: other recurrent layers are already there.

This was reported in https://discuss.pytorch.org/t/quantization-not-decreasing-model-size-static-and-qat/87319/11

### Versions

master

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @jgong5 @Xia-Weiwen @leslie-fang-intel
torch.utils.tensorboard import fails if a new protobuf > 3.20 is installed (bug in tensorboard/tensorflow but better guard against it)
Labels: oncall: visualization

### 🐛 Describe the bug

With a new version of protobuf 4.21.8, `import torch.utils.tensorboard` unconditionally imports `tensorboard.compat` / `tensorboard.compat.event_pb2`, etc which were generated with an old `protoc` (and friends which isn't happy given). Pytorch version 1.10 is quite antique, but probably the tensorboard writer code hasn't changed in a while.

This also affects pytorch lightning and all other dependent packages, such as nemo.

Sorry for an image stack trace  - a friend stumbled on this bug, not myself.

I'd prefer that `import torch.utils.tensorboard` succeeded, but only creating `SummaryWriter` instance would fail in case of dependency issues (with protobuf / tensorboard etc). Also, it might be good if it just worked with newer protobuf, maybe a non-compat tensorboard functionality could be used or explicitly some good protobuf version could be checked.

![photo_2022-12-07_14-27-00](https://user-images.githubusercontent.com/1041752/206147822-96eff842-cbe7-41a5-b67c-0e774c8d5bac.jpg)

### Versions

pytorch-lightning                 1.6.3
torch                             1.10.1
protobuf 4.21.8
tensorboard                       2.8.0
"Reached a code path in Module.get_extra_state() that should never be called."
Labels: needs reproduction, oncall: jit, triaged

### 🐛 Describe the bug

I got the run time error mentioned in the title when calling `get_extra_state` on a `RecursiveScriptModule` created from an `nn.Dropout` module.
I was just playing around so don't know if it even makes sense, but since the error specifically told me to report it I thought I should.
If nothing else, this error should probably be changed.

Image of entire error stack: 
![image](https://user-images.githubusercontent.com/20074572/206148420-032e8035-2921-4839-bb86-ee58b1419c73.png)



### Versions

Collecting environment information...
PyTorch version: 1.13.0
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro
GCC version: (tdm64-1) 9.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.6 | packaged by conda-forge | (main, Oct 24 2022, 16:02:16) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19044-SP0
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2060
Nvidia driver version: 516.94
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin\cudnn_ops_train64_8.dll
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.3
[pip3] torch==1.13.0
[pip3] torchaudio==0.13.0
[pip3] torchvision==0.14.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2021.4.0           haa95532_640
[conda] mkl-service               2.4.0           py310h2bbff1b_0
[conda] mkl_fft                   1.3.1           py310ha0764ea_0
[conda] mkl_random                1.2.2           py310h4ed8f06_0
[conda] numpy                     1.23.3          py310h60c9a35_0
[conda] numpy-base                1.23.3          py310h04254f7_0
[conda] pytorch                   1.13.0          py3.10_cuda11.7_cudnn8_0    pytorch
[conda] pytorch-cuda              11.7                 h67b0de4_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.13.0                   pypi_0    pypi
[conda] torchvision               0.14.0                   pypi_0    pypi

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
Get the error: AttributeError: Can't pickle local object 'convert_frame.<locals>._convert_frame'
Labels: triaged, bug, oncall: pt2, module: dynamo

### 🐛 Describe the bug

When adding the line:
`model = torch.compile(model)` after loading the model, this error occurs. When removing the line, the script functions as intended. 

### Error logs

  File "/opt/anaconda3/envs/ml1/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/anaconda3/envs/ml1/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/opt/anaconda3/envs/ml1/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/opt/anaconda3/envs/ml1/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/anaconda3/envs/ml1/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/anaconda3/envs/ml1/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/opt/anaconda3/envs/ml1/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'convert_frame.<locals>._convert_frame'


### Minified repro

_No response_

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @soumith @ngimel
[JIT] INTERNAL ASSERT FAILED `torch.add` with boolean primitive constant
Labels: oncall: jit

### 🐛 Describe the bug

```python
import torch

def fn(x):  # Error ~ bool primitive
    return torch.add(x, 1)

traced = torch.jit.trace(fn, (torch.zeros((), dtype=torch.int),))

def fn(x):  # Error ~ bool primitive
    return torch.add(x, False)

# eager: OK
torch.add(torch.zeros((), dtype=torch.bool), True)
# jit trace
traced = torch.jit.trace(fn, (torch.zeros((), dtype=torch.bool),))
```

Similar to https://github.com/pytorch/pytorch/issues/90366 but for `torch.add`.

### Versions

<details><summary><b>Env </b> <i>[click to expand]</i></summary>
<div>

```python
"""
Collecting environment information...
PyTorch version: 1.14.0.dev20221202+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: version 3.25.0
Libc version: glibc-2.35

Python version: 3.9.12 (main, Apr  5 2022, 06:56:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 515.86.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.3
[pip3] onnx2torch==1.5.3
[pip3] torch==1.14.0.dev20221202+cu117
[pip3] torchaudio==0.14.0.dev20221203+cu117
[pip3] torchtriton==2.0.0+0d7e753227
[pip3] torchvision==0.15.0.dev20221203+cpu
[conda] numpy                     1.23.3                   pypi_0    pypi
[conda] onnx2torch                1.5.3                    pypi_0    pypi
[conda] torch                     1.14.0.dev20221202+cu117          pypi_0    pypi
[conda] torchaudio                0.14.0.dev20221203+cu117          pypi_0    pypi
[conda] torchtriton               2.0.0+0d7e753227          pypi_0    pypi
[conda] torchvision               0.15.0.dev20221203+cpu          pypi_0    pypi
"""
```

</div>
</details>

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
[ONNX] test_mask_rcnn in test_models_onnxruntime.py failed with ONNX version==1.13.0
Labels: module: onnx, triaged, onnx-triaged

### 🐛 Describe the bug

In https://github.com/pytorch/pytorch/pull/90337, the test failed when ONNX version bumped to 1.13.0.

### Versions

Nightly PyTorch
[RFC] Allow FSDP mixed precision for only certain type of submodules
Labels: oncall: distributed, module: fsdp

### 🚀 The feature, motivation and pitch

Mixed Precision frequently runs into unsupported operator for the lower precision issue, or just worse / degraded accuracy if some operators / modules are run in lower precision.

For FSDP native mixed precision, if users are aware of these operators and the modules that they belong to, it could be useful to simply exclude these modules from mixed precision and still get the performance enhancements by having rest of the module in mixed precision. 

One useful API might be to ignore some modules for mixed precision, or the reverse direction, specify explicitly which modules to be mixed precision.

For the latter, we could add a `modules_for_mixed_precision` arg to `MixedPrecision` configuration, if this is non-None, it specifies the types of modules we should apply mixed precision on. 

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
[Tracking Issue] Mixed precision does not work with ignored modules 
Labels: oncall: distributed, module: fsdp

### 🐛 Describe the bug

This is just a tracking issue, the root cause and repro is given in https://github.com/pytorch/pytorch/pull/89971

### Versions

main

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Inconsistent Hash of IValue between aten/src/ATen/core/ivalue.cpp and aten/src/ATen/core/Dict_inl.h
Labels: oncall: jit

### 🐛 Describe the bug

Recently, I'm using libtorch 1.13.0 using C++ in VS 2022. I want to create a torch::Dict, which maps tuple<string, string, string> to tuple<string, string, string>. After I create the tuple key, I use torch::IValue::hash(key) and this works. However, when I want to insert (key, value) into dict, the program raises error in DictKeyHash::operator()(const IValue& ivalue) (the function is in aten/src/ATen/core/Dict_inl.h). It says can't hash IValues. 

The reason is that DictKeyHash::operator() only supports to hash int, string, double, complex double, bool, tensor and device. However, the funciton torch::IValue::hash in aten/src/ATen/core/ivalue.cpp supports None, Bool, Double, Tensor, Storage, Int, SymInt, SymFloat, String, Tuple, and Device. **There is a conflict between two hash function. The first one cannot hash None, Storage, SymInt, SymFloat, Tuple, and the second one cannot hash complex double.**

Is it a bug? Can these hash functions be consistent? Thanks!

Here's my code.

`#include <iostream>
#include <torch/torch.h>

int main() {
    torch::TupleTypePtr pt = torch::TupleType::create({torch::StringType::get(), torch::StringType::get(), torch::StringType::get()});
    std::cout << pt->str() << "\n";

    c10::Dict<torch::IValue, torch::IValue> dict(pt, pt);
    torch::IValue key(std::make_tuple("one", "one", "one"));
    std::cout << torch::IValue::hash(key) << "\n";
    dict.insert(key, key); // <- will fail in DictKeyHash::operator()
    return 0;
}
`

### Versions

Windows 10, Visual Studio 2022, C++, libtorch 1.13.0 with cuda

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel @jbschlosser
Unknown buildin op: aten::pad
Labels: oncall: jit

### 🐛 Describe the bug

I'm loading a torchscript EffiecientNet model from Monai into c++ via:

`	auto m = torch::jit::load(datapath + "enet-b0_.pt");
`

Here is the text from the error that gets thrown:

Unknown builtin op: aten::pad. Here are some suggestions:  aten::svd aten::cat aten::prod aten::put aten::ord aten::grad aten::pop aten::gcd aten::var aten::std aten::add aten::max aten::... ...}

The libtorch code is running under Windows 10, built with VS2019, version 1.10.2+cu113.

I checked the current version of op_registry.cpp, but don't see aten::pad listed, so I'm guessing getting a new build won't help.


### Versions

Collecting environment information...
PyTorch version: 1.13.0
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Microsoft Windows Server 2019 Standard
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.8.8 (default, Apr 13 2021, 15:08:03) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.17763-SP0
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: Tesla V100S-PCIE-32GB
GPU 1: Tesla V100S-PCIE-32GB
GPU 2: Tesla V100S-PCIE-32GB
GPU 3: Tesla V100S-PCIE-32GB

Nvidia driver version: 516.94
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\bin\cudnn_ops_train64_8.dll
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.20.1
[pip3] numpydoc==1.1.0
[pip3] torch==1.13.0
[pip3] torchaudio==0.13.0
[pip3] torchinfo==1.7.1
[pip3] torchvision==0.14.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2021.2.0           haa95532_296
[conda] mkl-service               2.3.0            py38h2bbff1b_1
[conda] mkl_fft                   1.3.0            py38h277e83a_2
[conda] mkl_random                1.2.1            py38hf11a4ad_2
[conda] numpy                     1.20.1           py38h34a8a5c_0
[conda] numpy-base                1.20.1           py38haf7ebc8_0
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1
[conda] pytorch                   1.13.0          py3.8_cuda11.6_cudnn8_0    pytorch
[conda] pytorch-cuda              11.6                 h867d48c_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.13.0                   pypi_0    pypi
[conda] torchinfo                 1.7.1              pyhd8ed1ab_0    conda-forge
[conda] torchvision               0.14.0                   pypi_0    pypi

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
torch._dynamo.exc.Unsupported: dynamic shapes: arange
Labels: triaged, bug, oncall: pt2

### 🐛 Describe the bug

While giving a try with pytorch2 on OpenNMT-py

using these two lines:
```
    rawmodel = build_model(model_opt, opt, vocabs, checkpoint)
    model = torch.compile(rawmodel, fullgraph=True, backend='nvprims_aten')
```

### Error logs

getting this:
```
from user code:
   File "/home/vincent/nlp/OpenNMT-py/onmt/encoders/transformer.py", line 126, in forward
    mask = ~sequence_mask(src_len).unsqueeze(1)
  File "/home/vincent/nlp/OpenNMT-py/onmt/utils/misc.py", line 58, in sequence_mask
    return (torch.arange(0, max_len, device=lengths.device)

Set torch._dynamo.config.verbose=True for more information


You can suppress this exception and fall back to eager by setting:
    torch._dynamo.config.suppress_errors = True
```

### Minified repro

_No response_

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
quantization qconfig: can we set per-channel quant as default for qnnpack?
Labels: oncall: quantization, triaged

### 🐛 Describe the bug

The current default qconfig for qnnpack is per-tensor quantization.  Can we update the default qnnpack qconfig to per-channel quantization?  I heard that per-channel has been supported for awhile in the qnnpack kernels - is that still true?  Is there anything blocking us from upgrading the default?

Motivation: PTQ accuracy on MobileNetV3 + ImageNet 1k is:
* 0.75282 fp32 baseline
* 0.40968 with qnnpack default (per-tensor quant)
* 0.6747 with qnnpack default but per-channel quant enabled

### Versions

master

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @jgong5 @Xia-Weiwen @leslie-fang-intel
quantization observers: can we relax the default epsilon value?
Labels: oncall: quantization, low priority, triaged

### 🐛 Describe the bug

In `https://github.com/pytorch/pytorch/blob/master/torch/ao/quantization/observer.py#L208`, the epsilon value used to determine the uniform quantization scale is defined as

```
eps=torch.finfo(torch.float32).eps
```

This resolves to 1.19e-7 for float32, and the value from `finfo` is chosen so that he smallest representable number such that 1.0 + eps != 1.0.  This definition is generic might not apply well to choosing the quantization scale, I would expect the smallest uniform quantization scale value to be determined by restrictions of the underlying backend.  For example, for XNNPACK the recommended epsilon value is 2e-12.

On the pretrained weights of MobileNetV3 on ImageNet-1K from torchvision, the weights of the `features.4.block.2.fc1` and `features.4.block.2.fc2` layers are too small in magnitude for PTQ to work correctly with the current default scale value, leading to poor quantization accuracy of these layers in PTQ.

We should consider epsilon in a smarter way.  I would expect the overall impact to be low as this is an edge case, but it will be a free accuracy win for PTQ for that edge case.

### Versions

master

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @jgong5 @Xia-Weiwen @leslie-fang-intel
cannot backward()
Labels: needs reproduction, triaged, module: mps

### 🐛 Describe the bug

loc("total derivative last state"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/810eba08-405a-11ed-86e9-6af958a02716/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":228:0)): error: input types 'tensor<1x61x64xf32>' and 'tensor<1x32x64xf32>' are not broadcast compatible
LLVM ERROR: Failed to infer result type(s).

when I try to train model on maps which contains nn.LSTM  block.   when backward().    it crashed.
however,  cpu is ok


### Versions

pytorch  1.13

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
make_fx loses node.stack_trace / turn on AOTAutograd by default for all backends
Labels: high priority, triaged, oncall: pt2

### 🐛 Describe the bug

It appears that make_fx loses the node.stack_trace on the relevant nodes. This makes it difficult to integrate into a production compiler since it makes debugging of the resulting model much more difficult. It would be nice if make_fx preserved the node.stack_trace.

```
import torch
import torch._dynamo as dynamo
import functorch

from typing import List

def my_backend(gm: torch.fx.GraphModule,
               example_inputs: List[torch.Tensor]):
    gm.print_readable()
    make_fx_gm = functorch.make_fx(gm)(*example_inputs)
    make_fx_gm.print_readable()
    return make_fx_gm

@dynamo.optimize(my_backend)
def f(x):
    return x * x

f(torch.rand(3))
```

Output:

```
# The graph captured by TorchDynamo
class GraphModule(torch.nn.Module):
    def forward(self, x : torch.Tensor):
        # File: /tmp/repro2.py:16, code: return x * x
        mul = x * x;  x = None
        return (mul,)

# The result of make_fx
class <lambda>(torch.nn.Module):
    def forward(self, arg0_1: f32[3]):
        # No stacktrace found for following nodes <------------ missing node.stack_trace
        mul: f32[3] = torch.ops.aten.mul.Tensor(arg0_1, arg0_1);  arg0_1 = None
        return (mul,)
        
```

### Versions

Collecting environment information...
PyTorch version: 1.14.0.dev20221201+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux rodete (x86_64)
GCC version: (Debian 12.2.0-3) 12.2.0
Clang version: 14.0.6-2
CMake version: version 3.25.0
Libc version: glibc-2.35

Python version: 3.10.8 (main, Nov  3 2022, 15:17:13) [GCC 12.2.0] (64-bit runtime)
Python platform: Linux-5.19.11-1rodete1-amd64-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.24.0rc1
[pip3] torch==1.14.0.dev20221201+cpu
[pip3] torchvision==0.15.0.dev20221201+cpu
[conda] Could not collect

cc @ezyang @gchanan @zou3519 @soumith @msaroufim @wconstab @ngimel @bdhirsh
Is it possible to add a parameter in torch.onnx.export to skip the prim::PythonOp subgraph process when exporting the autograd function?
Labels: module: onnx, triaged

### 🚀 The feature, motivation and pitch

Torch 1.13 adds a new feature: Inlined prim::PythonOp for Autograd Function Export (https://github.com/pytorch/pytorch/pull/74765). When tracing, all operations in the autograd function will be recorded as a subgraph of autograd function node. However, this subgraph seems to be useless if a custom symbolic function is implemented in autograd function defination or registered by register_custom_symbolic_op, in which cases the autograd function will be exported as one single node.
In my cases, some operations in my custom autograd functions may also be incompatible with this subgraph process or the following passes. Is it possible that a parameter is added in torch.onnx.export interface to control this subgraph trace or just skip subgraph tracing when a symbolic function is registered by register_custom_op_symbolic or implemented in the autograd function defination?

### Alternatives

_No response_

### Additional context

_No response_
Why torch.mode return different value between CPU and GPU
Labels: module: cuda, triaged

### 🐛 Describe the bug

For this case,
```
tensor_input = torch.Tensor([[[[-0.6572, -0.0377,  0.2676, -0.2568, -1.6279, -0.3259, -0.1349,
           -0.6699,  1.0273,  0.0203, -0.7080,  0.9360,  0.8535,  0.0132,
            1.2920, -0.4414, -0.5073, -0.5352,  0.2313,  0.1196, -0.7681,
           -0.9087, -0.4175, -0.0583, -1.1299,  1.5000,  0.0756,  0.4622,
           -0.5273,  1.7432, -0.8896,  1.7295, -0.7310, -0.7080, -0.0253,
            0.7202,  2.2656,  1.2324,  1.0000,  0.8584, -3.2207,  0.0425,
           -1.3242, -0.0217,  0.2297, -0.3833, -0.0539,  1.2920, -0.6719,
            0.3425,  1.4785,  0.6108,  0.5913, -1.3027, -1.0791]]]])
value_gpu, index_gpu = torch.mode(tensor_input.cuda(), -1, False)
value_cpu, index_cpu = torch.mode(tensor_input.cpu(), -1, False)
print("value_gpu is:  ", value_gpu)
print("value_cpu is: ", value_cpu)
```
the running result is:
```
value_gpu is:   tensor([[[1.2920]]], device='cuda:0')
value_cpu is:  tensor([[[-0.7080]]])
```
For the input tensor, 1.2920 and -0.7080, both values ​​above appear twice, why CPU return -0.7080, while GPU return 1.2920 ?

If all point values ​​appear only once, CPU and GPU will return the minimum, why partial point value appear twice, CPU still return the minimum, while GPU will not return the minimum? 

I wonder if this phenomenon is reasonable?

### Versions

pytorch v1.10.2 with CUDA10.2

cc @ngimel
[Distributed] `Invalid scalar type` when `dist.scatter()` boolean tensor
Labels: oncall: distributed

### 🐛 Describe the bug

I am trying to scatter boolean tensor but failed with error: `Invalid scalar type` while it works well with dtype=uint8.

**torch version: 1.13.0+cu117**

```
def worker(rank, world_size):
    ...
    dtype = th.uint8
    dtype = th.bool
    scatter_list = [ th.tensor([True, True, True, True], dtype=dtype) for _ in range(4) ]
    gather_list = [ th.tensor([False, False, False, False], dtype=dtype) for _ in range(4) ]
    for i in range(world_size):
        dist.scatter(gather_list[i], scatter_list if i == rank else [], src=i)
```
```
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 2290, in scatter
    work.wait()
RuntimeError: Invalid scalar type
```

This error also happens in other distributed call like `dist.all_reduce()`.
```
import os
import torch
from torch import distributed as dist
import torch.multiprocessing as mp

def worker(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    device = 'cpu' #'cuda'
    dist.init_process_group('gloo' if device == 'cpu' else 'nccl', rank=rank, world_size=world_size)

    dtype = torch.bool
    #dtype = torch.uint8
    tensor = torch.tensor([0,1], dtype=dtype)
    dist.all_reduce(tensor)

if __name__ == '__main__':
    mp.spawn(worker, args=(2,), nprocs=2, join=True)
```

### Versions

Collecting environment information...
PyTorch version: 1.13.0+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Jun 22 2022, 20:18:18)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-1022-aws-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 515.65.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.24.0rc1
[pip3] torch==1.13.0
[pip3] torchaudio==0.13.0
[pip3] torchvision==0.14.0
[conda] No relevant packages

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Strategy for optimizing away transient dynamic shapes / device syncs
Labels: triaged, bug, oncall: pt2

### 🐛 Describe the bug

Context: https://github.com/pytorch/xla/issues/4248

The key subtlety here is that although in principle we can perform these optimizations, we need to make sure not to "give up" too early by graph breaking immediately when the sync/dynamically sized output occurs. This may mean some metadata associated with tensors has to be computed more lazily (since to compute it eagerly would require a sync.) Also related to unbacked symints.

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym 

### Error logs

_No response_

### Minified repro

_No response_
Unexpected behaviour of 1.13.0
Labels: triaged, module: advanced indexing

### 🐛 Describe the bug

Hello, an unexpected behaviour appeared in PyTorch 1.13.0. I do not see it in Release Notes. Or is it a feature? Why can't I do it this way?

The following code works well in 1.12.1 but fails in 1.13.0:

```python
import torch

mask = torch.full((3,), True, dtype=bool, device='cuda')
array = torch.tensor([1, 2, 3], dtype=torch.float32, device='cpu')
array = array[mask]
print(array)
```

1.13.0 gives the following error:

```bash
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)
```

### Versions

pytorch=1.13.0
Option to let DistributedDataParallel know in advance unused parameters at each forward pass
Labels: oncall: distributed, enhancement

### 🚀 The feature, motivation and pitch

**Motivation:** In models with stochastic depth and the like, at each forward pass some layers (or parts of them) are skipped and thus one needs to set `find_unused_parameters=True`, which makes training much slower in general. Yet, one can implement these models in such a way that the unused parameters at each step are known **in advance** (e.g., the layer sampling is done before the model forward pass and not on-the-fly). It would then be great if we could feed this information to the DDP model so that it doesn't need to find the unused parameters.

The usage could be something like the following:

```python
model = DistributedDataParallel(model, find_unused_parameters=False)
for x in dataloader:
  # random_layer_sampling should return the same layers across GPUs
  layers_to_skip = random_layer_sampling(model)
  p = get_parameters(layers_to_skip)
  output = model(x, unused_parameters=p)
  ...
```

### Alternatives

Currently there is no option other than setting `find_unused_parameters=True`.

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu