gopikrsmscs/torch-issues huggingface dataset:
Process get killed when running torch.combinations
Labels: needs reproduction, module: performance, triaged, module: edge cases

### 🐛 Describe the bug

Process get killed even when feeding tensor with lower dimensions:


```
import torch
import numpy as np
arg_1_tensor = torch.randint(-512,16384,[4], dtype=torch.int16)
arg_1 = arg_1_tensor.clone()
arg_2 = 17
arg_3 = False
try:
  res = torch.combinations(arg_1,r=arg_2,with_replacement=arg_3,)
except Exception as e:
  print("Error:"+str(e))
```



### Versions

```
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.15 (main, Nov 24 2022, 14:31:59)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1660 Ti
Nvidia driver version: 525.78.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==1.13.1
[pip3] torchaudio==0.13.1
[pip3] torchvision==0.14.1
[conda] blas                      1.0                         mkl  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.23.5           py39h14f4228_0  
[conda] numpy-base                1.23.5           py39h31eccc5_0  
[conda] pytorch                   1.13.1          py3.9_cuda11.6_cudnn8.3.2_0    pytorch
[conda] pytorch-cuda              11.6                 h867d48c_1    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.13.1               py39_cu116    pytorch
[conda] torchvision               0.14.1               py39_cu116    pytorch
```

cc @ngimel
Floating point exception when running torch.nn.AdaptiveMaxPool3d
Labels: triaged

### 🐛 Describe the bug

Probably due to zero argument:

```
results = dict()
import torch
arg_1 = 0
arg_2 = True
arg_class = torch.nn.AdaptiveMaxPool3d(arg_1,return_indices=arg_2,)
arg_3_0_tensor = torch.rand([2, 2, 4, 4, 4], dtype=torch.float32)
arg_3_0 = arg_3_0_tensor.clone()
arg_3 = [arg_3_0,]
try:
  results["res_cpu"] = arg_class(*arg_3)
except Exception as e:
  print("Error:"+str(e))
arg_class = arg_class.cuda()
arg_3_0 = arg_3_0_tensor.clone().cuda()
arg_3 = [arg_3_0,]
try:
  results["res_gpu"] = arg_class(*arg_3)
except Exception as e:
  print("Error:"+str(e))

print(results)
```

### Versions

```
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.15 (main, Nov 24 2022, 14:31:59)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1660 Ti
Nvidia driver version: 525.78.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==1.13.1
[pip3] torchaudio==0.13.1
[pip3] torchvision==0.14.1
[conda] blas                      1.0                         mkl  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.23.5           py39h14f4228_0  
[conda] numpy-base                1.23.5           py39h31eccc5_0  
[conda] pytorch                   1.13.1          py3.9_cuda11.6_cudnn8.3.2_0    pytorch
[conda] pytorch-cuda              11.6                 h867d48c_1    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.13.1               py39_cu116    pytorch
[conda] torchvision               0.14.1               py39_cu116    pytorch

```
Process get killed when running torch.normal
Labels: triaged, module: numpy

### 🐛 Describe the bug

Process get killed when feeding negative large or even small tensors:

```
import torch
import numpy as np
arg_1_tensor = torch.neg(torch.rand([60000], dtype=torch.float16))
arg_1 = arg_1_tensor.clone()
arg_2_tensor = torch.rand([60000, 1], dtype=torch.float64)
arg_2 = arg_2_tensor.clone()
try:
  res = torch.normal(arg_1,arg_2,)
except Exception as e:
  print("Error:"+str(e))
```

### Versions

```
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.15 (main, Nov 24 2022, 14:31:59)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1660 Ti
Nvidia driver version: 525.78.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==1.13.1
[pip3] torchaudio==0.13.1
[pip3] torchvision==0.14.1
[conda] blas                      1.0                         mkl  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.23.5           py39h14f4228_0  
[conda] numpy-base                1.23.5           py39h31eccc5_0  
[conda] pytorch                   1.13.1          py3.9_cuda11.6_cudnn8.3.2_0    pytorch
[conda] pytorch-cuda              11.6                 h867d48c_1    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.13.1               py39_cu116    pytorch
[conda] torchvision               0.14.1               py39_cu116    pytorch
```

cc @mruberry @rgommers
segfault when running torch.lu_unpack
Labels: module: crash, triaged, module: linear algebra, module: edge cases

### 🐛 Describe the bug

Probably due to negative tensor.


```
import torch
import numpy as np
arg_1_tensor = torch.rand([1, 0, 5, 5], dtype=torch.float64)
arg_1 = arg_1_tensor.clone()
arg_2_tensor = torch.randint(-512,256,[1, 0], dtype=torch.int32)
arg_2 = arg_2_tensor.clone()
arg_3 = True
try:
  res = torch.lu_unpack(arg_1,arg_2,unpack_pivots=arg_3,)
except Exception as e:
  print("Error:"+str(e))
```

### Versions

```
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.15 (main, Nov 24 2022, 14:31:59)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1660 Ti
Nvidia driver version: 525.78.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==1.13.1
[pip3] torchaudio==0.13.1
[pip3] torchvision==0.14.1
[conda] blas                      1.0                         mkl  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.23.5           py39h14f4228_0  
[conda] numpy-base                1.23.5           py39h31eccc5_0  
[conda] pytorch                   1.13.1          py3.9_cuda11.6_cudnn8.3.2_0    pytorch
[conda] pytorch-cuda              11.6                 h867d48c_1    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.13.1               py39_cu116    pytorch
[conda] torchvision               0.14.1               py39_cu116    pytorch

```

cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano
[FSDP] Add `foreach` support to `FSDP.clip_grad_norm_()`
Labels: oncall: distributed, triaged, module: fsdp

See https://github.com/pytorch/pytorch/pull/91846 for `torch.nn.utils.clip_grad_norm_()` changes. Since `FSDP.clip_grad_norm_()` has to re-implement the logic to include gradient unsharding, we have to manually upstream improvements like this.



cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501
iter(TensorVariable) fail
Labels: triaged, bug, oncall: pt2

### 🐛 Describe the bug

Dynamo does not support iterating thru a tensor variable so far. Dynamo fail to trace even if the first dimension of the TensorVariable is static. In this case, the fix is simple since we can unroll the loop and generated items by calling `getitem`. But the problem becomes more complex if the first dimension of the tensor variable is dynamic.

We have discussions between shunting314 , @ezyang , @voznesenskym , @suo , @Chillee . Here are some thoughts for possible solutions
- rewrite user code by using torch.map explicitly
- let user pass in the information indicating if a dimension is dynamic or not
- assume the dimension to be static, specialize on that when compiling the fx graph and add a guard to the guard set

So the underlying problem is how we handle loop with potentially variable number of iterations.


User code that fail to be traced: https://www.internalfb.com/code/fbsource/[ca89130b9eae0efd5bb1cbbff1006cc1556d0b4f]/fbcode/vision/fair/detectron2/detectron2/export/c10.py?lines=483-489 


### Error logs

_No response_

### Minified repro

_No response_

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
set_default_device/torch.device has performance impact for non-factory functions
Labels: module: performance, triaged, module: __torch_function__

### 🐛 Describe the bug

Because it is implemented as a TorchFunctionMode, we interpose ALL functions even if we actually do nothing on those functions. We have ideas for how to optimize this but have delayed doing so until someone actually has a problem.  This is a placeholder issue to track people having this problem.

### Versions

master

cc @hameerabbasi @rgommers @peterbell10 @soumith @msaroufim @wconstab @ngimel @bdhirsh
API to check for errors in c10d.ProcessGroupNCCL
Labels: oncall: distributed

### 🚀 The feature, motivation and pitch

ProcessGroupNCCL has two options to detect and deal with errors:

1. `NCCL_BLOCKING_WAIT` which is blocking and causes host to device synchronization resulting in perf degradation but raises an appropriate exception in-line in your training code.
2. `NCCL_ASYNC_ERROR_HANDLING` which polls in the background and kills the process if errors are detected.

In certain cases neither of these options are ideal and users would like to inspect ProcessGroup errors themselves and appropriately recover from them if possible.

As a result, the proposal here is to expose an API like `ProcessGroup.hasErrorOrTimedOut` based on which users can take custom actions if an error is indeed discovered. Happy to contribute a PR for this.

### Alternatives

Considered `NCCL_BLOCKING_WAIT` and `NCCL_ASYNC_ERROR_HANDLING` as mentioned above

### Additional context

_No response_

cc @mrshenli @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Torchscript troubles with complex values. RuntimeError: isInt() INTERNAL ASSERT FAILED
Labels: oncall: jit, triaged, module: complex

### 🐛 Describe the bug

Torchscript errors out when creating imaginary coefficients. This simple example runs fine in pytorch without torchscript. I'd expect to get the same result with torch and torchscript. 

```
import torch

class SimpleEx(torch.nn.Module):
    def __init__(self):
        super(SimpleEx, self).__init__()
        self.k = 1.23

    def forward(self):
        a = torch.randn(3)
        b = torch.randn(11,11)
        return 1j*self.k/2*(1-a[1])/a[0]*b

if __name__ == '__main__':

    se = torch.jit.script(SimpleEx())
    se_out = se()
    print(se_out)
```

The following error is produced when running as a module

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/user/SimpleEx.py", line 17, in <module>
    se_out = se()
  File "/home/user/my_venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "/home/user/SimpleEx.py", line 12, in forward
        a = torch.randn(3)
        b = torch.randn(11,11)
        return 1j*self.k/2*(1-a[1])/a[0]*b
               ~~~~~~~~~~~ <--- HERE
RuntimeError: isInt() INTERNAL ASSERT FAILED at "../aten/src/ATen/core/ivalue.h":602, please report a bug to PyTorch.



### Versions

Collecting environment information...
PyTorch version: 1.13.1+cpu
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Nov 14 2022, 12:59:47)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-136-generic-x86_64-with-glibc2.29
Is CUDA available: False
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: GPU 0: Quadro P1000
Nvidia driver version: 495.29.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.4
[pip3] torch==1.13.1+cpu
[pip3] torchvision==0.14.1+cpu
[conda] Could not collect

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel @ezyang @anjali411 @dylanbespalko @mruberry @Lezcano @nikitaved
Update quantization to make source files complient with /Zc:lambda
Labels: triage review, module: windows, oncall: quantization, triaged

# Summary
While working on this PR: [#91909](https://github.com/pytorch/pytorch/pull/91909) which uses a c++ 17 optional flag to enable the updated lambda preprocessor https://learn.microsoft.com/en-us/cpp/build/reference/zc-lambda?view=msvc-170, I was unable to get windows builds to pass applying this globally. The failing tests: 
```
C:\actions-runner\_work\pytorch\pytorch\aten\src\ATen\native\quantized\cuda\AffineQuantizer.cu(45): error C2326: 'scalar_t at::native::_GLOBAL__N__809cb215_18_AffineQuantizer_cu_95ed4547_5828::quantize_tensor_per_tensor_affine_cuda::<lambda_1>::()::<lambda_1>::()::<lambda_1>::operator ()(float,scalar_t) const': function cannot access 'qmin'
C:\actions-runner\_work\pytorch\pytorch\aten\src\ATen\native\quantized\cuda\AffineQuantizer.cu(45): error C2326: 'scalar_t at::native::_GLOBAL__N__809cb215_18_AffineQuantizer_cu_95ed4547_5828::quantize_tensor_per_tensor_affine_cuda::<lambda_1>::()::<lambda_1>::()::<lambda_1>::operator ()(float,scalar_t) const': function cannot access 'qmax'
C:\actions-runner\_work\pytorch\pytorch\aten\src\ATen\native\quantized\cuda\AffineQuantizer.cu(45): error C2326: 'scalar_t at::native::_GLOBAL__N__809cb215_18_AffineQuantizer_cu_95ed4547_5828::quantize_tensor_per_tensor_affine_cuda::<lambda_1>::()::<lambda_2>::()::<lambda_1>::operator ()(float,scalar_t) const': function cannot access 'qmin'
```

In c++ 20 the updated lambda expression will become the default. So in order to smooth the transition we should update the lambda expression in this file to be conforming.

cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel
Improve Fake Tensor Error When Data Ptr is Accessed
Labels: triaged, module: fakeTensor

### 🐛 Describe the bug

The error is not very informative as is:
```
The tensor has a non-zero number of elements, but its data is not allocated yet. Caffe2 uses a lazy allocation, so you will need to call mutable_data() or raw_mutable_data() to actually allocate memory.?
```

### Versions

...
Spurious side effect diff when cond branches call different functions in outer scope
Labels: triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

Consider the following code where we're calling `cond` with branches `shallow` and `deep`, such that `deep` references functions in outer scope that `shallow` doesn't.

```python
class Module(torch.nn.Module):
    def forward(self, pred, x):
        return self.indirection(pred, x)

    def indirection(self, pred, x):
        def true_fn(y):
            return y + 2

        def false_fn(y):
            return y - 2

        def shallow(x):
            return x * 2

        def deep(x):
            return cond(
                x[0][0] > 0,
                true_fn,
                false_fn,
                [x],
            )

        return cond(pred, shallow, deep, [x])
```

Trying to export this code leads to an exception:
```
<snip>
  File "<snip>/torch/_dynamo/variables/torch.py", line 831, in call_function
    unimplemented(true_cmp.diff(false_cmp))
  File "<snip>/torch/_dynamo/exc.py", line 67, in unimplemented
    raise Unsupported(msg)
torch._dynamo.exc.Unsupported: id_to_variable keys: odict_keys([140465014772608, 140465014771968, 140465014772496]) != odict_keys([])
```

### Versions

PyTorch version: 1.14.0a0+gited9cd47
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.1 (x86_64)
GCC version: Could not collect
Clang version: 13.0.0 (clang-1300.0.29.30)
CMake version: version 3.22.1
Libc version: N/A

Python version: 3.9.13 (main, Aug 25 2022, 18:29:29)  [Clang 12.0.0 ] (64-bit runtime)
Python platform: macOS-10.16-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] numpydoc==1.4.0
[pip3] torch==1.14.0a0+gited9cd47
[conda] blas                      1.0                         mkl
[conda] mkl                       2021.4.0           hecd8cb5_637
[conda] mkl-include               2022.1.0           hecd8cb5_209
[conda] mkl-service               2.4.0            py39h9ed2024_0
[conda] mkl_fft                   1.3.1            py39h4ab4a9b_0
[conda] mkl_random                1.2.2            py39hb2f4e1b_0
[conda] numpy                     1.21.5           py39h2e5f0a9_3
[conda] numpy-base                1.21.5           py39h3b1a694_3
[conda] numpydoc                  1.4.0            py39hecd8cb5_0
[conda] torch                     1.14.0a0+gited9cd47           dev_0    <develop>

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @mlazos @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
Major bug in Transformers' masks
Labels: high priority, triage review, oncall: transformer/mha, module: correctness (silent)

### 🐛 Describe the bug

In torch.nn.functional, line 5174 in current code:
![image](https://user-images.githubusercontent.com/4967417/213253672-3dfd0784-8b4c-4513-b1f0-d14a9a964612.png)

`attn_mask = attn_mask.masked_fill(key_padding_mask, float("-inf"))`

Fills `-inf` value wherever `key_padding_mask` is True. However, `key_padding_masks` actually contains -inf values instead of True values. So the padding mask is actually ignored. Replacing by:

`attn_mask = attn_mask.masked_fill(key_padding_mask < 0, float("-inf")`

Gives the correct expected mask.



### Versions

Current github code.  (master branch, Jan 18, 2023)

cc @ezyang @gchanan @zou3519 @albanD @mruberry @jbschlosser @walterddr @saketh-are @bhosmer @cpuhrsch @erichan1
Pytorch AMP performance issue.
Labels: triaged, module: memory format, module: amp (automated mixed precision)

### 🐛 Describe the bug

I found that different models have large speed differences between AMP and FP32 training configurations with NHWC and NCHW data types. pytorch seems to only optimize matmul and conv2d, while batchnorm support for nhwc is poor. The batchnorm layer of nhwc is very slow, and the speed recovers after freezing the bn layers. In addition, the transpose of nchw also consumes 10% of the latency. I collected performance data.



Demo codes:
`
```
import torch
import torch.nn as nn
import time
import torch.backends.cudnn as cudnn
from torchvision.models import resnet101
cudnn.benchmark=True/False

class SimpleLayer(nn.Module):
    def __init__(self):
        super(SimpleLayer, self).__init__()
        self.conv1=nn.Sequential(nn.Conv2d(192,192,3,1,1),nn.ReLU(),nn.Conv2d(192,192,3,1,1),nn.ReLU(),nn.Conv2d(192,192,3,1,1),nn.ReLU(),nn.Conv2d(192,192,3,1,1),nn.ReLU())

    def forward(self,x):
        return self.conv1(x)

Model=SimpleLayer/resnet101().to("cuda:0", memory_format=torch.channels_last)
Model.eval()
Data=torch.randn(8,3,224,224).to("cuda:0", memory_format=torch.channels_last)

while True:
    stimes=time.time()
    with torch.amp.autocast(device_type='cuda'):
        for _ in range(10):
            _=Model(Data)
    torch.cuda.synchronize()
    times=time.time()-stimes
    print(10/times)
```
`

Speed SimpleLayer
![image](https://user-images.githubusercontent.com/4530735/213177319-2ea9a3e4-f807-4c94-aea7-9a3a45bc579c.png)
VGG19
![image](https://user-images.githubusercontent.com/4530735/213177361-e2ada260-7688-4e5d-badb-74f6513967ac.png)
Resnet101
![image](https://user-images.githubusercontent.com/4530735/213177393-8b918186-e250-4afe-b13b-4b8805af32fb.png)
EfficientNet-B6
![image](https://user-images.githubusercontent.com/4530735/213177434-cc6ac25b-b7ca-40cb-8a52-cdc81f31bca3.png)
SwinBase
![image](https://user-images.githubusercontent.com/4530735/213177470-eeab7372-a714-4677-97fd-b35421b79bc7.png)

resnet101-cudnn-amp-nchw
![image](https://user-images.githubusercontent.com/4530735/213177559-b83a0a7f-659b-4e76-9b1a-ac568e7799fc.png)
resnet101-cudnn-amp-nhwc
![image](https://user-images.githubusercontent.com/4530735/213177568-264d9cf6-d275-4e80-8d6d-8891c03015f6.png)



### Versions

Pytorch 1.11 Cudnn 11.6

cc @jamesr66a @mcarilli @ptrblck @leslie-fang-intel @jgong5
multiprocessing not work on WSL2
Labels: module: multiprocessing, triaged, module: wsl

### 🐛 Describe the bug

Hi there,

**🐛 Describe the bug**

Torch multiprocessing seems not work on WSL2 + Ubuntu20.04. 

```
import torch
from torch.multiprocessing import set_start_method, Pipe, Process

def func(data):
    print("in func")
    # data = conn.recv()
    print(data) # .item()

if __name__ == "__main__":
    set_start_method('spawn', force=True)
    
    device = torch.device("cuda") # "cuda:0"
    data = torch.tensor([1, 2, 3]).to(device)
    proc = Process(target=func, args=(data,))
    proc.start()
    proc.join()
```

When trying to run this code, it is stuck. (Even `collect_env.py` stuck)


**Versions**:
```
Cuda 11.7
WSL2: ubuntu20.04
Pytorch:  1.13.1+cu117
```
I actually tried other pytorch versions still the same issue.

Thanks.


### Versions

Hi there,

**🐛 Describe the bug**

Torch multiprocessing seems not work on WSL2 + Ubuntu20.04. 

```
import torch
from torch.multiprocessing import set_start_method, Pipe, Process

def func(data):
    print("in func")
    # data = conn.recv()
    print(data) # .item()

if __name__ == "__main__":
    set_start_method('spawn', force=True)
    
    device = torch.device("cuda") # "cuda:0"
    data = torch.tensor([1, 2, 3]).to(device)
    proc = Process(target=func, args=(data,))
    proc.start()
    proc.join()
```

When trying to run this code, it is stuck. (Even `collect_env.py` stuck)


**Versions**:
```
Cuda 11.7
WSL2: ubuntu20.04
Pytorch:  1.13.1+cu117
```
I actually tried other pytorch versions still the same issue.

Thanks.


cc @VitalyFedyunin @ejguan
INTERNAL ASSERT FAILED: Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created.
Labels: oncall: distributed

### 🐛 Describe the bug

When data gets too big, an error is thrown and crashes.

[E request_callback_no_python.cpp:559] Received error while processing request type 261: false INTERNAL ASSERT FAILED at "../torch/csrc/distributed/rpc/rref_context.cpp":393, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created.

It only happens when data stored on ram gets too big

### Versions

torch==1.13.0

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
[Inductor] support complex dtypes
Labels: triaged, module: complex, module: random, module: inductor

### 🚀 The feature, motivation and pitch

I see Inductor cpp codgegen `DTYPE_TO_CPP`, `DTYPE_TO_ATEN` do not support complex dtypes.
https://github.com/pytorch/pytorch/blob/9b173b87b2587ee085aaef65e1a0fdebd90d5229/torch/_inductor/codegen/cpp.py#L31-L55

 
For example, with `randn` (complex dtype support is missing for other ops as well (e.g., `rand`, `sqrt`, etc))
```python
import torch

def foo(x):
    return torch.randn(x.size(), dtype=x.dtype)

opt_foo = torch.compile(foo)
print("eager: ", foo(torch.empty(1, dtype=torch.complex64)))
print("inductor: ", opt_foo(torch.empty(1, dtype=torch.complex64))) 
```
raises the following error from codegen
```
return f"static_cast<{DTYPE_TO_CPP[dtype]}>(randn_cpu({seed}, {offset}));"
KeyError: 'torch.complex64\n\nWhile executing %randn : [#users=1] = call_method[target=randn](args = (%ops, %bitwise_xor, %constant_1, torch.complex64), kwargs = {})\nOriginal traceback:\nNone'
```


I wonder if there's any context to why complex dtypes are currently generally not supported in Inductor.

For `randn`, we can extend `Philox4_32` `randn` to support complex dtypes.
```
Z ~ Complex Normal(0, 1)
Z = 1/sqrt(2) * [X + Xj],     X ~ Normal(0, 1) 
```



### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @anjali411 @dylanbespalko @mruberry @Lezcano @nikitaved @pbelevich @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire
operations failed in TorchScript interpreter
Labels: oncall: jit

### 🐛 Describe the bug

when I use  torch.jit._get_trace_graph to trace my model on GPU，it happens that
The PyTorch internal failed reason is:
The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "/xxxx/python37/lib/python3.7/site-packages/torchvision/models/detection/roi_heads.py", line 466, in _onnx_paste_masks_in_image_loop
    res_append = torch.zeros(0, im_h, im_w)
    for i in range(masks.size(0)):
        mask_res = _onnx_paste_mask_in_image(masks[i][0], boxes[i], im_h, im_w)
                   ~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
        mask_res = mask_res.unsqueeze(0)
        res_append = torch.cat((res_append, mask_res))
  File "/xxx/site-packages/torchvision/models/detection/roi_heads.py", line 430, in _onnx_paste_mask_in_image
    zero = torch.zeros(1, dtype=torch.int64)

    w = box[2] - box[0] + one
        ~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    h = box[3] - box[1] + one
    w = torch.max(torch.cat((w, one)))
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!


### Versions

pytorch 1.12

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
TypeError: no implementation found for 'torch._ops.aten.max.default' on types that implement __torch_dispatch__: [<class 'torch.masked.maskedtensor.core.MaskedTensor'>]
Labels: triaged, module: masked operators

### 🐛 Describe the bug

My problem with MaskedTensor is that the stacktrace is extremely incorrect and vague, I am unable to debug the source of any error. It claims here that the "max" is not supported but there is no max in the code, and it doesn't even reference a relevant line, there is no way the for loop line is erroring as there is no pytorch code there.


```
/usr/local/lib/python3.8/dist-packages/torch/masked/maskedtensor/core.py:299: UserWarning: max is not implemented in __torch_dispatch__ for MaskedTensor.
If you would like this operator to be supported, please file an issue for a feature request at https://github.com/pytorch/maskedtensor/issues with a minimal reproducible code snippet.
In the case that the semantics for the operator are not trivial, it would be appreciated to also include a proposal for the semantics.
  warnings.warn(msg)
Traceback (most recent call last):
  File "adam_aug_lagrangian.py", line 90, in <module>
    for outermost_training_iter in range(num_training_iters):
  File "/usr/local/lib/python3.8/dist-packages/torch/masked/maskedtensor/core.py", line 274, in __torch_function__
    ret = func(*args, **kwargs)
TypeError: no implementation found for 'torch._ops.aten.max.default' on types that implement __torch_dispatch__: [<class 'torch.masked.maskedtensor.core.MaskedTensor'>]

```

### Versions

```
Collecting environment information...
PyTorch version: 1.13.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Nov 14 2022, 12:59:47)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-132-generic-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA RTX A5000
Nvidia driver version: 465.19.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.1
[pip3] pytorch-minimize==0.0.2
[pip3] torch==1.13.1
[conda] Could not collect

```
fft.fftshift, fft.ifftshift, roll not implemented
Labels: triaged, module: fft, module: mps

### 🚀 The feature, motivation and pitch

I am working on Bragg CDI reconstruction and the algorithm uses fftshift and ifftshift. I would like to be able to run the code on MPS device.

### Alternatives

I can implement the functions with roll but it's also not supported for the MPS device.

### Additional context

_No response_

cc @mruberry @peterbell10 @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
backward(inputs= does not need to execute grad_fn of the inputs
Labels: module: bc-breaking, module: autograd, triaged, actionable, topic: bc breaking

Per title, we currently execute grad_fn of the inputs when running `.backward()` with the `inputs=` argument. We should avoid unnecessarily doing this extra compute since it is not actually necessary for the computation of the gradients wrt the inputs. 

This is bc-breaking. This is mostly a bug-fix but a case where users may depend on this behavior is if they depend on hooks registered to Node to execute for the backward(inputs= case. Fixing this will require some changes to how we compute `needed` in the engine.

cc @ezyang @gchanan @albanD @zou3519 @gqchen @pearu @nikitaved @Lezcano @Varal7
Simplify module backward hooks to use multi-grad hooks instead
Labels: module: bc-breaking, module: autograd, module: nn, triaged, needs research, topic: bc breaking

Since modules can take in multiple inputs and return multiple outputs, currently we use a dummy custom autograd Function to ensure that there is a single grad_fn Node that we can attach our hook to. A multi-grad hook can be used to solve this issue in a less hacky way.

Note that this would be a bc-breaking change, as the hooks would be register to tensor instead of grad_fn, and the recorded backward graph would be different.

cc @ezyang @gchanan @albanD @zou3519 @gqchen @pearu @nikitaved @Lezcano @Varal7 @mruberry @jbschlosser @walterddr @mikaylagawarecki @saketh-are
jit.fork stalls multiprocessing dataloader
Labels: oncall: jit, module: dataloader, module: data

### 🐛 Describe the bug

Using `torch.jit.fork` with multiprocessing dataloaders stalls the dataloader.

I would expect one of the following:

1. The dataloader would simply work.
2. An error would result from trying to run `jit.fork` outside of the main process.
3. The documentation would warn of incompatability between `jit.fork` and parallel dataloaders.

Option #2 and/or #3 would probably be most realistic, due to the nature of the `multiprocessing` module.

``` python
import torch as th

def foo(x):
    return x+1

@th.jit.script
def jit_fork_foo(x):
    return th.jit.wait(th.jit.fork(foo, x))

class Dataset(th.utils.data.IterableDataset):
    def __iter__(self):
        while True:
            yield jit_fork_foo(th.tensor([1]))

ds = Dataset()
dl_nomp = th.utils.data.DataLoader(ds, batch_size=2, num_workers=0)
dl_mp = th.utils.data.DataLoader(ds, batch_size=2, num_workers=2)

print("No multiprocessing ", next(iter(dl_nomp))) #returns as expected
print("multiprocessing ", next(iter(dl_mp))) #stalls
``` 

### Versions

```PyTorch version: 1.14.0a0+410ce96
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.24.1
Libc version: glibc-2.31

Python version: 3.8.10 (default, Nov 14 2022, 12:59:47)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-125-generic-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY

Nvidia driver version: 470.141.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.7.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.7.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.7.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.7.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.7.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.7.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.7.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] pytorch-quantization==2.1.2
[pip3] torch==1.14.0a0+410ce96
[pip3] torch-tensorrt==1.3.0a0
[pip3] torchtext==0.13.0a0+fae8e8c
[pip3] torchvision==0.15.0a0
```

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel @SsnL @VitalyFedyunin @ejguan @NivekT
"Get Started" tells us to use the anaconda installer for PyTorch 3.x - but this should be python 3.x
Labels: oncall: binaries

### 📚 The doc issue

[Anaconda](https://pytorch.org/get-started/locally/#anaconda)
To install Anaconda, you will use the [64-bit graphical installer](https://www.anaconda.com/download/#windows) for PyTorch 3.x.

### Suggest a potential alternative/fix

[Anaconda](https://pytorch.org/get-started/locally/#anaconda)
To install Anaconda, you will use the [64-bit graphical installer](https://www.anaconda.com/download/#windows) for Python 3.x.

Tiny issue (o: hope the feedback helps

cc @ezyang @seemethere @malfet
InstanceNorm operator support for Vulkan devices
Labels: triaged, module: vulkan

### 🚀 The feature, motivation and pitch

At present, only BatchNorm op is supported for Vulkan drivers. The model file throws an error when using InstanceNorm on a Vulkan device ( Adreno )

### Alternatives

I have replaced InstanceNorm with BatchNorm for the time being

### Additional context

_No response_
Always install cpu version automatically
Labels: oncall: binaries

### 🐛 Describe the bug

Hi!

I use following cmd to install pytorch
```
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=10.2 -c pytorch
```

however I got the result:

```
## Package Plan ##

  environment location: D:\anaconda3

  added / updated specs:
    - cudatoolkit=10.2
    - pytorch==1.12.1
    - torchaudio==0.12.1
    - torchvision==0.13.1


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-22.9.0               |   py38haa95532_0         888 KB  defaults
    cudatoolkit-10.2.89        |       h74a9793_1       317.2 MB  defaults
    libuv-1.40.0               |       he774522_0         255 KB  defaults
    pytorch-1.12.1             |      py3.8_cpu_0       133.8 MB  pytorch
    pytorch-mutex-1.0          |              cpu           3 KB  pytorch
    torchaudio-0.12.1          |         py38_cpu         3.5 MB  pytorch
    torchvision-0.13.1         |         py38_cpu         6.2 MB  pytorch
    ------------------------------------------------------------
                                           Total:       461.9 MB

The following NEW packages will be INSTALLED:

  cudatoolkit        anaconda/pkgs/main/win-64::cudatoolkit-10.2.89-h74a9793_1
  libuv              anaconda/pkgs/main/win-64::libuv-1.40.0-he774522_0
  pytorch            pytorch/win-64::pytorch-1.12.1-py3.8_cpu_0
  pytorch-mutex      pytorch/noarch::pytorch-mutex-1.0-cpu
  torchaudio         pytorch/win-64::torchaudio-0.12.1-py38_cpu
  torchvision        pytorch/win-64::torchvision-0.13.1-py38_cpu

The following packages will be UPDATED:

  conda               pkgs/main::conda-4.9.2-py38haa95532_0 --> anaconda/pkgs/main::conda-22.9.0-py38haa95532_0


Proceed ([y]/n)?
```

I wonder why conda install cpu version automatically? How to install gpu version?

This is my hardware information:
```
Tue Jan 17 10:04:01 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 516.94       Driver Version: 516.94       CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ... WDDM  | 00000000:65:00.0  On |                  N/A |
|  0%   43C    P8    20W / 370W |    885MiB / 24576MiB |      6%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
```

Could you please give me some advice? Thanks in advance.

### Versions

pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=10.2

cc @ezyang @seemethere @malfet
distributions.Beta returning incorrect results at 0 and 1
Labels: module: distributions, triaged

### 🐛 Describe the bug

The beta distribution has support either on [0, 1] or, for some implementations, on (0, 1). However, the PyTorch version appears to have support on [0, 1] but it returns `-inf` or `nan` on those values.

```
In [31]: beta = torch.distributions.Beta(torch.tensor([1.0]), torch.tensor([1.0]), validate_args=True)

In [32]: beta.support
Out[32]: Interval(lower_bound=0.0, upper_bound=1.0)

In [33]: beta.log_prob(torch.tensor([1.0]))
Out[33]: tensor([nan])

In [34]: beta.log_prob(torch.tensor([0.0]))
Out[34]: tensor([nan])

In [35]: beta = torch.distributions.Beta(torch.tensor([2.0]), torch.tensor([2.0]), validate_args=True)

In [36]: beta.log_prob(torch.tensor([0.0]))
Out[36]: tensor([-inf])

In [37]: beta.log_prob(torch.tensor([1.0]))
Out[37]: tensor([-inf])
```

Note that we did not get a `ValueError` when we passed 0 and 1 even though `validate_args=True` was set, but we did get `-inf` and/or `nan` even though that's not the correct answer. Also, the `support` seems to indicate that 0 and 1 are legal (though I couldn't find documentation of the `Interval` class so I'm not sure if that's supposed to be inclusive or exclusive).

### Versions

PyTorch version: 1.13.0+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 11 (bullseye) (x86_64)
GCC version: (Debian 10.2.1-6) 10.2.1 20210110
Clang version: Could not collect
CMake version: version 3.18.4
Libc version: glibc-2.31

Python version: 3.10.6 (main, Oct 11 2022, 01:13:46) [GCC 10.2.1 20210110] (64-bit runtime)
Python platform: Linux-5.10.0-20-cloud-amd64-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A100-SXM4-40GB
Nvidia driver version: 520.61.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] torch==1.13.0+cu116
[pip3] torchvision==0.14.0+cu116
[conda] Could not collect


cc @fritzo @neerajprad @alicanb @nikitaved
[discussion] Fused MLPs
Labels: feature, triaged, oncall: pt2, module: inductor

### 🚀 The feature, motivation and pitch

I just stumbled on https://twitter.com/DrJimFan/status/1615018393601716224, there is https://github.com/NVlabs/tiny-cuda-nn which fuses small MLPs for fast training and inference. Can Inductor / Triton generate similar PTX / memory placement?

It seems at least as a good comparison target for benchmark

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @mlazos @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire
test_qnnpack_add fails
Labels: oncall: mobile, module: xnnpack

### 🐛 Describe the bug

The test fails with this:
```
test_qnnpack_add (quantization.core.test_quantized_op.TestQNNPackOps) ... Falsifying example: test_qnnpack_add(
    A=(array([1.], dtype=float32), (1.0, 0, torch.qint8)),
    zero_point=127,
    scale_A=0.057,
    scale_B=0.008,
    scale_C=0.003,
    self=<quantization.core.test_quantized_op.TestQNNPackOps testMethod=test_qnnpack_add>,
)
FAIL
```

The test_qnnpack_add_broadcast fails due to similar reasons.

With a bit of code reading I found this:
- `qA.dequantize()` and `qB.dequantize()` are zero due to the scaling and a zero_point of 127
- Hence the [groundtruth C](https://github.com/pytorch/pytorch/blob/76c88364ed6ffcef41743c8370d6ff1306e0100d/test/quantization/core/test_quantized_op.py#L5914) is also zero which is passed to `_quantize` yielding `qC = [0]`
- But the `qC_qnnp.int_repr = [127]`

More intermediates:
```
qA = tensor([0.], size=(1,), dtype=torch.qint8,
       quantization_scheme=torch.per_tensor_affine, scale=0.057, zero_point=127)
qB = tensor([0.], size=(1,), dtype=torch.qint8,
       quantization_scheme=torch.per_tensor_affine, scale=0.008, zero_point=127)
qA/qB.dequantize() = tensor([0.])
qC_qnnp = tensor([0.3810], size=(1,), dtype=torch.qint8,
       quantization_scheme=torch.per_tensor_affine, scale=0.003, zero_point=0)
qC_qnnp.int_repr() = tensor([127], dtype=torch.int8)
qC_qnnp.dequantize() = tensor([0.3810])
```


### Versions

PyTorch 1.12.1

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel
CapabilityBasedPartitioner incorrectly sorts the graph, causing optimizer return/output node to be first
Labels: triaged, module: fx, oncall: pt2

### 🐛 Describe the bug

Hello. I prepared code to test custom backends for torch.compile functionality. One of the things my backend is doing is using fx.Interpreter to go through all operations (via fake tensors) and propagate device info. Then I use this information to cluster some of the operations via CapabilityBasedPartitioner. For purpose of this bug I have simplified the code, it is available here:
https://gist.github.com/kbadz-pl/9bff0459fa37f4231125c0163aec4b89

Issue is that when I try to run partitioning and merging over optimizer code, it seems to wrongly treat return/output node because it has no arguments/predecessors. It ends up being first and effectively causing optimizer to not run at all (checked in the run traces).

SGD graph before partitioning:
![image](https://user-images.githubusercontent.com/122799060/212693505-fb75ea68-d5eb-4c90-943c-e496b900f38e.png)

SGD graph after partitioning and merging:
![image](https://user-images.githubusercontent.com/122799060/212693793-df170ca8-5315-4809-8e64-6376e44482b4.png)


### Versions

PyTorch version: 2.0.0.dev20230116+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Nov 14 2022, 12:59:47) [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-126-generic-x86_64-with-glibc2.29
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] torch==2.0.0.dev20230116+cpu
[pip3] torchaudio==2.0.0.dev20230115+cpu
[pip3] torchvision==0.15.0.dev20230115+cpu
[conda] Could not collect

cc @ezyang @SherlockNoMad @soumith @EikanWang @jgong5 @wenzhe-nrv @msaroufim @wconstab @ngimel @bdhirsh