gopikrsmscs/torch-issues huggingface dataset:
[Inductor] incorrect result of vision_maskrcnn
Labels: triaged, bug

### 🐛 Describe the bug

Vision_maskrcnn of torchbench has incorrect result when using inductor.
RuntimeError: The size of tensor a (36) must match the size of tensor b (34) at non-singleton dimension 0

Reproduce command:
python benchmarks/dynamo/torchbench.py --accuracy --float32 -dcpu --inductor-settings --float32 -n50 --inductor --no-skip --quiet -k "vision_maskrcnn"


### Error logs

cpu  eval  vision_maskrcnn                    
Traceback (most recent call last):
  File "benchmarks/dynamo/torchbench.py", line 349, in <module>
    main(TorchBenchmarkRunner(), original_dir)
  File "/home/bzheng/workspace/pytorch/benchmarks/dynamo/common.py", line 1532, in main
    return maybe_fresh_cache(run, args.cold_start_latency and args.only)(
  File "/home/bzheng/workspace/pytorch/benchmarks/dynamo/common.py", line 772, in inner
    return fn(*args, **kwargs)
  File "/home/bzheng/workspace/pytorch/benchmarks/dynamo/common.py", line 1845, in run
    runner.run_one_model(
  File "/home/bzheng/workspace/pytorch/benchmarks/dynamo/common.py", line 1267, in run_one_model
    status = self.check_accuracy(
  File "/home/bzheng/workspace/pytorch/benchmarks/dynamo/common.py", line 1104, in check_accuracy
    if not same(
  File "/home/bzheng/workspace/pytorch/torch/_dynamo/utils.py", line 774, in same
    return len(ref) == len(res) and all(
  File "/home/bzheng/workspace/pytorch/torch/_dynamo/utils.py", line 775, in <genexpr>
    same(ai, bi, fp64_refi, cos_similarity, tol, equal_nan, exact_dtype)
  File "/home/bzheng/workspace/pytorch/torch/_dynamo/utils.py", line 785, in same
    same(
  File "/home/bzheng/workspace/pytorch/torch/_dynamo/utils.py", line 831, in same
    if torch.allclose(ref, res, atol=tol, rtol=tol, equal_nan=equal_nan):
RuntimeError: The size of tensor a (36) must match the size of tensor b (34) at non-singleton dimension 0
ERROR
cpu             gmean=nanx mean=nanx
vision_maskrcnn gmean=nanx mean=nanx
0               gmean=nanx mean=nanx
0.0000          gmean=nanx mean=nanx

### Minified repro

python benchmarks/dynamo/torchbench.py --accuracy --float32 -dcpu --inductor-settings --float32 -n50 --inductor --no-skip --quiet -k "vision_maskrcnn"
cudagraphify Dynamo's nvFuser backend
Labels: open source, module: nvfuser, module: dynamo, ciflow/inductor, no-stale

This PR rewrites the "nvprims_nvfuser" code to reuse `cudagraphify` function.


cc @kevinstephano @jjsjann123 @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
Add a config option to raise errors instead of warnings in nvFuser integration
Labels: triaged, module: nvfuser

        FWIW, this test will be less fragile if you have a way to convert nvfuser warnings into true errors. Otherwise, if someone adds an unrelated warning that this test happens to exercise, this test will start failing

_Originally posted by @ezyang in https://github.com/pytorch/pytorch/pull/88186#discussion_r1010485596_
      

cc @kevinstephano @jjsjann123
[docs] torch.is_neg/torch.Tensor.is_neg not documented
Labels: module: docs, triaged

### 📚 The doc issue

It seems to be public but there is no documentation.
```python
>>> torch.is_neg(t)
False
>>> torch.is_neg.__doc__
>>> 
```


### Suggest a potential alternative/fix

_No response_

cc @svekars @carljparker
`torch.nn.RReLU` not reporting `lower > upper` on CUDA
Labels: module: cuda, triaged

### 🐛 Describe the bug

If we set `lower > upper` for a `torch.nn.RReLU` layer, it does not report error when the layer is built, or even when it is applied to a CUDA tensor. It is discovered only after being applied to a tensor on CPU. 

``` python
m = torch.nn.RReLU(lower=0.3, upper=0.2)
x = torch.tensor([-1., -1, -1, -1]).cuda()
mx = m(x)
print(mx)
y = torch.tensor([-1., -1, -1, -1])
my = m(y)
```

Run the code above and you get: 

```
tensor([-0.2930, -0.2569, -0.2189, -0.2043], device='cuda:0')
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-3-44e43dbfd4d8>](https://localhost:8080/#) in <module>
      4 print(mx)
      5 y = torch.tensor([-1., -1, -1, -1])
----> 6 my = m(y)

2 frames
[/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py](https://localhost:8080/#) in rrelu(input, lower, upper, training, inplace)
   1673         result = torch.rrelu_(input, lower, upper, training)
   1674     else:
-> 1675         result = torch.rrelu(input, lower, upper, training)
   1676     return result
   1677 

RuntimeError: Expected from <= to to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
```

Maybe we can check for the validity of `lower` and `upper` immediately when the layer is constructed. It will not affect runtime performance much since it is only done once, not every time when it is applied to a tensor. 

### Versions

Collecting environment information...
PyTorch version: 1.12.1+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.22.6
Libc version: glibc-2.26

Python version: 3.7.15 (default, Oct 12 2022, 19:14:55)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: True
CUDA runtime version: 11.2.152
CUDA_MODULE_LOADING set to: 
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.12.1+cu113
[pip3] torchaudio==0.12.1+cu113
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.13.1
[pip3] torchvision==0.13.1+cu113
[conda] Could not collect

cc @ngimel
Moving tensor to GPU by .cuda() gets stucked when AMD Secure Encripted Virtualization (SEV) is activated
Labels: module: cuda, module: rocm, triaged

### 🐛 Describe the bug

My workstation is equipped with an AMD Threadripper 3960x and GPUs of 30 series, with Ubuntu 22.04.1 installed. 

Before I activated the AMD SEV, everything is normal. After I activated the AMD SEV by:
1. change the line <GRUB_CMDLINE_LINUX_DEFAULT="quite splash"> to <GRUB_CMDLINE_LINUX_DEFAULT="quite splash mem_encrypt=on kvm_amd.sev=1"> in /etc/default/grub to enable AMD SEV.
2. run 'sudo update-grub'.
3. reboot.

The ".to('cuda')" or ".cuda()" operations get stucked for Pytorch tensor: I tested in the terminal
```
import torch
a = torch.rand(5).cuda()
```
and got stucked (no response for a long time). Meanwhile, I founded that the GPU memory does not change and there was only one CPU core busy.

After I disabled the AMD SEV by restoring <GRUB_CMDLINE_LINUX_DEFAULT="quite splash">  in  /etc/default/grub, the bug disappeared.



### Versions

CPU: AMD Threadripper 3960x
OS: Ubuntu 22.04.01
Linux kernel:  5.15.0-52-generic
GCC version: gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0

Tested driver: nvidia-driver-510, nvidia-driver-520
Tested pytorch version: 1.12, 1.3 (latest)
Tested python version: 3.10.4, 3.10.6.


cc @ngimel @jeffdaily @sunway513 @jithunnair-amd @ROCmSupport
`torch.mm` Trigger RuntimeError with UndefinedBehaviorSanitizer
Labels: triaged, module: sanitizers

### 🐛 Describe the bug

A test case for `torch.mm` aborted because of runtime error when it's run with the undefined behavior sanitizer. Without sanitizers, the program terminated normally.
Reproduction:
```python
import torch

def test():
    arg_1_tensor = torch.rand([0, 64], dtype=torch.float32)
    arg_1 = arg_1_tensor.to_sparse()
    arg_2_tensor = torch.rand([64, 1], dtype=torch.float32)
    arg_2 = arg_2_tensor.to_sparse()
    res = torch.mm(arg_1,arg_2,)

test()
```
Error log:
```bash
/home/yuyao/dev/pytorch/c10/core/TensorImpl.h:1495:38: runtime error: applying non-zero offset 8 to null pointer
```
It seems that the error can be triggered only if we use sparse tensors.

### Versions

```
PyTorch version: 1.14.0a0+gita86278b
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: 11.1.0-6
CMake version: version 3.24.1
Libc version: glibc-2.35

Python version: 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-50-generic-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.5
[pip3] torch==1.14.0a0+gita86278b
[pip3] torchvision==0.13.1
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.21.5           py39he7a7128_1  
[conda] numpy-base                1.21.5           py39hf524024_1  
[conda] numpydoc                  1.2                pyhd3eb1b0_0
```
☂️  Issues that trigger crashes due to corner-case API usages
Labels: triaged, module: edge cases

### 🐛 Describe the bug

Does not seem like a real user issues, but detected using some sort of API fuzzing:
- https://github.com/pytorch/pytorch/issues/87960
- https://github.com/pytorch/pytorch/issues/87961
- https://github.com/pytorch/pytorch/issues/87963
- https://github.com/pytorch/pytorch/issues/87964
- https://github.com/pytorch/pytorch/issues/94594


Sanitizer detected issues:
- https://github.com/pytorch/pytorch/issues/88724
- https://github.com/pytorch/pytorch/issues/88940
- https://github.com/pytorch/pytorch/issues/88939

### Versions

None
Conv2d is not deterministic when input tensor has different strides
Labels: module: convolution, triaged

### 🐛 Describe the bug

I would expect that if I pass two identical tensors through a Conv2d in deterministic mode, they would produce an identical output. However this is not the case if the tensors are identical in every way except their stride - in that case the output is different. A difference in stride doesn't cause the two tensors to not be `torch.equal`, which is especially confusing - two "equal" tensors can produce a different output.

```python
import os
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"
import torch
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.benchmark = False


conv = torch.nn.Conv2d(3, 3, kernel_size=2).cuda()
in_a = torch.randn(4, 3, 64, 64).cuda()
out_a = conv(in_a)
in_b = torch.clone(in_a.permute(0, 2, 3, 1), memory_format=torch.contiguous_format).permute(0, 3, 1, 2)
out_b = conv(in_b)
print(in_a.stride())  # (12288, 4096, 64, 1)
print(in_b.stride())  # (12288, 1, 192, 3)
print(torch.equal(in_a, in_b)) # True
print(torch.equal(out_a, out_b))  # False
```

Ideally this should be fixed such that differences in stride don't affect the output of otherwise-deterministic operations, but at minimum the page on [reproducibility](https://pytorch.org/docs/stable/notes/randomness.html) should mention this. 

### Versions

Collecting environment information...
PyTorch version: 1.13.0+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.9.5 (default, Nov 23 2021, 15:27:38)  [GCC 9.3.0] (64-bit runtime)
Python platform: Linux-5.13.0-30-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 470.103.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] geotorch==0.2.0
[pip3] mypy==0.971
[pip3] mypy-boto3-ec2==1.17.41.0
[pip3] mypy-boto3-s3==1.17.41.0
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.2
[pip3] pytorch-lightning==1.7.3
[pip3] torch==1.13.0+cu116
[pip3] torchmetrics==0.7.0
[pip3] torchvision==0.13.0+cu113
[conda] Could not collect
AvgPool2D output shapes are inconsistent when ceil_mode=True
Labels: triaged, module: correctness (silent), module: pooling

### 🐛 Describe the bug

Certain inputs produce unexpected output shapes from [AvgPool2D](https://pytorch.org/docs/stable/generated/torch.nn.AvgPool2d.html) when `ceil_mode=True`.

## Example 1

```python
>>> t = torch.ones((1, 3, 32, 32))
>>> pool = torch.nn.AvgPool2d(8, stride=4, ceil_mode=True)
>>> pool(t).shape
torch.Size([1, 3, 7, 7])
```

This is surprising behavior because there are 8 possible windows with `stride=4` (not 7) for both height and width, each starting from the following indexes:
```
0
4
8
12
16
20
24
28
```

The final window at index 28 is a partial window, but should be included when `ceil_mode=True`, otherwise `ceil_mode` seems to serve no purpose. (This window does not start in a padding region as there is no padding used here.)

## Example 2
```python
>>> t = torch.ones((1, 3, 4, 4))
>>> pool = torch.nn.AvgPool2d(3, stride=3, ceil_mode=True)
>>> pool(t).shape
torch.Size([1, 3, 2, 2])
```

With `stride=3`, we have 2 potential start positions:
```
0
3
```

Torch chooses to use both of them. But based on the behavior observed in Example 1 (where the last frame was discarded) we would have expected output shape of `[1, 3, 1, 1]`.

### Versions

PyTorch version: 1.12.1+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Amazon Linux 2 (x86_64)
GCC version: (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15)
Clang version: Could not collect
CMake version: version 3.18.4
Libc version: glibc-2.2.5

Python version: 3.7.10 (default, Jun  3 2021, 00:02:01)  [GCC 7.3.1 20180712 (Red Hat 7.3.1-13)] (64-bit runtime)
Python platform: Linux-5.4.209-129.367.amzn2int.x86_64-x86_64-with-glibc2.2.5
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.20.0
[pip3] torch==1.12.1
[pip3] torch-neuron==1.12.1.2.0.0.0
[conda] Could not collect

cc @ezyang @gchanan @zou3519
Refactor `torch.return_types.topk` to behave like a `namedtuple` or a `dict`
Labels: triaged, enhancement, module: sorting and selection

### 🚀 The feature, motivation and pitch

When you use `torch.topk`, it returns a namedtuple-like object of type `torch.return_types.topk`. Unfortunately, it is not possible to use it as a dictionary. For example:

```python
import torch

x = torch.topk(torch.randn(10, 10), k=5)
assert isinstance(x, torch.return_types.topk)

# This is okay:
x.indices

# This will give an error:
x['indices']
```

Ideally, accessing the returned item as a dict would make it more useful and reduce the amount of post-processing needed. Alternatively, it could behave like a namedtuple by allowing the use of the documented method [`_asdict()`](https://docs.python.org/3/library/collections.html#collections.somenamedtuple._asdict) (which is officially supported and named this way to avoid collision). For example:

```python
# Potential improvement:
di = x._asdict()
di['indices']  # Should show the indices
di['values']  # should show the values
```

### Alternatives

_No response_

### Additional context

_No response_
Add eq, to, masked_select, index_select, narrow to nested tensors
Labels: triaged, module: nestedtensor

### 🚀 The feature, motivation and pitch

I am working on better implementation of SQL operators in [TQP](https://www.vldb.org/pvldb/vol15/p2811-he.pdf) and I am exploring how to leverage nested tensors for this. However in order to use nested tensors in practice without bloating the memory we would love to have support for `eq`, `to`, `masked_select`, `index_select` and `narrow`.

### Alternatives

Use `to_padded_tensor` but this will increase memory pressure.

### Additional context

_No response_

cc @cpuhrsch @jbschlosser @bhosmer @drisspg @mikaylagawarecki
Python Dispatcher registrations beyond BackendSelect do nothing
Labels: triaged, module: dispatch, module: python dispatcher

### 🐛 Describe the bug

Take https://github.com/albanD/subclass_zoo/blob/main/use_cpu_for_rng.py and replace BackendSelect with CUDA. The test fails, because we never hit python dispatcher. Probably BackendSelect needs to restore the Python Dispatcher key before redispatching.

### Versions

master
WIP: feat: LARS optimizer
Labels: module: optimizer, module: mkldnn, open source, release notes: nn, ciflow/inductor

Followup to [#6323](https://github.com/pytorch/vision/issues/6323).

Addition of LARS optimizer.

- [ ] LARS optimizer
- [ ] Tests
- [ ] Documentation
- [ ] Multi-Tensor support
- [ ] Extra params (e.g., ~~maximize~~, ~~differentiable~~, foreach)
- [ ] .pyi

Reference implementations: [[1](https://lightning-flash.readthedocs.io/en/0.5.0/_modules/flash/core/optimizers/lars.html#LARS)]

cc @vincentqb @jbschlosser @albanD @janeyx99 @crcrpar @gujinghui @PenghuiCheng @XiaobingSuper @jianyuh @jgong5 @mingfeima @sanchitintel @ashokei @jingxu10 @min-jean-cho @yanbing-j @Guobing-Chen @Xia-Weiwen @soumith @voznesenskym @penguinwu @anijain2305 @EikanWang @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @desertfire
ProcessGroupNCCL watchdog can't catch NCCL comm initialization issues
Labels: oncall: distributed, triaged, module: c10d

### 🐛 Describe the bug

ProcessGroupNCCL watchdog is designed to catch hanging collectives and tear down the process, but in the special case of first collective, ProcessGroupNCCL initializes NCCL communicators, which can hang. Since no comm future object has been created, watchdog cannot handle this situation. 

### Versions

main

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Add nondeterministic alert to `torch.Tensor.scatter()`
Labels: triaged, module: determinism

### 🐛 Describe the bug

``` python
torch.use_deterministic_algorithms(True)
input = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8, 9])
index = torch.tensor([0, 1, 1, 1, 2])
src = torch.tensor([11, 12, 13, 14, 15])
result = torch.Tensor.scatter(input, 0, index, src)
print(result)

inputCuda = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8, 9]).cuda()
indexCuda = torch.tensor([0, 1, 1, 1, 2]).cuda()
srcCuda = torch.tensor([11, 12, 13, 14, 15]).cuda()
resultCuda = torch.Tensor.scatter(inputCuda, 0, indexCuda, srcCuda)
print(resultCuda)
```

Run this code and you will get different results on CPU and on GPU: 

```
tensor([11, 14, 15,  4,  5,  6,  7,  8,  9])
tensor([11, 12, 15,  4,  5,  6,  7,  8,  9], device='cuda:0')
```

The value on position 1 is overwritten for 3 times with 12, 13 and 14. It looks like CUDA overwrites it in a reversed order of indices. This is not reported even with `torch.use_deterministic_algorithms(True)`. 

Related: #55516, #50750

### Versions

Collecting environment information...
PyTorch version: 1.12.1+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.22.6
Libc version: glibc-2.26

Python version: 3.7.15 (default, Oct 12 2022, 19:14:55)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: True
CUDA runtime version: 11.2.152
CUDA_MODULE_LOADING set to: 
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.12.1+cu113
[pip3] torchaudio==0.12.1+cu113
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.13.1
[pip3] torchvision==0.13.1+cu113
[conda] Could not collect

cc @mruberry @kurtamohler
convert torch.jit.script model to ONNX get wrong result
Labels: module: onnx, triaged, onnx-needs-info

### 🐛 Describe the bug

I want to convert  `torch.jit.script` model  to ONNX, however the inference of ONNX and raw pytorch is different. My code snippet：
```python
    import torch.nn as nn
    import torch
    import torch.nn.functional as F
    import numpy as np
    import onnxruntime as ort
    import pdb


    class test_net(nn.Module):
        def __init__(self,):
            super(test_net, self).__init__()
            self.model = nn.Conv3d(3,64,kernel_size=(1,3,3), stride=(2,1,2))
            self.relu = nn.ReLU()
            self.relu6 = nn.ReLU6()
            self.relu66 = nn.ReLU6()

        def forward(self, x):
            out1 = self.model(x)
            f_mean = torch.mean(out1) # -> ReduceMean
            out2 = torch.div(out1, f_mean)
            o1 = self.relu(out2)
            o2 = self.relu6(out2)
            o3 = self.relu66(out2)   # script模式下被优化掉了
            out = torch.cat([o1,o2,o3])

            return out

    # 模型构建和运行
    imgh, imgw = 24, 94
    net = test_net().eval() # 若存在batchnorm、dropout层则一定要eval() 使得BN层参数不更新
    dummy_input = torch.randn(1, 3, 3, imgh, imgw)# n c d h w
    torch_out = net.forward(dummy_input)# net(dummy_input)


    # export onnx
    dynamic_axes = {'input': {3: 'height', 4: 'width'}, 'output': {3: 'height', 4: 'width'}} # 配置动态分辨率
    onnx_pth = "test-conv-relu.onnx"

    
    # 传入 scriptModule
    net_script= torch.jit.script(test_net().eval()) # 生成一个 ScriptModule
    # 需要外加配置 example_outputs，用来获取输出的shape和dtype，无需运行模型
    torch.onnx.export(net_script, dummy_input, onnx_pth, input_names=['input'], output_names=['output'], opset_version=11, 
        dynamic_axes=dynamic_axes)
    # example_outputs=[torch_out]

    onnx_session = ort.InferenceSession(onnx_pth, providers=['CUDAExecutionProvider'])
    onnx_blob = dummy_input.data.numpy()
    onnx_out = onnx_session.run(None, {'input':onnx_blob})[0]

    print('mean diff = ', np.mean(onnx_out - torch_out.data.numpy()))
```
There is a significant difference between `onnx_out ` and  `torch_out`

PS:  It seems Pytorch 1.13  `torch.onnx.export` does not have `example_outputs`  parameter

### Versions

Pytorch Version: 1.13.0a0+08820cb
Cannot import `traverse_dps` from torch.data.utils.graph
Labels: triaged, module: data

### 🐛 Describe the bug

Cannot import `traverse_dps` from `torch.data.utils.graph`.

```
from torchtext.datasets import WikiText2
```
does not work, but instead throws the error:
```
[/usr/local/lib/python3.7/dist-packages/torchdata/datapipes/utils/_visualization.py](https://localhost:8080/#) in <module>
     11 
     12 from torch.utils.data.datapipes.iter.combining import _ChildDataPipe, IterDataPipe
---> 13 from torch.utils.data.graph import traverse_dps
     14 
     15 if TYPE_CHECKING:

ImportError: cannot import name 'traverse_dps' from 'torch.utils.data.graph' (/usr/local/lib/python3.7/dist-packages/torch/utils/data/graph.py)
```

### Versions

Using Google Collab. Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.13.0
[pip3] torchaudio==0.12.1+cu113
[pip3] torchdata==0.5.0
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.13.1
[pip3] torchvision==0.13.1+cu113
[conda] Could not collect

cc @VitalyFedyunin @ejguan @NivekT
Different behaviour in sparse matmul
Labels: module: sparse, triaged

The same code with and without the utilization of a **sparse** tensor gives an error:

The following is a working example of the code **without** sparse tensor:
``` python
a = torch.randn(2, 3, 3).requires_grad_(True)
print(a)

b = torch.randn(3, 1, requires_grad=True)
print(b)

y = torch.matmul(a, b)
print(y)

y.sum().backward()
print(a.grad)
```

This is the same example code with a sparse tensor utilization:
``` python
a = torch.randn(2, 3, 3).to_sparse().requires_grad_(True)
print(a)

b = torch.randn(3, 1, requires_grad=True)
print(b)

y = torch.matmul(a, b)
print(y)

y.sum().backward()
print(a.grad)
```

The code with the sparse variant gives the following error:

```
NotImplementedError                       Traceback (most recent call last)
 Cella 22 in <cell line: 7>()
      [4] b = torch.randn(3, 1, requires_grad=True)
      [5] print(b)
----> [7] y = torch.matmul(a, b)
      [8] print(y)
     [10] y.sum().backward()

NotImplementedError: Tensors of type SparseTensorImpl do not have is_contiguous
```

I'm not very confident that I am not wrong, but I think that it is better if matmul for dense and sparse tensor follows the same behaviour.  
If I do not understand something, how can I replicate the same behaviour in the sparse case? 

Thank you! 

### Versions

Collecting environment information...
PyTorch version: 1.11.0+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Could not collect
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.12 (tags/v3.9.12:b28265d, Mar 23 2022, 23:52:46) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19041-SP0
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to:
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] Could not collect
[conda] Could not collect

cc @nikitaved @pearu @cpuhrsch @amjames @bhosmer
Minifier doesn't work on DebertaForQuestionAnswering
Labels: triaged, bug

### 🐛 Describe the bug

Build pytorch e4a8661ab84022c1bff622c6d2f6e679180b1df5

Apply this patch

```
diff --git a/torch/_dynamo/optimizations/backends.py b/torch/_dynamo/optimizations/backends.py
index 660e7a5ca56..d53733bec73 100644
--- a/torch/_dynamo/optimizations/backends.py
+++ b/torch/_dynamo/optimizations/backends.py
@@ -552,7 +552,8 @@ def cudagraphs_inner(model, inputs, copy_outputs=True):
 def aot_autograd(subgraph, **kwargs):
     def _wrapped_bw_compiler(*args, **kwargs):
         # stop TorchDynamo from trying to compile our generated backwards pass
-        return disable(bw_compiler(*args, **kwargs))
+        bwd_func = disable(bw_compiler)(*args, **kwargs)
+        return disable(bwd_func)
 
     bw_compiler = kwargs.get("bw_compiler") or kwargs["fw_compiler"]
     kwargs["bw_compiler"] = _wrapped_bw_compiler
```

Deberta failures accuracy with aot_eager, but the minifier doesn't do anything

```
$ pp PYTHONUNBUFFERED=1 TORCHDYNAMO_REPRO_AFTER="dynamo" TORCHDYNAMO_REPRO_LEVEL=4  python benchmarks/dynamo/huggingface.py  --accuracy --backend aot_eager --training --only DebertaForQuestionAnswering
[2022-10-29 19:30:33,226] torch._dynamo.testing: [WARNING] High loss value alert - 6.63. Can result in unstable gradients.
cuda train DebertaForQuestionAnswering        [2022-10-29 19:30:33,509] torch._dynamo.testing: [WARNING] High loss value alert - 6.63. Can result in unstable gradients.
[2022-10-29 19:30:33,626] torch._dynamo.testing: [WARNING] High loss value alert - 6.63. Can result in unstable gradients.
[2022-10-29 19:30:33,731] torch._dynamo.testing: [WARNING] High loss value alert - 6.63. Can result in unstable gradients.
[2022-10-29 19:30:54,042] torch._dynamo.testing: [WARNING] High loss value alert - 6.63. Can result in unstable gradients.
[2022-10-29 19:30:54,050] torch._dynamo.utils: [ERROR] RMSE (res-fp64): 0.00016, (ref-fp64): 0.00000 and shape=torch.Size([50265, 768])
[2022-10-29 19:30:54,050] torch._dynamo.utils: [ERROR] Accuracy failed for key name deberta.embeddings.word_embeddings.weight.grad
```

### Error logs

_No response_

### Minified repro

_No response_
Inductor gives obscure error when FX graph to be compiled returns tuple
Labels: triaged, bug

### 🐛 Describe the bug

```
        var_mean = torch.ops.aten.var_mean.correction(add, [2], correction = 0, keepdim = True);  add = None
```

triggers an assert error in inductor

### Error logs

```
  File "/data/users/ezyang/pytorch-tmp2/torch/_inductor/graph.py", line 299, in run_node                    
    result = super().run_node(n)                                                                            
  File "/data/users/ezyang/pytorch-tmp2/torch/fx/interpreter.py", line 171, in run_node
    return getattr(self, n.op)(n.target, args, kwargs)         
  File "/data/users/ezyang/pytorch-tmp2/torch/_inductor/graph.py", line 267, in output         
    assert all(
AssertionError: ((TensorBox(StorageBox(
  Pointwise(
    'cuda',
    torch.float32,
    load(buf1, i1) / index_expr(1024, torch.float32),
    ranges=[1, 128, 1],
    origins={var_mean}
  )
)), TensorBox(StorageBox(
  Pointwise(
    'cuda',
    torch.float32,
    load(buf2, i1) / index_expr(1024, torch.float32),
    ranges=[1, 128, 1],
    origins={var_mean}
  )
))),)

While executing return ((getitem, getitem_1),)
```

### Minified repro

```
import torch._inductor.overrides
import torch
from torch import tensor, device
import torch.fx as fx
from torch._dynamo.testing import rand_strided
from math import inf
from torch.fx.experimental.proxy_tensor import make_fx
# torch version: 1.13.0a0+git3eb2722
# torch cuda version: 11.4
# torch git version: 3eb27229dd74dd0bea434326c471f16c50e558a4
# CUDA Info: 
# nvcc: NVIDIA (R) Cuda compiler driver 
# Copyright (c) 2005-2021 NVIDIA Corporation 
# Built on Sun_Aug_15_21:14:11_PDT_2021 
# Cuda compilation tools, release 11.4, V11.4.120 
# Build cuda_11.4.r11.4/compiler.30300941_0 
# GPU Hardware Info: 
# NVIDIA A100-PG509-200 : 8 
from torch.nn import *
class Repro(torch.nn.Module):
    def __init__(self):
        super().__init__()
    
    
    def forward(self, add):
        var_mean = torch.ops.aten.var_mean.correction(add, [2], correction = 0, keepdim = True);  add = None
        return (var_mean,)
        
args = [((1, 128, 1024), (131072, 1024, 1), torch.float32, 'cuda')]
args = [rand_strided(sh, st, dt, dev) for (sh, st, dt, dev) in args]
mod = make_fx(Repro().to(device="cuda"))(*args)
from torch._inductor.compile_fx import compile_fx_inner
from torch._dynamo.debug_utils import same_two_models
compiled = compile_fx_inner(mod, args)
assert same_two_models(mod, compiled, args, only_fwd=True), "Accuracy failed"
```
Turning on minifier causes bug to go away (on DebertaForMaskedLM)
Labels: triaged, bug

### 🐛 Describe the bug

1. Check out and build pytorch at e4a8661ab84022c1bff622c6d2f6e679180b1df5 (Oct 29, before breaking commit was reverted)
2. Run `python benchmarks/dynamo/huggingface.py  --accuracy --backend inductor --training --only DebertaForMaskedLM`, it fails
3. Run `TORCHDYNAMO_REPRO_AFTER=dynamo python benchmarks/dynamo/huggingface.py  --accuracy --backend inductor --training --only DebertaForMaskedLM`, it passes

### Error logs

https://www.internalfb.com/intern/paste/P543371878/

### Minified repro

minifier did not work
A segment fault can be triggered in fbgemm_pack_gemm_matrix_fp16
Labels: module: crash, triaged, module: third_party, module: half

### 🐛 Describe the bug

The following code can trigger a segment fault:
````python
import torch
import numpy as np
print(torch.__version__)
input = torch.rand([12, 10, 0, 8, 7], dtype=torch.float32)
res = torch.fbgemm_pack_gemm_matrix_fp16(
    input
)
````
Output:
````
1.13.0+cu117
Segmentation fault (core dumped)
````

### Versions

PyTorch version: 1.13.0+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 11.0.0-2~ubuntu20.04.1
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.10.6 (main, Oct 24 2022, 16:07:47) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla V100-PCIE-16GB
Nvidia driver version: 520.61.05
cuDNN version: Probably one of the following:
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn.so.8.1.1
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.1.1
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.1.1
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.1.1
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.1.1
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.1.1
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8.2.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.2.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.2.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.2.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.2.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.2.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.2.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.4
[pip3] torch==1.13.0
[conda] numpy                     1.23.4                   pypi_0    pypi
[conda] torch                     1.13.0                   pypi_0    pypi
Einsum Optimization Tracker
Labels: module: performance, triaged, module: linear algebra

There are many things one can do to improve einsum (in terms of perf and usability!) so this issue seeks to track them all. 

## The Multi-Contractions Case
As of torch 1.13, `torch.einsum` automatically uses opt-einsum to optimize the contraction path for when there are 3+ tensors to contract and opt-einsum is installed. As of now, section 1 of https://github.com/pytorch/pytorch/issues/60295 has been completed. 
Tasks remaining include to:
- [ ] Achieve numpy compatibility by adding optimize as an torch.einsum API kwarg (our previous attempt was blocked by failing JIT tracing https://github.com/pytorch/pytorch/pull/85908)
- [ ] [P0] Submit a fix to https://github.com/dgasmith/opt_einsum to set `torch.backends.opt_einsum.enabled = False` to avoid recomputing the path when opt_einsum calls torch.einsum as a backend. https://github.com/dgasmith/opt_einsum/pull/205
- [ ] Test out cuQuantum perf as an opt_einsum replacement
- [ ] Add path computation caching by equation + op size

## The Single Contraction Case
The path to faster single contractions includes:
- [ ] Benchmarking einsum with torch.inductor, experimenting with replacing bmm with mul -> sum for discontiguous tensors.
- [ ] Testing out cuTensor perf (to see if there are significant improvements since Yaroslav's last benchmarks in #57121.)

## Additional Context
See previous relevant issues https://github.com/pytorch/pytorch/issues/60295 and https://github.com/pytorch/pytorch/issues/57121 for context.

cc @VitalyFedyunin @ngimel @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano
multi-node distributed training rank0 hang at dataloader after a few epochs
Labels: oncall: distributed, module: dataloader

### 🐛 Describe the bug

I am using pytorch to do multi-GPU distributed training. When I use 1-node 8-GPUs, the training works well. However, when I used 4-nodes 32-GPUs, after N epochs (N can be 1 to 3), the training will freeze. From the debugger, it shows that rank0 is stuck at dataloader place with the following stack. And all other 31 ranks are waiting for rank0.  For this hang case, dataloader uses num_workers=8. When I changed num_workers=4, the job can run longer before hang. When I changed num_workers=0, the job won't hang.

This is 100% reproducible. I ran the same tests about 20 times.

Each time it will stuck at the same code place: rank0 will hang at dataloader, all other ranks will wait because of all_reduce(). When this hangs, rank0 already finished several steps (e.g. X is 5).
It always stuck at rank0.

The stack for rank0:
```
Thread 0x7FA6E7C01740 (idle): "MainThread"
    wait (threading.py:302)
    get (queue.py:170)
    __next__ (fairseq/data/iterators.py:648)
    __iter__ (fairseq/data/iterators.py:59)
    _chunk_iterator (fairseq/data/iterators.py:528)
    __iter__ (fairseq/data/iterators.py:59)
    __iter__ (fairseq/logging/progress_bar.py:256)
    train (fairseq_cli/train.py:274)
    inner (contextlib.py:75)
    main (fairseq_cli/train.py:165)
    distributed_main (fairseq/distributed/utils.py:326)
    call_main (fairseq/distributed/utils.py:352)
    cli_main (fairseq_cli/train.py:499)
    <module> (train.py:14)
```
The stack for other ranks:
```
Thread 0x7F6F32481740 (active): "MainThread"
    _all_reduce_dict (fairseq/distributed/utils.py:661)
    all_reduce_dict (fairseq/distributed/utils.py:667)
    _fast_stat_sync_sum (fairseq/trainer.py:1195)
    _aggregate_logging_outputs (fairseq/trainer.py:1135)
    train_step (fairseq/trainer.py:708)
    inner (contextlib.py:75)
    train (fairseq_cli/train.py:278)
    inner (contextlib.py:75)
    main (fairseq_cli/train.py:165)
    distributed_main (fairseq/distributed/utils.py:326)
    call_main (fairseq/distributed/utils.py:352)
    cli_main (fairseq_cli/train.py:499)
    <module> (train.py:14)
```

### Versions

pytorch 1.11
OS: Linux Ubuntu 20.4
Python version: 3.8
CUDA: 11.5
GPU: A100 80GB
fairseq: 1.0.0a0


cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @SsnL @VitalyFedyunin @ejguan @NivekT
`torch.distributed` crash with abort only inside if
Labels: oncall: distributed, module: crash

### 🐛 Describe the bug

`torch.distributed` crash with abort:
```
import torch
import torch.distributed as dist
if True:
    input_tensor = torch.randn(4, 3)
    dist.FileStore('./')
```
```
abort (core dumped) 
```
However, if we remove the if statement, it will not crash
```
import torch
import torch.distributed as dist
input_tensor = torch.randn(4, 3)
dist.FileStore('./')
```
```
<torch._C._distributed_c10d.FileStore at 0x7f5b68e42b70>
```

### Versions

pytorch 1.12.1

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
crash in `torch.package.PackageExporter`
Labels: triaged, module: edge cases

### 🐛 Describe the bug

`torch.package.PackageExporter` crash with abort.

```
import torch
data = torch.randn(2, 3)
print(data)
torch.package.PackageExporter(data, './test_package.zip')
```
```
tensor([[-1.3200, -1.0457, -0.0773],
        [ 0.6770,  1.8295, -0.8031]])
abort (core dumped) 
```
This is probably related to https://github.com/pytorch/pytorch/issues/85329, except that in the code example above, no Error is thrown and it directly crashes.

### Versions

pytorch 1.12.1
crash when call `torch.set_num_interop_threads` twice
Labels: module: crash, triaged

### 🐛 Describe the bug

Crash with abort when call `torch.set_num_interop_threads` more than once.

According to [doc](https://pytorch.org/docs/stable/generated/torch.set_num_interop_threads.html), `set_num_interop_threads`:
> can only be called once and before any inter-op parallel work is started (e.g. JIT execution).

However, I think this can be handled with an Exception instead of directly crash.

### Versions

pytorch 1.12.1

cc @ezyang @gchanan @zou3519
VS2022Preview ParallelCommon.cpp.obj : fatal error LNK1161: invalid export specification
Labels: module: build, triaged, module: static linking

### 🐛 Describe the bug

```
[5554/5844] Linking CXX shared library bin\torch_cpu.dll
FAILED: bin/torch_cpu.dll lib/torch_cpu.lib
cmd.exe /C "cd . && C:\Prog\cmake\bin\cmake.exe -E vs_link_dll --intdir=caffe2\CMakeFiles\torch_cpu.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MICROS~4\2022\Preview\VC\Tools\MSVC\1434~1.319\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\torch_cpu.rsp  /out:bin\torch_cpu.dll /implib:lib\torch_cpu.lib /pdb:bin\torch_cpu.pdb /dll /version:0.0 /machine:x64 /ignore:4049 /ignore:4217 /ignore:4099 /INCREMENTAL:NO  -WHOLEARCHIVE:C:/Github/pytorch/build/lib/caffe2_protos.lib -WHOLEARCHIVE:C:/Github/pytorch/build/lib/onnx.lib  && cd ."
LINK: command "C:\PROGRA~1\MICROS~4\2022\Preview\VC\Tools\MSVC\1434~1.319\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\torch_cpu.rsp /out:bin\torch_cpu.dll /implib:lib\torch_cpu.lib /pdb:bin\torch_cpu.pdb /dll /version:0.0 /machine:x64 /ignore:4049 /ignore:4217 /ignore:4099 /INCREMENTAL:NO -WHOLEARCHIVE:C:/Github/pytorch/build/lib/caffe2_protos.lib -WHOLEARCHIVE:C:/Github/pytorch/build/lib/onnx.lib /MANIFEST /MANIFESTFILE:bin\torch_cpu.dll.manifest" failed (exit code 1161) with the following output:
ParallelCommon.cpp.obj : fatal error LNK1161: invalid export specification
```

after some comparsion
VS2022Preview 17.4.0 Preview 5.0
emit this
```
/EXPORT:?init@?1??lazy_init_num_threads@internal@at@@YAXXZ@4_NA,THREAD_DATA
```
while
VS2019 16.11.20
```
/EXPORT:?init@?1??lazy_init_num_threads@internal@at@@YAXXZ@4_NA,DATA 
```

here the object files
[obj_files.zip](https://github.com/pytorch/pytorch/files/9885294/obj_files.zip)

Workaround?
idk

Thanks.

CC @vladimir-aubrecht 

### Versions

pytorch trunk
commit f9679184116f1d29c483c2b2a4c3a9d730be4694
VS2022
```
C:\Github\pytorch>cl -v
Microsoft (R) C/C++ Optimizing Compiler Version 19.34.31932 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.
```
VS2019
```
C:\Github\pytorch_VS2019>cl -v
Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30146 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.
```

cc @malfet @seemethere
Autograd doesn't stop executing backward graph early enough in situations involving set_
Labels: module: autograd, triaged, has workaround, actionable

### 🐛 Describe the bug

Before the bug, the reproducing program is a bit whacky, so I need to explain why it might be reasonable to expect it not to error.

Suppose I have a naughty custom autograd function, that returns gradients of the wrong size. If I use grad() to cut off autograd computation before I actually get to the naughty autograd function, I would expect to never actually hit the error case. And indeed, this program runs successfully:

```
import torch

class F(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        return x.clone()

    @staticmethod
    def backward(ctx, grad_output):
        return torch.zeros(6, requires_grad=True)

x = torch.zeros(4, requires_grad=True)
y = F.apply(x)
z = y.clone()

print(torch.autograd.grad((z,), (y,), grad_outputs=torch.ones_like(z)))
```

In general, I would expect that no matter how naughty the history of how x was created, so long as I grad with respect to it as an input, it should be "as if" I had detached it from the rest of the graph, all sins absolved.

Here is a program for which I am too naughty, and this invariant does not hold:

```
x = torch.zeros(0)
x.requires_grad = True
x2 = x.clone()
with torch.no_grad():
    x2.set_(torch.zeros(5).storage(), 0, (5,), (1,))
y = x2.clone()

print(torch.autograd.grad((y.sum(),), (x2,)))
```

Fails with:

```
  File "auto.py", line 26, in <module>
    print(torch.autograd.grad((y.sum(),), (x2,)))
  File "/raid/ezyang/pytorch-scratch2/torch/autograd/__init__.py", line 300, in grad
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Function CloneBackward0 returned an invalid gradient at index 0 - got [5] but expected shape compatible with [0]
```

This minimal repro is from real code extracted from MetaConverter (written by yours truly) which tries to synthesize variables which resemble input variables as much as possible in all properties. Note that if you ablate the `no_grad`, the program passes, but I am not allowed to do this as in some real programs x may be a leaf variable.

I have a workaround for this problem, which is to allocate an appropriately sized x from the get go, so this is not a blocker to fix. But it would be good to better understand what is going on here, in case there is a more serious bug.

cc @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 

### Versions

master
AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute 'next'
Labels: module: dataloader, triaged

### 🐛 Describe the bug

the same code can run in pytorch11.3， but can’t in 11.7.。。
 inputs_x, targets_x, index_x = labeled_train_iter.next()
AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute 'next'

### Versions

version
cuda11.8
cudnn8.6.0
python3.8
torch version：1.14.0


cc @SsnL @VitalyFedyunin @ejguan @NivekT
link error happen when intergrate libtorch to other tool
Labels: module: build, module: cpp, triaged

Hi, author. I recently try to integrate libtorch to a symbolic execution tool [klee](https://github.com/klee/klee), but always fail to linke libtorch in the last stage.

The part of CMakeList is

```text
set(Torch_DIR ${LIB_TORCH_ROOT}/share/cmake/Torch)
find_package(Torch REQUIRED)

if (Torch_FOUND)
    message(STATUS "Torch library found!")
    message(STATUS "include path: ${TORCH_INCLUDE_DIRS}")
    message(STATUS "libraries path: ${TORCH_LIBRARIES}")
else ()
    message(FATAL_ERROR "Could not locate Torch" \n)
endif()

xxxx

target_link_libraries(KleeCore "${TORCH_LIBRARIES}")
``` 

This could work when I write a simple libtool, but fail to link intergrating it to klee.

I print the value of `${TORCH_LIBRARIES}$`, and found the value is `torch;torch_library;<path2libtorch>/lib/libc10.so`, is this correct?  I also notice there is a variable `${TORCH_LIBRARY}` in `TorchConfig.cmake`, is that variable useful for cmake? How could I fix this error?

cc @malfet @seemethere @jbschlosser