gopikrsmscs/torch-issues huggingface dataset:
[MPS] `.to('mps')` zeroes out elements in tensors taking up >=2^32 bytes
Labels: triaged, module: mps

### 🐛 Describe the bug

`.to('mps')` zeroes out elements in float or int tensors that at least 2^32 bytes in size. In the example below, all assertions pass.
```python
import torch

def test_to(dtype):
    element_size = torch.ones(1, dtype=dtype).element_size()
    width = 32_768
    height = 2**32 // width // element_size

    lt_2to32 = torch.ones(width-1, height, dtype=dtype)
    eq_2to32 = torch.ones(width,   height, dtype=dtype)
    assert element_size * lt_2to32.numel()  < 2**32     # 2^31.9999 bytes
    assert element_size * eq_2to32.numel() == 2**32     # 2^32      bytes

    assert torch.all(lt_2to32.to("mps").to("cpu") == 1) # 2^31.9999 bytes -> all ones
    assert torch.all(eq_2to32.to("mps").to("cpu") == 0) # 2^32      bytes -> all zeros

    del lt_2to32, eq_2to32

test_to(torch.float16)
test_to(torch.float32)
test_to(torch.int32)
test_to(torch.int64)
```

```

### Versions

PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.2.1 (arm64)
GCC version: Could not collect
Clang version: 15.0.7
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.6 (main, Aug 29 2022, 10:06:59) [Clang 13.1.6 (clang-1316.0.21.2.5)] (64-bit runtime)
Python platform: macOS-13.2.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Max

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.24.2
[pip3] torch==1.13.1
[pip3] torchaudio==2.0.0
[pip3] torchtext==0.13.1
[pip3] torchvision==0.15.0
[conda] Could not collect

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
No GPU found, using CPU during preprocessing Error processing dataset with NsfHifiGAN 
Labels: needs reproduction, module: windows, module: cuda, triaged

### 🐛 Describe the bug

Description

I'm trying to process a dataset using the extract_features.py script in Python, which uses the NsfHifiGAN model to generate audio features. However, when I run the script on my GPU machine, I get the following error message:

```
`python` tools/preprocessing/extract_features.py --config configs/svc_hubert_soft.py --path dataset --clean
2023-03-13 20:45:32.352 | INFO     | __main__:<module>:213 - Using 1 workers
2023-03-13 20:45:32.352 | WARNING  | __main__:<module>:218 - No GPU found, using CPU
2023-03-13 20:45:32.352 | INFO     | __main__:<module>:221 - Cleaning *.npy files...
2023-03-13 20:45:32.383 | INFO     | __main__:<module>:227 - Done!
2023-03-13 20:45:32.586 | INFO     | __main__:<module>:231 - Found 476 files, processing...
  0%|                                                                                          | 0/476 [00:00<?, ?it/s]2023-03-13 20:45:32.586 | INFO     | __main__:init:50 - Rank 0 uses device cpu
No CUDA runtime is found, using CUDA_HOME='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7'
Using cache found in C:\Users\MSI/.cache\torch\hub\bshall_hubert_main
2023-03-13 20:45:34.221 | ERROR    | __main__:safe_process:192 - Error processing dataset\train\96.wav
2023-03-13 20:45:34.222 | ERROR    | __main__:safe_process:193 - class `NsfHifiGAN` in fish_diffusion/modules/vocoders/nsf_hifigan/nsf_hifigan.py: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the `CPU.`
```
Steps to reproduce

1. Run the following command:

`python tools/preprocessing/extract_features.py --config configs/svc_hubert_soft.py --path dataset --clean`

2.  Expected behavior

The script should process the dataset without any errors.

3. Actual behavior

The script throws the above-mentioned error message.

4. Workarounds attempted

I tried running these codes to set the device to GPU, as follows:
`pip uninstall torch torchaudio torchvisison -y`
`pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu117`

However, this did not resolve the issue. It almost froze my computer instead when it says collecting torch

Thank you in advance for any help or suggestions you can provide. Please let me know if you need any further information.


### Versions

Additional information

I'm running Python 3.10 on a Windows 11 machine.
I have installed PyTorch and other dependencies as required.
The dataset consists of 476 audio .wav files based on my voice.
My NVIDIA model with CUDA drivers
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_19:00:59_Pacific_Daylight_Time_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0



cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @ngimel
[FSDP] Make FSDP support local optimizer state_dict
Labels: oncall: distributed, triaged, module: fsdp

### 🚀 The feature, motivation and pitch

The current FSDP support FULL_STATE_DICT, LOCAL_STATE_DICT, and SHARDED_STATE_DICT for model state_dict. However, for optimizer state_dict, FSDP only supports FULL_STATE_DICT and SHARDED_STATE_DICT -- the local state_dict version is missing. It is hard for users to do distributed checkpoint if the returned local state_dict is not wrapped with ShadedTensor or DTensor.

**Code pointer:**
1. Add the flag for local_state_dict version for `_optim_state_dict` in _optim_utils.py:
2. Adding a conversion from tensor to ShardedTensor wrapping in 
https://github.com/pytorch/pytorch/blob/31137a63a75b40d9721544cc22843ede6b12e79d/torch/distributed/fsdp/_optim_utils.py#L1464
The subsequent logic also needs to be adjusted for local_state_dict. 


### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Harden composable fully_shard: Checklist
Labels: oncall: distributed, triaged, module: fsdp

### 🚀 The feature, motivation and pitch

The following are features that should be checked / hardened in order to roll out fully_shard as an alternative to class-based FSDP:

[ ] Test with ShardedGradScaler 
[ ] Test clip_grad_norm support
[ ] End to end and unittest for optimizer state checkpointing
[ ] support + API for summon_full_params
[ ] able to print to debug model wrapping structure like class-based FSDP
[ ] Any integration needed with torch.distributed.checkpoint

... more to be added as uncovered.

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
PyTorch SGEMV is using 1 single core on AMD CPUs (very slow)
Labels: module: rocm, triaged

### 🐛 Describe the bug

```python
import torch
import time

a = torch.randn((2048, 2048))
b = torch.randn((2048, 2048))
v = torch.randn((2048,))
c = torch.randn((2048, 2))
d = torch.randn((2048, 4))
e = torch.randn((2048, 8))

print("Warmup")
for i in range(5):
    a @ b
    a @ v
    a @ c
    a @ d
    a @ e

print("Bench large mat @ large mat")
totaldur = 0
N = 25
for i in range(N):
    start = time.time()
    a @ b
    totaldur += time.time() - start
print("Average: %.3fms" % (totaldur / N * 1000,))
print()
print("Bench large mat @ vec")
totaldur = 0
N = 1000
for i in range(N):
    start = time.time()
    a @ v
    totaldur += time.time() - start
print("Average: %.1fus" % (totaldur / N * 1000000,))
print()
print("Bench large mat @ 2 columns")
totaldur = 0
for i in range(N):
    start = time.time()
    a @ c
    totaldur += time.time() - start
print("Average: %.1fus" % (totaldur / N * 1000000,))
print()
print("Bench large mat @ 4 columns")
totaldur = 0
for i in range(N):
    start = time.time()
    a @ d
    totaldur += time.time() - start
print("Average: %.1fus" % (totaldur / N * 1000000,))
print()
print("Bench large mat @ 8 columns")
totaldur = 0
for i in range(N):
    start = time.time()
    a @ e
    totaldur += time.time() - start
print("Average: %.1fus" % (totaldur / N * 1000000,))
```
24-core 48-thread Threadripper 2970WX @ 3GHz
128GB RAM, DDR4 2800MHz
Output:
```
Warmup
Bench large mat @ large mat
Average: 64.457ms

Bench large mat @ vec <====== THIS IS VERY SLOW
Average: 1271.8us

Bench large mat @ 2 columns <====== THIS IS SLOW TOO
Average: 351.3us

Bench large mat @ 4 columns <====== THIS IS SLOW TOO
Average: 356.3us

Bench large mat @ 8 columns
Average: 502.6us
```

### Versions

Note: this is using torch rocm, but torch cpu has the same issue

```
Collecting environment information...
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
PyTorch version: 1.13.1+rocm5.2
Is debug build: False
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: 5.2.21151-afdc89f8

OS: NixOS 23.05 (Stoat) (x86_64)
GCC version: (GCC) 12.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.10 (main, Feb  7 2023, 12:19:31) [GCC 12.2.0] (64-bit runtime)
Python platform: Linux-6.1.15-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: AMD Radeon Graphics
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: 5.2.21151
MIOpen runtime version: 2.17.0
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] facenet-pytorch==2.5.2
[pip3] numpy==1.24.1
[pip3] torch==1.13.1+rocm5.2
[pip3] torchaudio==0.13.1+cpu
[pip3] torchvision==0.14.1+cpu
[conda] Could not collect
```

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport
Not allow force merge when lint fails and not because of broken trunk
Labels: module: ci, triaged

### 🐛 Describe the bug

### Motivation

Lint jobs are not flaky (https://github.com/pytorch/pytorch/issues/93156, https://github.com/pytorch/pytorch/pull/94255) and failures usually end up with reverts (https://github.com/pytorch/pytorch/pull/96580, https://github.com/pytorch/pytorch/pull/96612).  Even when force merging, it's a good practice to wait until the jobs finish first. 

Arguably, not allowing force merge when lint fails is a better UX than getting the PR reverted "only" because of lint.

#### Proposal

Not allow force merge when lint fails not because of broken trunk.  If the latter happens where lint jobs are already broken in trunk, these failures are likely not related to the PR.

### Versions

PyTorch CI

cc @seemethere @malfet @pytorch/pytorch-dev-infra @ZainRizvi
Request for adding Warning/Error feature when dropout set to 1.0 in Transformer layer
Labels: oncall: transformer/mha

### 🚀 The feature, motivation and pitch

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">The model is not &quot;turned off during training&quot;. With dropout=1.0, for dropout layers you&#39;ll get all zero at train and, apparently, identity at test. I don&#39;t think pytorch should have allowed dropout=1.0. It should be ValueError, not sure I get the reasoning there.</p>&mdash; Andrej Karpathy (@karpathy) <a href="https://twitter.com/karpathy/status/1635062921398206464?ref_src=twsrc%5Etfw">March 12, 2023</a></blockquote> 

As suggested by Andrej Karpathy, it makes sense to raise an error or atleast a warning when the user sets dropout in the transformer layer.  So it would be great if someone can decide whether it should raise an error or warning , and which warning or error to raise in the above case.

P.S :  I would like to be **assigned** the task for **implementation** of this issue.

### Alternatives

_No response_

### Additional context
Could you look into this? @ezyang @gchanan @jerryzh168
_No response_

cc @jbschlosser @bhosmer @cpuhrsch @erichan1
Dataloader should kill & restart workers when timeout is hit
Labels: module: dataloader, triaged

### 🚀 The feature, motivation and pitch

When using `timeout`, instead of crashing when the timeout is hit, the Dataloader should instead kill and restart problematic workers. Ideally, the worker should also be able to report the stack frame it is stuck on when being killed. This would be extremely useful for debugging code that works when `num_workers=0` but doesn't when `num_workers>0`. It also can save quite a bit of frustration when training hangs.

### Alternatives

_No response_

### Additional context

Currently, I am running into an issue where the dataloader keeps hanging. I suspect it is something to do with my machine, but have no way to debug it since the workers hang instead of crash. No error is reported when `timeout` is used, and I do not run into any issues with using `num_workers=0`. Hence, having the workers be killed on hang would be great.

cc @SsnL @VitalyFedyunin @ejguan @NivekT @dzhulgakov
[fix] angle for -0.0
Labels: module: cpu, module: mkldnn, open source, NNC, release notes: quantization, topic: not user facing, ciflow/mps, module: inductor, module: dynamo, ciflow/inductor, module: export

Fixes #78413


cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @gujinghui @PenghuiCheng @jianyuh @min-jean-cho @yanbing-j @Guobing-Chen @Xia-Weiwen @EikanWang @voznesenskym @penguinwu @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @anijain2305
Build errors in two Vulkan files
Labels: oncall: mobile, module: vulkan

### 🐛 Describe the bug

Build problems occur in two files with GCC 12.2.0:
```
aten/src/ATen/native/vulkan/impl/Arithmetic.cpp:20:1: error: control reaches end of non-void function [-Werror=return-type]
aten/src/ATen/native/vulkan/impl/Registry.cpp:36:27: error: loop variable 'key' of type 'const std::string&' {aka 'const std::__cxx11::basic_string<char>&'} binds to a temporary constructed from type 'const char* const' [-Werror=range-loop-construct]
```

### Versions

I am trying to build v2.0.0-rc6 from the release/2.0 branch.
Tensor Permutation Along Given Axis
Labels: feature, triaged, module: python frontend

### 🚀 The feature, motivation and pitch

The current `torch.perm` gives shared permutation ordering like the example below:
```python
import torch

# permute on the second dimension
x = torch.randn(3, 5, 4)
perm = torch.randperm(x.size(dim))
shuffled_x = x[:, perm, :]
```
The perm will shuffle the second dimension in the same ordering that is given by perm, which is the same of each other dimensions. 

It will be nice to have a function that permutes that given dimension of `x` **independently**.

### Alternatives

I've write a possible solution that has been tested by myself, and it works as intended.

```python
import torch

def shuffle_tensor(x: torch.Tensor, dim: int = 0) -> torch.Tensor:
    """
    Shuffle the tensor along the specified dimension independently 
    for each slice along that dimension.
    
    [NOTE] this is different to torch.perm which is shared permutation ordering.
    
    For example, for a 2D tensor:
        x = tensor([[0, 0, 1, 6, 2],
                    [5, 4, 7, 2, 4]])
            
    Aftering shuffling `x` becomes:
            tensor([[0, 0, 6, 2, 1],
                    [5, 7, 2, 4, 4]])
                    
    This works for any given dimension `dim` for an arbitary shape input tensor `x`.
    
    Args:
        x (torch.Tensor): The tensor to shuffle.
        dim (int, optional): The dimension along which to shuffle. Default is 0.

    Returns:
        torch.Tensor: The shuffled tensor.
    """
    # Get the shape of the slice of x along the specified dimension
    x_shape = tuple(x.shape[:dim+1])

    # Get the indices that would sort a random tensor along the specified dimension
    indices = torch.argsort(torch.rand(x_shape), dim=dim)
    
    # Add singleton dimensions to indices to match the shape of x
    n = x.dim() - dim - 1   
    indices = indices[(..., ) + (None,) * n] 
    indices = indices.to(x.device) * torch.ones_like(x)
    indices = indices.long()
    
    # Shuffle the tensor using gather
    shuffled_x = torch.gather(x, dim=dim, index=indices)
    return shuffled_x
```

Hopefully this can be added to the pytorch update.

Thanks,
Jie

### Additional context

_No response_

cc @albanD
[MPS] Incorrect results for cumsum with bool tensors
Labels: triaged, module: mps

### 🐛 Describe the bug

MPS produces incorrect results for cumsum with bool tensors. To reproduce this issue, run cumsum on the same bool tensor with CPU and MPS and compare the result.

```python
import torch

input = torch.randn(1000,2) < 0.5
mps_result = input.to('mps').cumsum(0)
cpu_result = input.cumsum(0)
torch.testing.assert_close(mps_result, cpu_result, check_device=False)
```

Output:
```
Traceback (most recent call last):
  File "/Users/brkirch/Desktop/test.py", line 6, in <module>
    torch.testing.assert_close(mps_result, cpu_result, check_device=False)
  File "/Users/brkirch/stable-diffusion-webui/venv-torch-nightly/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1514, in assert_close
    raise error_metas[0].to_error(msg)
AssertionError: Tensor-likes are not equal!

Mismatched elements: 1619 / 2000 (81.0%)
Greatest absolute difference: 768 at index (913, 1)
Greatest relative difference: 2.0 at index (190, 1)
```

A workaround is to cast the bool tensor to int32 before the cumsum operation. Also, MPS support for int64 with cumsum is currently broken on macOS 13.3 beta, see #96610.

### Versions

Collecting environment information...
PyTorch version: 2.1.0.dev20230311
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.3 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.202)
CMake version: version 3.25.1
Libc version: N/A

Python version: 3.10.10 (main, Feb  8 2023, 05:34:50) [Clang 14.0.0 (clang-1400.0.29.202)] (64-bit runtime)
Python platform: macOS-13.3-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Max

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] open-clip-torch==2.7.0
[pip3] torch==2.1.0.dev20230311
[pip3] torchvision==0.15.0.dev20230311
[conda] Could not collect

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
The output of torch.histc is incorrect on both CPU and CUDA
Labels: triaged, module: edge cases

### 🐛 Describe the bug

```python3
import torch
torch.set_printoptions(linewidth=160)
print("case 1", torch.histc(torch.linspace(0, 1, 2**28, device="cuda"), bins=3))
print("case 2", torch.histc(torch.linspace(0, 1, 2**28), bins=3))
print("case 3", torch.histc(torch.linspace(0, 1, 2**28, device="cuda"), bins=2048, min=0, max=2047))
print("case 4", torch.histc(torch.linspace(0, 1, 2**28), bins=2048, min=0, max=2047))
```

The output of case 2, 3, 4 are wrong.

```
case 1 tensor([89478488., 89478488., 89478480.], device='cuda:0')
case 2 tensor([50331648., 55924060., 50331648.])
case 3 tensor([16777216.,   131081.,        0.,  ...,        0.,        0.,        0.], device='cuda:0')
case 4 tensor([1.3422e+08, 1.3108e+05, 0.0000e+00,  ..., 0.0000e+00, 0.0000e+00, 0.0000e+00])
```

Expected results:
```python3
import numpy as np
print("case 1 & 2", np.histogram(np.linspace(0, 1, 2**28), bins=3)[0])
print("case 3 & 4", np.histogram(np.linspace(0, 1, 2**28), bins=2048, range=(0, 2047))[0])
'''
case 1 & 2 [89478485 89478485 89478486]
case 3 & 4 [268304384    131072         0 ...         0         0         0]
'''
```

### Versions

```
Collecting environment information...
PyTorch version: 1.13.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 专业版
GCC version: (x86_64-posix-sjlj-rev0, Built by MinGW-W64 project) 12.2.0
Clang version: 14.0.6
CMake version: version 3.25.3
Libc version: N/A

Python version: 3.10.10 (tags/v3.10.10:aad5f6a, Feb  7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19045-SP0
Is CUDA available: True
CUDA runtime version: 12.0.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3080
Nvidia driver version: 528.49
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=2904
DeviceID=CPU0
Family=198
L2CacheSize=2048
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2904
Name=Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] numpy==1.24.2
[pip3] torch==1.13.1+cu117
[pip3] torchaudio==0.13.1+cu117
[pip3] torchvision==0.14.1+cu117
[conda] Could not collect
```
No matching distribution found for torch==1.13.1+cu117
Labels: oncall: binaries, triaged

### 🐛 Describe the bug

Hi, I am trying to install pytorch with the specific CUDA version of 11.7 and I can only use pip for now. But the recommended command on the official website for Linux-pip-python-cuda 11.7 is a general command “pip3 install torch torchvision torchaudio” and this command always gave me CUDA version of 11.4 which is not what I want. However, when I tried to specify the version based on other commands suggested from this forum or other forum (e,g, github, StackOverflow etc.), it always said something about the 403 forbidden error and no matching distribution found for torch==1.13.1+cu117. I have attached a screenshot as the example. Could anybody please help me?


![image](https://user-images.githubusercontent.com/115084467/224496765-2bfcc126-3005-4a97-a8df-0e5d29b492b4.png)

My python version is either 3.8.10 or 3.10.2 (I tried both and none of them worked) and pip version is either 21.2.4 or 23.0.1 (default and updated version, I tried both and none of them worked)

Many thanks in advance.

### Versions

I am actually using a cloud computer so I don't have any rights to download this script, I am very sorry about this.

cc @ezyang @seemethere @malfet
Why doesn't PyTorch install the REAL nvidia cuDNN pip package?
Labels: oncall: binaries, module: cuda, triaged

### 🐛 Describe the bug

I requested that PyTorch upgrade the cuDNN v8.5 version that it was previously bundling to v8.7 due to a serious perf issue in [92288](https://github.com/pytorch/pytorch/issues/92288) 
When I pip install the default torch of 1.13.1 it also installs nvidia-cudnn-cu11 to provide the old cuDNN libraries.  For some reason to fix issue-92288 instead of upgrading "THE" cuDNN package to v8.7 you instead simply threw the cuDNN v8.7 libraries into torch/lib.

NVidia has long ago released cuDNN v8.7 and recently released v8.8.   Why not just install the REAL THING instead of just copying out the libraries and leaving nearly 1GB of the old nvidia-cudnn-cu11==8.5.0.96 stuff to potentially cause conflicts and waste disk space?

Of course, if I don't "UPGRADE" an old pytorch 1.13 to V2 and just "INSTALL" V2 in a clean env then V2 seems to no longer has the dependency on nvidia/cudnn so I don't get two copies of the lib's.  But here is the problem;  I know how to see EXACTLY where the libraries loaded into my python process is coming from using "pmap" on Linux.  If I tried to upgrade nvidia-cudnn-cu11 to v8.8 it DOES NOT use the cuDNN libraries in the OFFICIAL cuDNN package.  It sees the torch/lib copies first and loads those.  I can only image the problems less sophisticated users will incur when one day some says: Oh, BTW if you upgrade to cuDNN v8.8 in your env you can fix that issue you've been running into.  Stealing a pile of libraries, from cuDNN, isn't a real install of a package.


### Versions

.

cc @ezyang @seemethere @malfet @ngimel
Proposal: Disable GC in test suite; GC after every test case
Labels: triaged, module: flaky-tests, module: infra, module: testing, module: devx

### 🐛 Describe the bug

GC triggering at unpredictable times is sometimes a cause of nondeterminism in unit tests. It seems to me that it should be feasible to disable GC entirely, and only run GC at the end of tests (and perhaps selectively enable GC for particular tests that need it to work correctly). WDYT.

cc @ZainRizvi @kit1980 @huydhn @clee2000 @albanD 

### Versions

master
Enable functorch testing for rocm
Labels: module: rocm, triaged, module: functorch

While trying to enable functorch testing for rocm I see a couple of failures. Putting this issue up to track.

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @Chillee @samdow @soumith @kshitij12345 @janeyx99
tests for linearize fail under the dynamo CI config
Labels: triaged, oncall: pt2, module: functorch, module: dynamo

See skips over at https://github.com/pytorch/pytorch/blob/acd9df8a72195c226ddbe08fa0381f29cf502f0c/test/functorch/test_eager_transforms.py#L2532

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @Chillee @samdow @kshitij12345 @janeyx99 @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
early stopping
Labels: feature, triaged, needs design

### 🚀 The feature, motivation and pitch

It would be nice and easy to have the early stopping function/feature in Pytorch so that neural network training can stop before overfitting based on some criteria for instance algorithm stops after 3 consequent epochs with very small changes in validation loss. At the moment this is possible in the training loop by adding a condition but if this is done by a separate specific function, it would be much easier for users to utilize it.

### Alternatives

_No response_

### Additional context

_No response_
MPS Backend Doc, model = YourFavoriteNet() not defined
Labels: module: docs, triaged, actionable, module: mps

### 📚 The doc issue

I have used MacBook MPS documentation and when I run the code, it gives an error: YourFavoriteNet() is undefined. 

Reference:[(https://pytorch.org/docs/stable/notes/mps.html)]

### Suggest a potential alternative/fix

Fixing the document code and potentially adding YourFavoriteNet() function or class.

cc @svekars @carljparker @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
[torch.compile] Warning using BuiltinVariable.call_dict but not for {}
Labels: good first issue, triaged, oncall: pt2, release notes: dynamo

### 🐛 Describe the bug

I would expect torch.compile to work equally well for dictionaries created using `dict` as for dictionaries created with `{}`, but using `dict` leads to the warning `incorrect arg count <bound method BuiltinVariable.call_dict of BuiltinVariable(dict)> got an unexpected keyword argument 'loss' and no constant handler`.

Is this expected behavior?


Here's a small reproduction:

```
class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = torch.nn.Embedding(10, 8)

    def forward(self, input_ids):
        outputs = self.embedding(input_ids)
        loss = outputs.mean()
        return dict(loss=loss, outputs=outputs)  # warning
        # return {"loss": loss, "outputs": outputs} ok
 ```
test with
```
model = torch.compile(Model())
model(torch.randint(10, (4, 16)))
```

### Error logs

2.1.0.dev20230307
[2023-03-09 17:17:55,890] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-03-09 17:17:55,895] torch._dynamo.variables.builtin: [WARNING] incorrect arg count <bound method BuiltinVariable.call_dict of BuiltinVariable(dict)> got an unexpected keyword argument 'loss' and no constant handler
[2023-03-09 17:17:55,895] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function debug_wrapper
[2023-03-09 17:17:56,774] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 0
[2023-03-09 17:17:58,713] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 0
[2023-03-09 17:17:58,713] torch._dynamo.output_graph: [INFO] Step 2: done compiler function debug_wrapper


### Minified repro

_No response_

### Versions

PyTorch version: 2.1.0.dev20230307
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A


cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @davidberard98
'aten::affine_grid_generator' to ONNX opset version 14 is not supported
Labels: module: onnx, triaged

### 🐛 Describe the bug

UnsupportedOperatorError: Exporting the operator 'aten::affine_grid_generator' to ONNX opset version 14 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub: https://github.com/pytorch/pytorch/issues

### Versions

torch 1.13.1
Unable to move torch.jit.load-ed models to XLA devices
Labels: oncall: jit, module: xla

### 🐛 Describe the bug

Here are the steps to reproduce and the error:
```
>>> import torch
>>> import torch_xla.core.xla_model as xm
>>> x = torch.nn.Linear(8, 32)
>>> torch.jit.save(torch.jit.script(x), 'x.pt')
>>> xsr = torch.jit.load('x.pt')
>>> xsr.to(xm.xla_device())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 987, in to
    return self._apply(convert)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 668, in _apply
    assert isinstance(param, Parameter)
AssertionError
```
Here is when it works if I try just scripting (without save&load):
```
>>> x.to(xm.xla_device())
Linear(in_features=8, out_features=32, bias=True)
>>> xs = torch.jit.script(x)
>>> xs.to(xm.xla_device())
RecursiveScriptModule(original_name=Linear)
```
However, the version that's read back into xsr fails:
```
>>> xsr
RecursiveScriptModule(original_name=Linear)
>>> xsr.to(xm.xla_device())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 987, in to
    return self._apply(convert)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 668, in _apply
    assert isinstance(param, Parameter)
AssertionError
```

Let me know if you need any more information or a way to install pytorch/xla.

### Versions

pt-1.13

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel @bdhirsh @yyetim
Information about CPU in `collect_env` is too verbose
Labels: module: collect_env.py, triaged

### 🐛 Describe the bug

See this report for example:
```
PyTorch version: 2.0.0+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.25.0
Libc version: glibc-2.31

Python version: 3.10.9 (main, Mar 1 2023, 18:23:06) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-1031-aws-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A10G
Nvidia driver version: 520.61.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 48 bits physical, 48 bits virtual
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 23
Model: 49
Model name: AMD EPYC 7R32
Stepping: 0
CPU MHz: 2799.834
BogoMIPS: 5599.66
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 128 KiB
L1i cache: 128 KiB
L2 cache: 2 MiB
L3 cache: 16 MiB
NUMA node0 CPU(s): 0-7
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save rdpid

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] torch==2.0.0+cu118
[pip3] torchaudio==2.0.0+cu118
[pip3] torchvision==0.15.0+cu118
[conda] numpy 1.24.1 pypi_0 pypi
[conda] torch 2.0.0+cu118 pypi_0 pypi
[conda] torchaudio 2.0.0+cu118 pypi_0 pypi
[conda] torchvision 0.15.0+cu118 pypi_0 pypi
```

I.e. report is 70 lines long and info about CPU is 50+% of that.
Plan to just trim it to 10 lines max (and add offset)
cc: @jingxu10 
### Versions

CI
Compressed sparse constructor allows mixed `int32/int64` indices which leads to dtype promotion/demotion in conversions.
Labels: module: sparse, triaged

### 🐛 Describe the bug

As per title, namely we can create a tensor with `compressed_indices.dtype != plain_indices.dtype`. That not only can potentially break some kernels, it creates inconsistencies in `to_sparse(...)` conversions. For example:
```python
In [1]: import torch

In [2]: x = torch.rand(3, 3).to_sparse_csr()

In [3]: y = torch.sparse_csr_tensor(x.crow_indices().to(torch.int32), x.col_indices().to(torch.int64), x.values())

In [4]: z = y.to_sparse_csc()

In [5]: z.ccol_indices().dtype
Out[5]: torch.int32

In [6]: z.row_indices().dtype
Out[6]: torch.int32

In [7]: y = torch.sparse_csr_tensor(x.crow_indices().to(torch.int64), x.col_indices().to(torch.int32), x.values())

In [8]: z = y.to_sparse_csc()

In [9]: z.ccol_indices().dtype
Out[9]: torch.int64

In [10]: z.row_indices().dtype
Out[10]: torch.int64

```

### Versions

Current master.

cc @alexsamardzic @pearu @cpuhrsch @amjames @bhosmer
Add location information when exception are thrown in `torch.jit.annotations.try_ann_to_type`
Labels: oncall: jit

### 🚀 The feature, motivation and pitch

The functions `is_tuple`, `is_list`, `is_dict`, `is_optional`, `is_union`, `is_future`  (all in `torch/_jit_internal`) might throw exceptions when called from `try_ann_to_type`. These exception are not very useful. For example, one says:

```
RuntimeError: Attempted to use Dict without contained types. Please add contained type, e.g. Dict[int, int]
```

This exception together with the location information would be much more useful. For example, the location says:

```
(Pdb) print(loc)
SourceRange at:
  File "/home/schuetze/Documents/work/github/prediction_net/multimodal/models/heads/retina_head.py", line 187
    def forward(self, fpn_features: t.Dict, inputs: t.Dict,
                                    ~~~~~~ <--- HERE
                gts: t.Dict = None) -> t.Dict[str, t.Any]:
        """
```

Could we maybe add `location` as an optional parameter to the `is_X` function? Would you be open for a PR in that case? 


### Alternatives

Include location information when throwing the exception. 

### Additional context

_No response_

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Proxy Options for Pytorch Hub
Labels: triaged, module: hub

### 🚀 The feature, motivation and pitch

Pytorch Hub is frequently used to download models from GitHub to simplify dependency management for deep learning projects. For many people, me included, easy internet access to GitHub is not a given, and a proxy server may be useful or even necessary. Currently, Pytorch Hub can only be configured to use a proxy by setting the environment variable `HTTPS_PROXY`, and this method still does not work when using a SOCKS proxy server.

This issue is compounded by the fact that loading models using Pytorch Hub requires internet access even if the model is already cached. Even if GitHub is sporadically accessible, or if some workaround can be used to download the weights without a proxy server, the problem is not solved. A connection to GitHub must still be established every time the deep learning project is run, not just the first time when it needs to download the models. In my case, though I can jump through hoops to make a connection possible on my server, having to make the necessary changes every time my server runs a script that contains `torch.hub.load` is very problematic.

It would be helpful if Pytorch Hub can be configured to use a proxy server, even if it might not be realistic to make Pytorch Hub usable without internet access on every call to `torch.hub.load`.

### Alternatives

**1. Making the default network interface of the machine a tunnel**: Can negatively affect other applications and users of the machine. Complicated to set up, if split tunneling is needed to avoid some of the impact. Requires intervention from the system administrator.
**2. Using a router with proxy support**: Needs dedicated hardware. Network quality may be worse through the proxy, yet all applications on the system  generally are forced to the same network connection. Still requires intervention from the system administrator.
**3. Prefixing all python calls with `HTTPS_PROXY=...`**: can be difficult when tools such as IDEs, Jupyter, bash scripts etc. automatically spawn the python processes. Does not work with SOCKS proxy.

### Additional context

I've prepared a pull request that adds an additional function to the Pytorch Hub.

cc @nairbv @NicolasHug @vmoens @jdsgomes
[dynamo] add hook to modify instructions before/after instructions be generated
Labels: feature, triaged, module: dynamo

### 🚀 The feature, motivation and pitch

Our team is working on dynamo and pytorch code optimize, found it's easy to implement dynamo backend,but hard to modify instructions that generated by dynamo;
There are some usecase show that modify instructions is necessary:
1. want take whole graph infomation about compute and communication ,but `InstructionTranslator` break graph when it not support some function such as `AutogradFunction`, as dynamo backend, we can only see partial graph which called `compute graph`
2. modify just a little instructions after dynamo have generated them, such as auto add `checkpoint` function call or add context

we can make some discuss if this feature request is accepted ,and I will make a PR to implement this

for this feature requests ,I will modify code:
1. _dynamo/convert_frame.py _compile closure: transform function,before `InstructionTranslator(instructions) `
2. _dynamo/convert_frame.py _compile closure: transform function, before `instructions[:] = output.output_instructions`  
There also are some issue will be consider
1. _dynamo.hooks.py `Hooks` only have two hooks, using at: `_dynamo.optimize` like `def optimize(..,guard_export_fn=None,
    guard_fail_fn=None,)`,should it change to `optimize(..,hooks=None,)` to add more hooks? it will cause not compatible with old version code/unittest/doc .I will modify them all

@ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh

### Alternatives

_No response_

### Additional context

_No response_

cc @soumith @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
[export] "strict subset of traced input/output" error when huggingface `ModelOutput` is returned
Labels: triaged, module: dynamo, module: export

### 🐛 Describe the bug

Example script below. 

The error stems from discrepancy between outputs that were eagerly traced
https://github.com/pytorch/pytorch/blob/fe05266fda4f908130dea7cbac37e9264c0429a2/torch/_dynamo/eval_frame.py#L706
and outputs that were computed by captured graph
https://github.com/pytorch/pytorch/blob/fe05266fda4f908130dea7cbac37e9264c0429a2/torch/_dynamo/eval_frame.py#L688

The former returns `ModelOutput` type while the latter returns tuple. And since `pytree.tree_flatten` is unable to open up `ModelOutput`, the following `produce_matching` triggers the error.

https://github.com/pytorch/pytorch/blob/fe05266fda4f908130dea7cbac37e9264c0429a2/torch/_dynamo/eval_frame.py#L716-L720

```python
import traceback
import torch
import dataclasses
from torch import _dynamo as dynamo
from typing import Optional
from transformers.modeling_outputs import ModelOutput


@dataclasses.dataclass
class UserDefinedModelOutput(ModelOutput):
    x: Optional[torch.Tensor] = None
    y: Optional[torch.Tensor] = None


def fn(x, y):
    return UserDefinedModelOutput(x + y, x - y)


x = torch.randn(3, 4)
y = torch.randn(3, 4)


# AssertionError: Dynamo input and output is a strict subset of traced input/output
try:
    gm, guards = dynamo.export(fn, x, y)
except AssertionError as e:
    traceback.print_exc()


out = dynamo.optimize("eager", nopython=True)(fn)(x, y)
# UserDefinedModelOutput(x=tensor(...), y=tensor(...))
print(out)


```

### Versions

master

cc @soumith @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
`dynamo.export` "input not consistent with traced input" error when input default value type is `torch.Tensor`.
Labels: triaged, onnx-needs-info, module: dynamo, module: export

### 🐛 Describe the bug

Example script below. It looks like `b` is always traced as input, so the assertion error happens when `b` is not provided.
Similar scenario happens if the default is a container of `torch.Tensor`.

I know it is bad practice to set mutable defaults, and such code should be avoided, (but there might be models that are written this way). I wonder how should `dynamo.export` respond for these cases?

```python
import torch
import traceback
from torch import _dynamo as dynamo

def func(x, b=torch.tensor(1.0)):
    return x + b

x = torch.randn(3, 4)
b = torch.randn(3, 4)


# Trace without 'b'
# AssertionError: Dynamo input/output is not consistent with traced input/output
try:
    gm, guards = dynamo.export(func, x)
    gm.print_readable()

except AssertionError as e:
    traceback.print_exc()


# Trace with 'b'
# Succeed.
gm, guards = dynamo.export(func, x, b=b)
gm.print_readable()

```

### Versions

master

cc @soumith @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
[BE] Avoid .data usage in FSDP buffer casting
Labels: oncall: distributed, better-engineering, module: fsdp

### 🐛 Describe the bug

FSDP casts buffers for mixed precision, but uses `buf.data` to assign to, we should avoid the .data usage by de-registering the old buffer and re-registering the low precision buffer.

### Versions

main

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
views created in __torch_dispatch__ share storage but not version_counter
Labels: module: autograd, module: molly-guard, triaged, needs design, module: __torch_dispatch__

Usually the ADInplaceOrView kernel is responsible for handling this, but since we're operating under autograd, the version counter information is not correctly propagated.

```python
saved_b = None

class SaveATensorMode(TorchDispatchMode):
    def __torch_dispatch__(self, func, types, args=(), kwargs=None):
        global saved_b
        kwargs = {} if kwargs is None else kwargs
        out = func(*args, **kwargs)
        if func == torch.ops.aten.sin.default:
          saved_b = out.view_as(out)
        return out

a = torch.tensor(1.)
with SaveATensorMode():
    b = torch.sin(a)

assert b.data_ptr() == saved_b.data_ptr()

old_b_version = b._version
old_saved_b_version = saved_b._version
b.mul_(2)

print(b._version > old_b_version)  # True
print(saved_b._version > old_saved_b_version)  # False

old_b_version = b._version
old_saved_b_version = saved_b._version
saved_b.mul_(2)

print(b._version > old_b_version)  # False
print(saved_b._version > old_saved_b_version)  # True

```

Previously this was probably intentional for inference mode, but it may be useful to support version counter propagation for `__torch_dispatch__` use cases to prevent common silently correctness issues.

If this were a Tensor subclass, we could probably enable_reentrant_dispatch, but that may not work for modes (can we fix this?).

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @Lezcano @Varal7 @Chillee @samdow
Support managed memory backed dlpack with  torch.from_dlpack
Labels: feature, triaged, module: dlpack

### 🚀 The feature, motivation and pitch

Pytorch should support using managed memory based dlpack capsules.  

### Set Pools
```python3
import rmm
from rmm.allocators.torch import  rmm_torch_allocator
from rmm.allocators.cupy import  rmm_cupy_allocator
import torch
import cupy as cp

rmm.reinitialize(managed_memory= True)
cp.cuda.set_allocator(rmm_cupy_allocator)
torch.cuda.memory.change_current_allocator(rmm_torch_allocator)
```


```python3
dlpack_cap = cp.ones(shape=100).toDlpack()
t2 = torch.from_dlpack(dlpack_cap)
```
```python3
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[2], line 2
      1 dlpack_cap = cp.ones(shape=100).toDlpack()
----> 2 t2 = torch.from_dlpack(dlpack_cap)

File /datasets/vjawa/miniconda3/envs/cugraph_dev_feb_27/lib/python3.10/site-packages/torch/utils/dlpack.py:120, in from_dlpack(ext_tensor)
    117 else:
    118     # Old versions just call the converter
    119     dlpack = ext_tensor
--> 120 return _from_dlpack(dlpack)

RuntimeError: Unsupported device_type: 13

```


### Alternatives

A user can use cuda array interface to work around it for now. 

```python3
ar = cp.ones(shape=100)
t2 = torch.as_tensor(ar, device='cuda')
```

### Additional context

CC: @leofang , @jakirkham