gopikrsmscs/torch-issues huggingface dataset:
DISABLED test_vmapjvpvjp_svd_cuda_float32 (__main__.TestOperatorsCUDA)
Labels: triaged, skipped, module: functorch

Platforms: rocm

This test was disabled because it is failing on master ([recent examples](http://torch-ci.com/failure/functorch%2Ftest_ops.py%3A%3ATestOperatorsCUDA%3A%3Atest_vmapjvpvjp_svd_cuda_float32)).

cc @zou3519 @Chillee @samdow @soumith @kshitij12345 @janeyx99
Sparse is not available on Windows
Labels: module: sparse, module: build, module: windows, triaged

### 🐛 Describe the bug

To track following:
https://github.com/pytorch/pytorch/blob/d850c33bfe3f1d1f0040738718cacb04ee449bdc/aten/src/ATen/mkl/Sparse.h#L5-L11


Copy-n-paste from 2y.o [comment](https://github.com/pytorch/pytorch/pull/50937#issuecomment-779272492):

## What's wrong there
The following error message is generated in the Windows build jobs for this PR (e.g. [this one](https://app.circleci.com/pipelines/github/pytorch/pytorch/273229/workflows/816eb750-45b5-4fb8-9e34-850e7608a3f2/jobs/10881331))
```
error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MT_StaticRelease' doesn't match value 'MD_DynamicRelease' ...
```
## Root cause
The implementation of `mkl_sparse_d_create_csr` and `mkl_sparse_destroy` uses `malloc/new` and `free/delete` possibly, which requires UCRT. Currently, we link against the static variant of MKL with our code with /MD. With MSVC, you cannot have two copies of UCRT for a shared library / executable with different runtime library options (/MD and /MT).

## Possible solutions
### Solution 1
Remove the usages of `mkl_sparse_d_create_csr` and `mkl_sparse_destroy`
I don't know whether this is possible. But if possible, this is the easiest solution.

### Solution 2
Link against the dynamic variant of MKL instead.
In this solution, we have two choices.
#### Choice A
We depend on package managers completely. 
But the difficulty is that MKL doesn't provide every release of MKL in PyPI and Anaconda Cloud. e.g.  on PyPI, they have 2021.01 and 2019.0. However, on Anaconda Cloud, they have 2020.02 (defaults channel), 2020.04 (conda-forge channel) and 2021.01 (intel channel). So it is quite difficult for us to select the version of MKL to build PyTorch.
Another difficulty is that Anaconda Cloud doesn't provide the full set of development package for Windows. But we can install them through other package managers, e.g. nuget and choco.

#### Choice B
We copy the DLLs into the PyTorch package.
But the MKL DLLs are quite large. Uncompressed: 650M, Compressed: 128M

#### Choice C
A tradeoff between Choice A and B. But this still doesn't seem perfect.


cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer @seemethere @peterjc123 @mszhanyi @skyline75489 @nbcsm
torch.ops.aten.pow(2.0, 3) return unexpected value with complex type
Labels: module: internals, triaged, module: custom-operators

### 🐛 Describe the bug

torch.ops.aten.pow(2.0, 3) return unexpected value with complex type\

```python
torch.ops.aten.pow(2.0, 3)
```
Ideally it should return `8.0`, but it returns `(8.0+0j)`
![image](https://user-images.githubusercontent.com/29956693/226844140-486b11fd-b222-403c-829e-2abd4475d081.png)


### Versions

torch==2.1.0.dev20230320+cpu

cc @ezyang @bhosmer @smessmer @ljk53 @bdhirsh
MPS: `unique` and `unique_consecutive` extremely slow when `return_counts=True`
Labels: module: performance, triaged, module: mps

### 🐛 Describe the bug

There appears to be a severe performance issue with `unique` and `unique_consecutive` when `return_counts=True` and the device is mps. I performed the following test:

```
import timeit

import torch

t = torch.randint(10, size=(100000,), device='mps')
t = t[t.argsort()]
t_cpu = t.cpu()

number = 10

print(timeit.timeit('t.unique()', globals=globals(), number=number))
print(timeit.timeit('t.unique(return_inverse=True)', globals=globals(), number=number))
print(timeit.timeit('t.unique(return_counts=True)', globals=globals(), number=number))
print(timeit.timeit('t_cpu.unique()', globals=globals(), number=number))
print(timeit.timeit('t_cpu.unique(return_inverse=True)', globals=globals(), number=number))
print(timeit.timeit('t_cpu.unique(return_counts=True)', globals=globals(), number=number))
print()
print(timeit.timeit('t.unique_consecutive()', globals=globals(), number=number))
print(timeit.timeit('t.unique_consecutive(return_inverse=True)', globals=globals(), number=number))
print(timeit.timeit('t.unique_consecutive(return_counts=True)', globals=globals(), number=number))
print(timeit.timeit('t_cpu.unique_consecutive()', globals=globals(), number=number))
print(timeit.timeit('t_cpu.unique_consecutive(return_inverse=True)', globals=globals(), number=number))
print(timeit.timeit('t_cpu.unique_consecutive(return_counts=True)', globals=globals(), number=number))
```
Output:
```
0.026759792000000004
0.035945208000000006
2.092266542
0.0016070839999997588
0.004217375000000079
0.006821041999999888

0.009870165999999791
0.010194499999999884
2.0578671660000003
0.0006946670000003152
0.0007299169999992472
0.0006608329999995277
```
As you can see, `unique` and `unique_consecutive` with `return_counts=True` on mps is around 100x slower. This gets even worse (can be 1000-10000x slower) if I increase the number of elements.

### Versions

Collecting environment information...
PyTorch version: 2.1.0.dev20230320
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.2.1 (arm64)
GCC version: Could not collect
Clang version: 12.0.0 (clang-1200.0.32.28)
CMake version: version 3.26.0
Libc version: N/A

Python version: 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:38:11)  [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-13.2.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==1.24.2
[pip3] torch==2.1.0.dev20230320
[conda] numpy                     1.24.2                   pypi_0    pypi
[conda] torch                     2.1.0.dev20230320          pypi_0    pypi

cc @ngimel @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
[composable FSDP] clip_grad_norm
Labels: oncall: distributed, triaged, module: fsdp

### 🚀 The feature, motivation and pitch

FSDP clip_grad_norm should work with composable API: https://github.com/pytorch/pytorch/blob/master/torch/distributed/fsdp/fully_sharded_data_parallel.py#L962

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Desync debugger encounters traceMap error
Labels: oncall: distributed, triaged, module: c10d, bug

### 🐛 Describe the bug

An internal user reported the following stack trace:
```
[ProcessGroupNCCL.cpp:847] [Rank 58] NCCL watchdog thread terminated with exception: traceMap[thisSeq][myRank].second == kEventStart INTERNAL ASSERT FAILED at "torch/csrc/distributed/c10d/TraceUtils.h":244, please report a bug to PyTorch. Timeout rank [58] last trace item must be kEventStart. thisSeq = 463038, col = ALLREDUCE
Exception raised from retrieveDesyncReport at torch/csrc/distributed/c10d/TraceUtils.h:244 (most recent call first):
# 2  c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
# 3  c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
# 4  c10d::retrieveDesyncReport(c10::intrusive_ptr<c10d::Store, c10::detail::intrusive_target_default_null_type<c10d::Store> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int)
# 5  c10d::ProcessGroupNCCL::abortTimedOutCollectives(std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&)
# 6  c10d::ProcessGroupNCCL::ncclCommWatchdogInternal()
# 7  c10d::ProcessGroupNCCL::ncclCommWatchdog()
# 8  execute_native_thread_routine
# 9  start_thread
# 10 __clone3
```
It seems that the desync debugger missed to log the start of a collective, and hence when logging the completion it failed to find a match.

### Versions

master as of today

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @awgu
DISABLED test_vmapjvpvjp_linalg_svd_cuda_float32 (__main__.TestOperatorsCUDA)
Labels: triaged, skipped, module: functorch

Platforms: rocm

This test was disabled because it is failing on master ([recent examples](http://torch-ci.com/failure/functorch%2Ftest_ops.py%3A%3ATestOperatorsCUDA%3A%3Atest_vmapjvpvjp_linalg_svd_cuda_float32)).

Functorch test has recently been integrated into PyTorch default test suite, and this test starts to become flaky on ROCm

cc @zou3519 @Chillee @samdow @soumith @kshitij12345 @janeyx99
functorch roll-up issue for 2.1
Labels: triaged, module: functorch

P0 Critical (silent incorrectness, crashes)
- [x] https://github.com/pytorch/pytorch/issues/97161
- [x] https://github.com/pytorch/pytorch/issues/96887
- [x] https://github.com/pytorch/pytorch/issues/96854

P0 Interaction with PT2
- [ ] needs better error messages: https://github.com/pytorch/pytorch/issues/96041

UX
- [ ] Better error message for https://github.com/pytorch/pytorch/issues/94397
- [ ] https://github.com/pytorch/pytorch/issues/97248 (large)

Coverage
- [x] https://github.com/pytorch/pytorch/issues/95561
- [ ] https://github.com/pytorch/pytorch/issues/96855
- [x] https://github.com/pytorch/pytorch/issues/95888
- [x] https://github.com/pytorch/pytorch/issues/95738
- [x] https://github.com/pytorch/pytorch/issues/95057
- [x] https://github.com/pytorch/pytorch/issues/94630
- [x] https://github.com/pytorch/pytorch/issues/97558
- [x] https://github.com/pytorch/pytorch/issues/99187
Inductor "Original ATen:" doesn't work for backwards kernels
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

Example output:

```
# kernel path: /tmp/torchinductor_ezyang/rw/crwlqodm5jav743txhbrnyh5enri7mh6arxwt577cpal2zclclph.py
# Original ATen:·

triton_fused_8 = async_compile.triton('''
import triton
import triton.language as tl
from torch._inductor.ir import ReductionHint
from torch._inductor.ir import TileHint
from torch._inductor.triton_ops.autotune import reduction
from torch._inductor.utils import instance_descriptor

@reduction(
    size_hints=[32, 65536],
    reduction_hint=ReductionHint.DEFAULT,
    filename=__file__,
    meta={'signature': {0: '*fp32', 1: '*fp32', 2: 'i32', 3: 'i32', 4: 'i32', 5: 'i32', 6: 'i32'}, 'device': 0, 'constants': {}, 'mutated_arg_names': [], 'configs': [instance_descriptor(divisible_by_16=(0, 1, 3, 5, 6), equal_to_1=())]}
)
@triton.jit
def triton_(in_ptr0, out_ptr0, ks0, ks1, ks2, xnumel, rnumel, XBLOCK : tl.constexpr, RBLOCK : tl.constexpr):
    xoffset = tl.program_id(0) * XBLOCK
    xindex = xoffset + tl.arange(0, XBLOCK)[:, None]
    xmask = xindex < xnumel
    rbase = tl.arange(0, RBLOCK)[None, :]
    x0 = xindex
    _tmp1 = tl.zeros([XBLOCK, RBLOCK], tl.float32) + 0
```

### Versions

master

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh @Chillee
INTERNAL ASSERT FAILED at "../c10/cuda/CUDAGraphsC10Utils.h":73, please report a bug to PyTorch. Unknown CUDA graph CaptureStatus32729
Labels: triaged, module: cuda graphs

              how can i solve this error please 
seed_value=42
random.seed(seed_value)
np.random.seed(seed_value) <--------- error
torch.manual_seed(seed_value)
torch.cuda.manual_seed_all(seed_value) 

RuntimeError: false INTERNAL ASSERT FAILED at "../c10/cuda/CUDAGraphsC10Utils.h":73, please report a bug to PyTorch. Unknown CUDA graph CaptureStatus32729

_Originally posted by @cdmssa in https://github.com/pytorch/pytorch/issues/82886#issuecomment-1476349230_
            

cc @mcarilli
aten::sym_size is not using torch._ops.OpOverload in FX graph
Labels: triaged, module: dynamic shapes

### 🐛 Describe the bug

aten::sym_size has two overloads, but none of them are used. In FX graph, OverloadPacket itself seems to be used. Not sure if this is intended, or it's a bug, but it seems to be inconsistent to other ops.

```python
import torch
from torch.fx.experimental.proxy_tensor import make_fx

class MyModule(torch.nn.Module):
    def forward(self, x):
        return torch.flatten(x, start_dim=2, end_dim=3)

x = torch.randn(3, 5, 4, 5)
m = make_fx(MyModule(), tracing_mode="symbolic")(x)

for node in m.graph.nodes:
    if isinstance(node.target, torch._ops.OpOverloadPacket):
        print(type(node.target))  # <class 'torch._ops.OpOverloadPacket'>
        print(node.target.overloads())  # ['default', 'int']
```

### Versions

Versions of relevant libraries:
[pip3] mypy==0.960
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.23.1
[pip3] pytorch==2.1.0a0+git3e6bde0
[pip3] torchaudio==0.13.0.dev20220912+cpu
[pip3] torchvision==0.15.0a0+511924c
[conda] blas                      1.0                         mkl  
[conda] cpuonly                   2.0                           0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-include               2022.1.0           h06a4308_224  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.23.1           py39h6c91a56_0  
[conda] numpy-base                1.23.1           py39ha15fc14_0  
[conda] pytorch                   2.1.0a0+git3e6bde0           dev_0    <develop>
[conda] pytorch-mutex             1.0                         cpu    pytorch
[conda] torchaudio                0.13.0.dev20220912+cpu          pypi_0    pypi
[conda] torchvision               0.15.0a0+511924c          pypi_0    pypi


cc @ezyang @BowenBao
Sequential/Partial unpickling and loading of models
Labels: module: pickle, triaged

### 🚀 The feature, motivation and pitch

Given a `90GB` model file that should not be run, but rather just inspected, you might want to load it partially.

Example usage:

```py
descriptors = torch.load_tensors_desc(filename)
# at this state, the memory usage should be close to zero.
for desc in descriptors:
    tensor = torch.load_tensor_from_desc(filename, desc)
    # at this state, the mamory usage should be about the size of the tensor.
```

This is **not** a suggestion regarding syntax. At the moment, this is hardly possible:
- A normal `torch.load` will use about `90GB` to load the model.
- Implementing a custom `pickle.Unpickler` still wants to construct the `torch.storage`'s for all tensors, which *conceptually* should not be neccessary (also, doesn't work as some tensors may share storage)
- Reimplementing the pickle-format is tedious.

I'm not sure this is an easy patch, or if it even is possible. Please leave any suggestions in the comments.

### Alternatives

_No response_

### Additional context

_No response_
Building LibTorch on Ubuntu with Mac M1
Labels: module: build, triaged, module: torchbind, module: m1

I'm running into a similar issue as [#22116](https://github.com/pytorch/pytorch/issues/22116) but for a different setup:
Hardware: MacbookPro M1 
OS: Ubuntu 20.04.5 via VMware Fusion (followed this [guide](https://dev.to/daud99/installing-ubuntu-using-vmware-fusion-tech-preview-on-mac-m1-silicon-4b0e))

Followed the official documentation [here](https://pytorch.org/cppdocs/installing.html) and received the following error after running "cmake --build . --config Release":


```
bmazotti@ubuntu20045:~/Documents/spslam/example-app/build$ cmake --build . --config Release
Scanning dependencies of target example-app
[ 50%] Building CXX object CMakeFiles/example-app.dir/example-app.cpp.o
[100%] Linking CXX executable example-app
/usr/bin/ld: /home/bmazotti/Documents/spslam/libtorch/lib/libtorch.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/example-app.dir/build.make:88: example-app] Error 1
make[1]: *** [CMakeFiles/Makefile2:76: CMakeFiles/example-app.dir/all] Error 2
make: *** [Makefile:84: all] Error 2
```


Any help would be greatly appreciated, thanks in advance!

### Versions

cmake version 3.16.3

cc @malfet @seemethere @kulinseth @albanD @DenisVieriu97 @razarmehr @abhudev
Nightly conda binaries failed to pass tests since 2023-03-17 
Labels: oncall: binaries, oncall: releng, module: ci, triaged

### 🐛 Describe the bug

Starting from 2023-03-17, the conda binary started to fail at "Test PyTorch Binary" step . 
For example: https://github.com/pytorch/pytorch/actions/runs/4445030656/jobs/7805063867#step:15:1658
It failed to find sympy "No module named 'sympy'" 

![image](https://user-images.githubusercontent.com/109318740/226440336-e777bb80-f827-40ed-9d4a-e53664adde82.png)

As a result: https://anaconda.org/pytorch-nightly/pytorch is showing 2023-03-16 being the latest linux binary. 

### Versions

Nightly from 2023-03-17 and onwards till this issue is closed. 

cc @ezyang @seemethere @malfet @pytorch/pytorch-dev-infra @atalman
[FSDP][optim_state_dict] Need more comprehensive tests for optim_state_dict interface
Labels: oncall: distributed, triaged, module: fsdp

### 🚀 The feature, motivation and pitch

FSDP optim_state_dict has many different APIs that confuse users a lot. In a recent PR, the two new APIs have been proposed to consolidate all the use cases -- `optim_state_dict` and `optim_state_dict_to_load`. However, current unittests do not have enough coverage of the new interfaces. More test coverage is required for `state_dict_type` and `optim_state_dict` and `optim_state_dict_to_load`.

https://github.com/pytorch/pytorch/pull/97110 fixes a breakage.
https://github.com/pytorch/pytorch/pull/96534 introduce some tests but need more.

cc., @wz3

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Implement `torch.distributions.Poisson.cdf()`
Labels: module: distributions, feature, triaged

### 🚀 The feature, motivation and pitch

I'm currently working with Tobit Poisson loss functions, and I'm lacking the ability to calculate the Poisson CDF. Currently the `cdf` function for the Poisson distribution isn't implemented.

### Alternatives

I've tried using scipy to do this, but this doesn't go well when calculating grads. I'm currently using a very hacky workaround, which seems unnecessary. 

### Additional context

If any efficient workarounds are known, feel free to share them here :)
I'm currently using `torch.special.gammaincc(torch.floor(k+1),  lambda)`

cc @fritzo @neerajprad @alicanb @nikitaved
nn.Conv function to compute conv formula
Labels: module: docs, module: nn, triaged, actionable

### 🚀 The feature, motivation and pitch

When there is a `nn.Linear` after a `nn.ConvXd`, something like the below:

```python
nn.Sequential(
    nn.ConvXd(...),  # E.g. 2d or 3d
    nn.Flatten(),
    nn.Linear(input_shape, ...)  # Requires knowing the Conv equation
)
```

One has to think about the Conv equation ([image source](https://pytorch.org/docs/1.13/generated/torch.nn.Conv3d.html#torch.nn.Conv3d)):

> ![conv3d equation](https://user-images.githubusercontent.com/8990777/226217387-6dd1951e-c332-4098-853d-672df5ee0ab5.png)

On many different school projects, I find myself re-implementing this equation.  It would be really useful if `torch` had a free function that could be `import`ed to accomplish this.

It would be well tested and just be a handy way to enable DRY model code with `torch`'s `Conv` layers.

Would `torch` be open to having a free function contributed for this?  It could be a part of `nn` or `nn.functional`.

### Alternatives

Here is my current best function for this:

```python
def conv_conversion(
    in_shape: tuple[int, ...],
    kernel_size: int | tuple[int, ...],
    padding: int | tuple[int, ...] = 0,
    dilation: int | tuple[int, ...] = 1,
    stride: int | tuple[int, ...] = 1,
) -> tuple[int, ...]:
    """Perform a Conv layer calculation matching nn.Conv's defaults."""

    def to_tuple(value: int | tuple[int, ...]) -> tuple[int, ...]:
        return (value,) * len(in_shape) if isinstance(value, int) else value

    k, p = to_tuple(kernel_size), to_tuple(padding)
    dil, s = to_tuple(dilation), to_tuple(stride)
    return tuple(
        int((in_shape[i] + 2 * p[i] - dil[i] * (k[i] - 1) - 1) / s[i] + 1)
        for i in range(len(in_shape))
    )
```

I would like to gather feedback on how it can be further generalized, and then write some unit tests for it.

Cheers!

### Additional context

_No response_

cc @svekars @carljparker @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
[Feature Proposal: New Distributed Training Algorithms] LSGD and EASGD
Labels: oncall: distributed, triaged

### 🚀 The feature, motivation and pitch

Currently, PyTorch only offers support for DataParallelism. I am working on integrating two additional distributed training optimizers, EASGD and LSGD, into the codebase. The motivation is to enable a straightforward alternative of vanilla DataParallelism.

Vanilla DataParallelism:
``` python
model = DistributedDataParallel(model, device_ids=[args.gpu])
model.forward(data); model.backward(loss) ; optimizer.step() 
```

Alternative Distributed Training:
``` python
optimizer = CommOptimizer(optimizer, dist_optim_name='LSGD+', comm_period=1, dist_pulling_strength=0.1, local_pulling_strength=0.1)
model.forward(data); model.backward(loss) ; optimizer.step() 
```

The synchronous version could be implemented and wrapped up easily. A quick glance is available at [**SDTL GitHub**](https://github.com/yunfei-teng/SDTL).

The asynchronous version presents greater difficulty in implementation, but a prototype version could be found at [**LSGD GitHub**](https://github.com/yunfei-teng/LSGD).

### Alternatives

_No response_

### Additional context

LSGD:
* Paper: https://arxiv.org/abs/1905.10395
* LSGD Blog: https://yunfei-teng.github.io/LSGD-Blog

EASGD:
* Paper: https://arxiv.org/abs/1412.6651


cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
"Adaptive pool MPS: input sizes must be divisible by output sizes", I keep getting this error even when I try to adjust for size
Labels: triaged, module: mps

### 🐛 Describe the bug

I am running a facet-pytorch library for facial recognition. I initialised the mtcnn class and passed on MPS as the device so I can run the program on that GPU for faster results. 

'''
device = torch.device("mps")
mtcnn1 = MTCNN(image_size=(160,160) device=device)
'''

The problem starts when I try to iterate though an image folder and pass it in the mtcnn i.e.
for imge, index in dataLoader:
    face, prob = mtcnn0(image, return_prob=True) 

it keeps coming up with this error:

RuntimeError: Adaptive pool MPS: input sizes must be divisible by output sizes

Can someone help please?

### Versions

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
Optimize for mobile produces incorrect result with INSERT_FOLD_PREPACK_OPS optimization
Labels: oncall: jit, oncall: mobile, actionable

### 🐛 Describe the bug

When my model is saved with default parameters for `optimize_for_mobile` it yields incorrect output.
If I add `INSERT_FOLD_PREPACK_OPS` to the blocklist, the result is correct.

```python
torchscript_model = torch.jit.script(model)
torchscript_model_optimized = optimize_for_mobile(torchscript_model, optimization_blocklist={MobileOptimizerType.INSERT_FOLD_PREPACK_OPS})
torchscript_model_optimized._save_for_lite_interpreter("model.ptl")
```

I'm happy to provide any additional info, including my source code and model weights. However, it's proprietary, so I won't post them to the public.

### Versions

PyTorch version: 1.14.0.dev20221110
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.202)
CMake version: version 3.25.0
Libc version: N/A

Python version: 3.9.12 (main, Jun  1 2022, 06:34:44)  [Clang 12.0.0 ] (64-bit runtime)
Python platform: macOS-13.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==1.23.4
[pip3] numpydoc==1.2
[pip3] torch==1.14.0.dev20221110
[pip3] torchaudio==0.14.0.dev20221110
[pip3] torchfile==0.1.0
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.15.0.dev20221110
[conda] nomkl                     3.0                           0
[conda] numpy                     1.23.4                   pypi_0    pypi
[conda] numpydoc                  1.2                pyhd3eb1b0_0
[conda] torch                     1.14.0.dev20221110          pypi_0    pypi
[conda] torchaudio                0.14.0.dev20221110          pypi_0    pypi
[conda] torchfile                 0.1.0                    pypi_0    pypi
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.15.0.dev20221110          pypi_0    pypi

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
DDP static graph fails for static model
Labels: oncall: distributed, triaged, module: ddp

### 🐛 Describe the bug

```
class M(nn.Module):
                def __init__(self):
                    super().__init__()
                    self.a = nn.Linear(10, 10)
                    self.b = nn.Linear(10, 10)

                def forward(self, x):
                    a = self.a(x)
                    b = self.b(x)
                    return (a, b)

            m = M().cuda()
            ddp = torch.nn.parallel.DistributedDataParallel(
                m, device_ids=[self.rank], static_graph=True
            )
            inp = torch.rand(1, 10, device='cuda')
            for _ in range(6):
                out = ddp(inp)[0]
                out.sum().backward()
                # b's gradient should be None.
                self.assertEqual(None, m.b.weight.grad)
                self.assertEqual(None, m.b.bias.grad)
```

results in 
```
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your training graph has changed in this iteration, e.g., one parameter is used in first iteration, but then got unused in the second iteration. this is not compatible with static_graph set to True.
```

### Versions

main

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
How to get list of all valid devices?
Labels: module: docs, triaged

### 📚 The doc issue

`torch.testing.get_all_device_types()`
yields all valid devices on the current machine however unlike `torch._tensor_classes` , `torch.testing.get_all_dtypes()`, and `import typing; typing.get_args(torch.types.Device)`, there doesn't seem to be a comprehensive list of all valid device types, which gets listed  when I force an error

```
torch.devcie('asdasjdfas')
RuntimeError: Expected one of cpu, cuda, ipu, xpu, mkldnn, opengl, opencl, ideep, hip, ve, fpga, ort, xla, lazy, vulkan, mps, meta, hpu, privateuseone device type at start of device string: asdasjdfas
```

### Suggest a potential alternative/fix

```
torch._device_names = cpu, cuda, ipu, xpu, mkldnn, opengl, opencl, ideep, hip, ve, fpga, ort, xla, lazy, vulkan, mps, meta, hpu, privateuseone 

```

cc @svekars @carljparker
deprecate integral and boolean dtype support torch.logit and torch.special.logit
Labels: triaged, module: deprecation, module: special

### 🐛 Describe the bug

Valid domain for these functions is (0, 1). So it doesn't make sense to support integral or boolean inputs for them as there are only two valid values (which produce -inf and inf respectively).

Thanks @pmeier for finding this.

More context: https://github.com/pytorch/pytorch/pull/96124#issuecomment-1471973352

### Versions

master

cc @mruberry
[Feature Request] Compile compatible Neighborhood Algorithms for large Tensors
Labels: triaged, oncall: pt2

### 🚀 The feature, motivation and pitch

For large tensors, neighborhood algorithms are limited by large memory requirements, for example the squared-distance matrix of KNN.

Speaking from my field of 3D analysis, a pointcloud scan can easily have >100.000 coordinates. A reconstructed scene of 500m can have up to 1 Billion data points. Applying neighborhood algorithms on such data is only possible with specialized frameworks such as PyKeOps or FAISS. To my knowledge, this also affects graph convolutional networks

This request was denied in the past due to compatible frameworks. Now with the introduction of .compile() this has changed.

This request is for .compile() PyTorch native neighborhood algorithms, optimized for very large tensors.

### Alternatives

(Not .compile() compatible)
PyKeOps
Faiss

### Additional context

_No response_

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
Small learning rate with `capturable=True` causes Adam optimizer to blow up model parameters.
Labels: module: optimizer, triaged

### 🐛 Describe the bug

When setting the `capturable` argument of Adam optimizer to `True`, the call of `optimizer.step` will use the following routine (line 267~ 290 in `torch/optim/adam.py`):
```
        if capturable:
            ......
            step_size = lr / bias_correction1
            step_size_neg = step_size.neg()
            ......
            else:
                denom = (exp_avg_sq.sqrt() / (bias_correction2_sqrt * step_size_neg)).add_(eps / step_size_neg)

            param.addcdiv_(exp_avg, denom)

```
This will cause overflow when `lr` is very small.



### Versions

Below are some relevant version information.
```
PyTorch version: 1.12.0
Python version: 3.8.13 (default, Mar 28 2022, 11:38:47)  [GCC 7.5.0] (64-bit runtime)

Versions of relevant libraries:
[pip3] torch==1.12.0
[conda] pytorch                   1.12.0          py3.8_cuda11.3_cudnn8.3.2_0    pytorch
```

cc @vincentqb @jbschlosser @albanD @janeyx99
[mps] conv1d outputs zeros
Labels: triaged, module: mps

### 🐛 Describe the bug

HI! I'm getting tensors filled with zeros when using `mps`with "large" inputs and outputs.
```python
import torch
from torch.nn.functional import conv1d

device = "mps"
input_signal = torch.ones(size=(4, 1, 385360)).to(device)
kernel = torch.ones(size=(352, 1, 65536)).to(device)
output_signal = conv1d(input_signal, kernel)
print(output_signal) ## This should be full of ones, but it is however full of zeros.

#### This behaviour does not happen if, for example, the kernel is smaller
kernel = torch.ones(size=(1, 1, 1)).to(device)
output_signal = conv1d(input_signal, kernel)
print(output_signal) ## Has ones everywhere, as expected

```

I wonder if this is related to #88308 and #96716, as they are also resulting into matrices of zeros when using `mps`

### Versions

```PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.2.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.202)
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.10 (main, Mar  4 2023, 23:23:44) [Clang 14.0.0 (clang-1400.0.29.202)] (64-bit runtime)
Python platform: macOS-13.2.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M2 Pro```

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==2.0.0
[pip3] torchaudio==2.0.1
[pip3] torchmetrics==0.10.1
[conda] Could not collect

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
Adding sparse `addmv` and `triangular_solve` support on CPU - Mac OS - Apple Silicon M2
Labels: module: sparse, triaged, module: macos, module: arm

### 🚀 The feature, motivation and pitch

As discussed in https://github.com/pytorch/pytorch/issues/77764#issuecomment-1472510169 it would be helpful to have a CPU fallback for [`torch.addmv`](https://pytorch.org/docs/stable/generated/torch.addmv.html?highlight=addmv#torch.addmv) on Mac OS when using sparse matrices.

### Alternatives

Run on google colab or similar

### Additional context

Steps to reproduce the issue on Mac OS:
```python
import torch
print(f"Running PyTorch version: {torch.__version__}")

dtype = torch.float32
device = torch.device("cpu")
#device = torch.device("mps")
print(f"Using device: {device}")

#mat = torch.randn((4,4), dtype=dtype, device=device)
mat = torch.randn((4,4), dtype=dtype, device=device).relu().to_sparse_csr()
mvec = torch.randn((4,), dtype=dtype, device=device)
avec = torch.randn((4,), dtype=dtype, device=device)

ovec = torch.addmv(avec, mat, mvec)
print(ovec)
```
leading to
```
UserWarning: Sparse CSR tensor support is in beta state. If you miss a functionality in the sparse tensor support, please submit a feature request to https://github.com/pytorch/pytorch/issues. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/SparseCsrTensorImpl.cpp:56.)
  mat = torch.randn((4,4), dtype=dtype, device=device).relu().to_sparse_csr()
Traceback (most recent call last):
  File "[...]/test.py", line 14, in <module>
    ovec = torch.addmv(avec, mat, mvec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Calling addmv on a sparse CPU tensor requires compiling PyTorch with MKL. Please use PyTorch built MKL support.
```

It may be that MKL can be compiled for Mac OS (and thus shipped in the default pytorch distribution for mac) or maybe an less optimised alternative needs to be found (e.g. Eigen).

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer @malfet @albanD
GPU：7900xtx Pytorch2.0.0  rocBLAS error:
Labels: oncall: binaries, module: rocm, triaged

### 🐛 Describe the bug

```python
import torch
A = torch.ones(5, 5).to('cuda')
B = torch.ones(5, 5).to('cuda')
C = torch.matmul(A, B)
```
when program runing to calc C,i get this Message: 
```
rocBLAS error: Cannot read /home/kali/miniconda3/envs/pytorch2/lib/python3.9/site-packages/torch/lib/rocblas/library/TensileLibrary.dat: No such file or directory
```

It is found that the reason is the lack of the rocblas library file with the architecture of gfx1100, so putting the gxf1100 library file into the rocblas directory in the python environment of conda can solve this problem

### Versions

GPU：AMD 7900XTX
OS：Ubuntu 22.04
Pytorch: 2.0.0
install by: pip install 

Compiling the source code or repackaging the whl file containing the gfx1100 library files can solve this problem

cc @ezyang @seemethere @malfet @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport
Expected scalar type Half but found Float when running nn.MultiheadAttention with AMP
Labels: oncall: transformer/mha

### 🐛 Describe the bug

When performing self-attention using nn.MultiheadAttention with precision=16, I get `RuntimeError: expected scalar type Half but found Float). I believe it is related to [this issue](https://github.com/pytorch/pytorch/issues/84396, and this [unmerged PR](https://github.com/pytorch/pytorch/pull/84722) from @erichan1. 

I have gotten it to work by changing the code from `self.self_attn(x, x, x)` to `self.self_attn(x+1e-16, x, x)`, but I'm not sure if this could potentially cause other issues.

```
# from __init__()
self.self_attn = nn.MultiheadAttention(num_input_channels, 16, batch_first=True)
```

```
attn_out, attn_weights = self.self_attn(x, x, x)
  File "/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 1113, in forward
    return torch._native_multi_head_attention(
RuntimeError: expected scalar type Half but found Float
```

### Versions

torch 1.12.1 + cu11.6

cc @jbschlosser @bhosmer @cpuhrsch @erichan1
Performance Drop for linalg_ldl_factor and ldl_solve
Labels: triaged, enhancement, actionable, module: vmap, module: functorch

Hi, I have a function which vmaps over matrices, performs LDLT decomposition and solve them (together with other operations)
Are there plans to implement batching for this operation in the near future ?
Matrices which I'm handling are only positive semidefinite, so using LDLT decomposition would work much better than cholesky.

Thanks!


minimal example:
```python
import torch
import functorch

def solve(A, b):
    LD, pivots = torch.linalg.ldl_factor(A)
    return torch.linalg.ldl_solve(LD, pivots, b)

A = torch.randn(2, 3, 3)
A = A @ A.mT # make symmetric
B = torch.randn(2, 3, 1)

print(functorch.vmap(solve)(A, B))
```
```
UserWarning: There is a performance drop because we have not yet implemented the batching rule for aten::linalg_ldl_factor. Please file us an issue on GitHub so that we can prioritize its implementation. (Triggered internally at ../aten/src/ATen/functorch/BatchedFallback.cpp:82.)
  LD, pivots = torch.linalg.ldl_factor(A)
UserWarning: There is a performance drop because we have not yet implemented the batching rule for aten::linalg_ldl_solve. Please file us an issue on GitHub so that we can prioritize its implementation. (Triggered internally at ../aten/src/ATen/functorch/BatchedFallback.cpp:82.)
  return torch.linalg.ldl_solve(LD, pivots, b)
```

cc @zou3519 @Chillee @samdow @soumith @kshitij12345 @janeyx99
`cumprod` triggers INTERNAL ASSERT FAILED when `out` is a tensor on cuda but input is on cpu
Labels: module: error checking, triaged, module: assert failure

### 🐛 Describe the bug

`cumprod` triggers INTERNAL ASSERT FAILED when `out` is a tensor on cuda but input is on cpu

```py
import torch
from torch.func import jacrev
import torch.nn as nn
from torch.autograd.functional import jacobian


torch.manual_seed(420)

m = torch.rand(2, 3, 4, 5, 6, 7)
out = torch.rand([]).cuda()
m_cumprod = torch.cumprod(m, dim=0, out=out)
# RuntimeError: t == DeviceType::CUDA INTERNAL ASSERT FAILED 
# at "../c10/cuda/impl/CUDAGuardImpl.h":25, please report a bug to PyTorch
```

### Versions

```
PyTorch version: 2.0.0.dev20230105
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.9.15 (main, Nov 24 2022, 14:31:59)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 515.86.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==2.0.0.dev20230105
[pip3] torchaudio==2.0.0.dev20230105
[pip3] torchvision==0.15.0.dev20230105
[conda] blas                      1.0                         mkl
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py39h7f8727e_0
[conda] mkl_fft                   1.3.1            py39hd3c417c_0
[conda] mkl_random                1.2.2            py39h51133e4_0
[conda] numpy                     1.23.5           py39h14f4228_0
[conda] numpy-base                1.23.5           py39h31eccc5_0
[conda] pytorch                   2.0.0.dev20230105 py3.9_cuda11.7_cudnn8.5.0_0    pytorch-nightly
[conda] pytorch-cuda              11.7                 h67b0de4_2    pytorch-nightly
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] torchaudio                2.0.0.dev20230105      py39_cu117    pytorch-nightly
[conda] torchtriton               2.0.0+0d7e753227            py39    pytorch-nightly
[conda] torchvision               0.15.0.dev20230105      py39_cu117    pytorch-nightly
```

cc @malfet
[MPS] pinverse dtype error
Labels: triaged, module: linear algebra, module: mps

I am trying to do the pinverse on MPS machine. 

I am setting the variable as:
```
debug = torch.Tensor([[1,2,3],[4,5,6]]).to(device=torch.device('mps')).to(dtype=torch.float32)
```
confirming variable is loaded in mps:
````
>> debug.dtype
torch.float32
>> debug
tensor([[1., 2., 3.],
        [4., 5., 6.]], device='mps:0')
````

Calling the following command would give me error:
```
>> debug_pinv = debug.pinverse()

Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.
```

### Versions

CPU:
Apple M2 Max

Versions of relevant libraries:
[pip3] numpy==1.24.2
[pip3] torch==1.13.1
[pip3] torchaudio==0.13.1
[pip3] torchvision==0.14.1
[conda] numpy                     1.24.2                   pypi_0    pypi
[conda] torch                     1.13.1                   pypi_0    pypi
[conda] torchaudio                0.13.1                   pypi_0    pypi
[conda] torchvision               0.14.1                   pypi_0    pypi


### does this mean the pinverse's implementation is fixed on float64 for MPS? 

cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
Follow-ups to do after adding nested checkpoint
Labels: module: autograd, triaged, actionable

Tracking some follow ups to do after https://github.com/pytorch/pytorch/pull/90105 lands:
- [x] update docs to reflect ability to do backward within checkpointed region https://github.com/pytorch/pytorch/pull/96862
- [x] enable early-stopping by default, add an API to disable https://github.com/pytorch/pytorch/pull/96866
  - [x] update docs to mention the early stop feature https://github.com/pytorch/pytorch/pull/96866
- [ ] update docs/tutorial to feature nested use cases
- [ ] https://github.com/pytorch/pytorch/issues/96764

Missing anything?

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @Lezcano @Varal7
Improve checkpoint thread-safety
Labels: module: autograd, triaged, module: multithreading, needs design

### 🐛 Describe the bug

Checkpoint's uses a dictionary to track whether a given checkpoint has already been recomputed, and this dictionary can be shared between multiple threads. As a result we might run into something like the following (note that the GIL doesn't protect you here!):

Possible race condition:
- perform forward on two devices to force backward to be performed on two threads, since they spawned from the same backward, the two threads share the same graph_task_id
- both threads simultaneously try to execute two separate nodes created under the same checkpoint
- the test `if not frame.is_recomputed[gid]:` passes for both threads
- both threads do recompute for the same checkpoint! (something bad happens, at the very least, we recompute checkpoint unnecessarily)

See https://github.com/pytorch/pytorch/pull/90105#discussion_r1135722492 



### Versions

main

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @Lezcano @Varal7
[inductor] flaky rexnet_100 accuracy tests
Labels: triaged, oncall: pt2, module: inductor

### 🐛 Describe the bug

rexnet_100 has accuracy flakiness, sometimes marking accuracy failure due to `stem.bn.weight.grad`.

#96691 and #96474 disable rexnet_100 in CI.

For an example, see https://github.com/pytorch/pytorch/actions/runs/4402868441/jobs/7710977874.

### Versions

master / CI

cc @ezyang @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @soumith
Many padding Module fail memory_format tests
Labels: module: nn, triaged, module: memory format, actionable, module: intel

See https://github.com/pytorch/pytorch/pull/96641 that added these skips.
We should investigate why this happens and most likely fix them.

cc @ezyang @gchanan @zou3519 @mruberry @jbschlosser @walterddr @mikaylagawarecki @jamesr66a @frank-wei @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @saketh-are
when run python run_test.py -i test_ops_jit error like this. ValueError: option names {'--junit-xml-reruns'} already added
Labels: oncall: jit

### 🐛 Describe the bug


cd pytorch/test
python run_test.py -i test_ops_jit



### Versions

by torch version v1.13.1

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel