gopikrsmscs/torch-issues huggingface dataset:
hanging process with init_process_group(backend='mpi') cannot be killed 
Labels: oncall: distributed

## Issue description

Running mpirun with torch.distributed hangs. When parts of ranks raised error intentionally and the others are doing allgather, those who raised error cannot be killed and were waiting for all other processes to exit. On the other hand, the process that is doing allgather seems to be waiting for other processes too. This leads to all process hanging eventually.

## Code example

```
import torch.distributed as dist
import torch
import time
import sys
import os

dist.init_process_group(backend='mpi')
rank = dist.get_rank()
size = dist.get_world_size()
pid = os.getpid()

tensor_list = [torch.zeros(1, dtype=torch.int64) for _ in range(size)]
tensor = torch.arange(1, dtype=torch.int64) + rank

if rank == 0:
    print(f"Allgather: PID[{pid}]")
    dist.all_gather(tensor_list, tensor)
else:
    # to kill a process
    print(f"Runtime Error: PID[{pid}]")
    raise RuntimeError
    # sys.exit(-1)
    # dist.destroy_process_group() 
```

Error msg:
```
Runtime Error: PID[16955]
Runtime Error: PID[16956]
Runtime Error: PID[16957]
Traceback (most recent call last):
  File "torch_init.py", line 21, in <module>
    raise RuntimeError
RuntimeError
Traceback (most recent call last):
  File "torch_init.py", line 21, in <module>
    raise RuntimeError
RuntimeError
Traceback (most recent call last):
  File "torch_init.py", line 21, in <module>
    raise RuntimeError
RuntimeError
```
and hangs afterwards. (verified by checking 'top' cmd)

tried to use async allgather with timeout, and the wait() was able to terminate but still cannot exit afterwards.

- PyTorch or Caffe2:
- How you installed PyTorch (conda, pip, source): pip
- Build command you used (if compiling from source):
- OS: Ubuntu18
- PyTorch version: 1.8.1+cpu
- Python version: python3.7


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang
Quantization in Libtorch
Labels: oncall: quantization, triaged

### 🚀 The feature, motivation and pitch

I am working on a Libtorch program that is heavily performance constrained, and I was hoping to run inference using quantization to speed up the model and reduce memory usage. I would like for Libtorch to support static and dynamic quantization just like Pytorch, both for creating quantized models and using quantized torchscript models.

### Alternatives

I tried quantizing the model in Python and then using that exported torchscript .pt file in Libtorch, but that gave a `RuntimeError: Didn't find engine for operation quantized::conv2d_prepack NoQEngine` (I get the same error trying to import that model in Pytorch). I am using Windows 10.

### Additional context

_No response_

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo
Default repr of __get__ methods in __torch_function__ is bad
Labels: triaged, module: __torch_function__

### 🐛 Describe the bug

I got a function in `__torch_function__` that I couldn't handle. I wanted to print it out to see what was up:

```
>>> repr(func)
"<method-wrapper '__get__' of getset_descriptor object at 0x7f1208602a40>"
```

Completely useless. In this case, I needed to call `repr(func.__self__)` to get useful information about getset_descriptor.

This is actually sort of the inverse problem as https://github.com/pytorch/pytorch/issues/76534 where for a more useful bound method print, we want CPython to produce the repr of the self object cc @hameerabbasi @rgommers @peterbell10 @pmeier 

### Versions

master
Bug in dataloader iterator found by mypy
Labels: module: dataloader, triaged

While upgrading mypy, found a call to `_BaseDataLoaderIter.reset()` that appears to trivially fail ([link](https://github.com/pytorch/pytorch/blob/543eaac415bfba59e648a3da8be5c4f964f9bc6e/torch/utils/data/dataloader.py#L557)), as it does not provide a required positional argument.

cc @SsnL @VitalyFedyunin @ejguan @NivekT
[checkpoint] Stable file format for checkpoints
Labels: oncall: distributed, triaged, sharded_tensor

### 🚀 The feature, motivation and pitch

We need to provide a stable file format at some level for checkpoints so users can, at the very least, query which PyTorch version is necessary to correctly load it.



### Alternatives

_No response_

### Additional context

_No response_

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang
[checkpoint] Handle overlapping storage during save and load
Labels: oncall: distributed, triaged, sharded_tensor

### 🚀 The feature, motivation and pitch

When loading a checkpoint, we should check whether any of the tensors have overlapping storage and, if so, we're loading data from different storage_keys. We should eliminate the superfluous load is such cases.

A similar check should be performed when saving a checkpoint and use that to reduce the amount of IO done.

 

### Alternatives

_No response_

### Additional context

_No response_

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang
Supporting torch.tensor.apply_ over GPU
Labels: module: cuda, triaged

### 🚀 The feature, motivation and pitch

I came across a problem in my project where I needed that `apply_` function. It is interesting to see that it isn't supported over GPU. 

I was wondering if it is possible to extend it to GPU, perhaps for pure lambdas to begin with. 

If there are no known issues directly stopping this from happening, I would like to work on this. 

I feel that extending it to GPU for pure lambda callbacks is a good starting point. 

PS: Any pointers on the issue will be appreciated as my experience with Pytorch has been strictly with its application interface as of now :D 

### Alternatives

_No response_

### Additional context

_No response_

cc @ngimel
test_python_reference_meta_functions takes too long to run
Labels: module: tests, triaged

### 🐛 Describe the bug

Currently, these tests take nearly two minutes to run on CPU:

```
$ python test/test_ops.py -k test_python_reference_meta_functions                                                                                                              
...............................................................................................................................
...............................................................................................................................
...............................................................................................................................
......................................................................................................                         
----------------------------------------------------------------------                                                         
Ran 483 tests in 117.137s 
```

This is far too long for 483 tests; we're spending nearly 0.2s a test. It would be good to understand why these tests take so long.

### Versions

master

cc @mruberry
Adding Polyloss to `torch`
Labels: module: nn, module: loss, triaged

### 🚀 The feature, motivation and pitch

# POLYLOSS: A POLYNOMIAL EXPANSION PERSPEC- TIVE OF CLASSIFICATION LOSS FUNCTIONS
(Published as a conference paper at ICLR 2022) [link to paper](https://arxiv.org/abs/2204.12511)

This loss function implementation won't be difficult as we can use builtin crossentropy loss or focal loss to implement it. This loss function has overperformed other existing loss functions in some benchmark experiments. 

Here is a code snippet
```python3
# cross entropy as pollyloss
def poly1_cross_entropy(logits, labels, epsilon=1.0):
    # pt, CE, and Poly1 have shape [batch].
    pt = tf.reduce_sum(labels * tf.nn.softmax(logits), axis=-1) CE = tf.nn.softmax_cross_entropy_with_logits(labels, logits)
    Poly1 = CE + epsilon * (1 - pt)
    return Poly1

# focal loss as pollyloss
def poly1_focal_loss(logits, labels, epsilon=1.0, gamma=2.0): 
    # p, pt, FL, and Poly1 have shape [batch, num of classes]. p = tf.math.sigmoid(logits)
    pt = labels * p + (1 - labels) * (1 - p)
    FL = focal_loss(pt, gamma)
    Poly1 = FL + epsilon * tf.math.pow(1 - pt, gamma + 1)
    return Poly1
```

I would like to contribute to add this feature into `torch`

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345
`reshape` for distributions.
Labels: module: distributions, feature, triaged

### 🚀 The feature, motivation and pitch

The `expand` method for distribution instances is great for prepending batch dimensions and, possibly, changing the number of batch elements. Adding similar functionality that mirrors `reshape` on tensors would be useful for manipulating distributions. 

For example, suppose we want to evaluate the probability of a batch of beta random variables on the unit interval.

```python
>>> import torch

# Set up the distribution and the domain over which we want to evaluate.
>>> batch_shape = (3, 4)
>>> dist = torch.distributions.Beta(*torch.rand(2, *batch_shape))
>>> x = torch.linspace(0, 1, 50)

# This, expectedly fails, because the shapes don't match.
>>> dist.log_prob(x)
ValueError: Value is not broadcastable with batch_shape+event_shape: torch.Size([50]) vs torch.Size([3, 4]).

# We can use `expand` to prepend a batch dimension, then pad `lin` with unit 
# dimensions on the right that match the batch shape. Then move the batch 
# dimensions back to the front. Fine, but a bit fiddly.
>>> log_prob = dist.expand((1,) + batch_shape).log_prob(x[:, None, None])
>>> log_prob = log_prob.moveaxis(0, -1)
>>> log_prob.shape
(3, 4, 50)

# Suppose we had a `reshape` function that acts on the batch dimension. Then
# the above simply becomes.
>>> log_prob = dist.reshape(batch_shape + (1,)).log_prob(x)
>>> log_prob.shape
(3, 4, 50)
```

### Alternatives

See above regarding a combination of `expand`, right-padding of the argument, and `moveaxis`.

### Additional context

_No response_

cc @fritzo @neerajprad @alicanb @nikitaved
Torch `x += y.bmm(z)` is faster than `x.baddbmm_(y, z)`
Labels: module: performance, triaged, module: linear algebra

### 🐛 Describe the bug

The `x.badbmm_(y, z)` method feels like something that should be fast, since it doesn't have to create a temporary tensor for `y.bmm(z)` before it adds it to `x`.

However, from profiling it seems like `badbmm_` just makes a lot of calls to `aten::addmm_` and `aten::select`, and turns out to be about twice as slow as the naiive code.

### Versions

Pytorch pre-release version 1.12.0a0+fb 

cc @VitalyFedyunin @ngimel @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano
Performance bad on ARM AArch64 for PyTorch C++
Labels: module: build, triaged, module: arm

### 🚀 The feature, motivation and pitch

Hi PyTorch community, we have been trying to build the PyTorch on the CentOS-7/ Amazon Linux 2 on an aarch64 instance (Graviton) for our C++ project. However, we found our build performance is much worse comparing to the PyTorch aarch64 pip release. I am suspecting if there is any performance optimization libraries/flags we have not enabled during the build.

Do you mind sharing some build instruction/docker images that can build PyTorch python/C++ library for low GLIBC versions (precxx-11) on aarch64? Or C++ ARM distribution for libtorch 1.11.0 would also work.


### Alternatives

_No response_

### Additional context

_No response_

cc @malfet @seemethere
[torch::deploy] Remove `manager_` from the constructor and deconstructor of `InterpreterSession`
Labels: module: deploy, oncall: package/deploy, imported

[torch::deploy](https://github.com/pytorch/pytorch/blob/master/docs/source/deploy.rst) is a runtime for getting pytorch models to run on multiple threads in eager mode. There's a great whitepaper about torch.package and torch::deploy [here](https://arxiv.org/pdf/2104.00254.pdf).

Motivation: We want to be able to breakout interpreter manager from the interpreter. In practice what this means is that the interpreter has no reliance on the interpreter manager. With this we will be able to ship separate `deployinterpreter.so` files. This also allows people to write their own interpreter manager for high performance use cases.

Effectively what we need to do is remove the reliance of `manager_` from `Interpreter/ InterpreterSession`

Steps
1. Remove `manager_` from the constructor and deconstructor of `InterpreterSession` in `deploy.cpp`. 

3. Make corresponding changes to the function in `deploy.h`, and tests such as the ones found in `test_deploy.cpp`, `test_deploy_gpu.cpp`, etc.
4. Test by running the following commands for OSS from the `pytorch` root directory (with your pytorch conda env)

`git submodule sync && git submodule update --init --recursive`
`DEBUG=1 USE_CUDA=OFF USE_OPENMP=OFF USE_DISTRIBUTED=OFF USE_DEPLOY=ON BUILD_CAFFE2=OFF BUILD_CAFFE2_OPS=OFF python setup.py develop`
`python ~/local/pytorch/torch/csrc/deploy/example/generate_examples.py`
`./build/bin/test_deploy`

cc @wconstab
[torch::deploy] move create_movable to interpreter_manager
Labels: oncall: package/deploy, imported

[torch::deploy](https://github.com/pytorch/pytorch/blob/master/docs/source/deploy.rst) is a runtime for getting pytorch models to run on multiple threads in eager mode. There's a great whitepaper about torch.package and torch::deploy [here](https://arxiv.org/pdf/2104.00254.pdf).

Motivation: We want to be able to breakout interpreter manager from the interpreter. In practice what this means is that the interpreter has no reliance on the interpreter manager. With this we will be able to ship separate `deployinterpreter.so` files. This also allows people to write their own interpreter manager for high performance use cases.

Effectively what we need to do is remove the reliance of `manager_` from `Interpreter/ InterpreterSession`

Steps
1. In `deploy.cpp` move the function the function`ReplicatedObj InterpreterSession::createMovable(Obj obj)` to reside in the `InterpreterManager`, such that the manager uses itself to create the `ReplicatedObj`. As for which interpreterSession to use, ideally we could just pick a free one and use that in a function with the signature `InterpreterManager::createMovable(Obj obj)`. Furthermore, we'll let the user to input the InterpreterSession they want the object on as another parameter of `createMovable` by creating another function with the signature `InterpreterManager::createMovable(Obj obj, InterpreterSession session)`. 

3. Make corresponding changes to the function in `deploy.h`, and tests such as the ones found in `test_deploy.cpp`, `test_deploy_gpu.cpp`, etc.
4. Test by running the following commands for OSS from the `pytorch` root directory (with your pytorch conda env)

`git submodule sync && git submodule update --init --recursive`
`DEBUG=1 USE_CUDA=OFF USE_OPENMP=OFF USE_DISTRIBUTED=OFF USE_DEPLOY=ON BUILD_CAFFE2=OFF BUILD_CAFFE2_OPS=OFF python setup.py develop`
`python ~/local/pytorch/torch/csrc/deploy/example/generate_examples.py`
`./build/bin/test_deploy`
[torch::deploy] remove reliance on manager_ for unload
Labels: oncall: package/deploy, imported

[torch::deploy](https://github.com/pytorch/pytorch/blob/master/docs/source/deploy.rst) is a runtime for getting pytorch models to run on multiple threads in eager mode. There's a great whitepaper about torch.package and torch::deploy [here](https://arxiv.org/pdf/2104.00254.pdf).

Motivation: We want to be able to breakout interpreter manager from the interpreter. In practice what this means is that the interpreter has no reliance on the interpreter manager. With this we will be able to ship separate `deployinterpreter.so` files. This also allows people to write their own interpreter manager for high performance use cases.

Effectively what we need to do is remove the reliance of `manager_` from `Interpreter/ InterpreterSession`

Steps
1. In `deploy.cpp` in the function `ReplicatedObjImpl::unload(const Interpreter* onThisInterpreter) ` remove the `onThisInterpreter` logic.
2. Make corresponding changes to the function in `deploy.h`, and tests such as the ones found in `test_deploy.cpp`, `test_deploy_gpu.cpp`, etc.
3. Test by running the following commands for OSS from the `pytorch` root directory (with your pytorch conda env)

`git submodule sync && git submodule update --init --recursive`
`DEBUG=1 USE_CUDA=OFF USE_OPENMP=OFF USE_DISTRIBUTED=OFF USE_DEPLOY=ON BUILD_CAFFE2=OFF BUILD_CAFFE2_OPS=OFF python setup.py develop`
`python ~/local/pytorch/torch/csrc/deploy/example/generate_examples.py`
`./build/bin/test_deploy`
[torch::deploy] Remove manager_ from AquireSession
Labels: oncall: package/deploy, imported

[torch::deploy](https://github.com/pytorch/pytorch/blob/master/docs/source/deploy.rst) is a runtime for getting pytorch models to run on multiple threads in eager mode. There's a great whitepaper about torch.package and torch::deploy [here](https://arxiv.org/pdf/2104.00254.pdf).

Motivation: We want to be able to breakout interpreter manager from the interpreter. In practice what this means is that the interpreter has no reliance on the interpreter manager. With this we will be able to ship separate `deployinterpreter.so` files. This also allows people to write their own interpreter manager for high performance use cases.

Effectively what we need to do is remove the reliance of `manager_` from `Interpreter/ InterpreterSession`

Steps
1. In `deploy.cpp` in the function`InterpreterSession ReplicatedObj::acquireSession` remove `manager_`, have it such that if you attempt to `AquireSession` on an interpreter, the session will be on that interpreter or things will fail 
2. Make corresponding changes to the function in `deploy.h`, and tests such as the ones found in `test_deploy.cpp`, `test_deploy_gpu.cpp`, etc.
3. Test by running the following commands for OSS from the `pytorch` root directory (with your pytorch conda env)

`git submodule sync && git submodule update --init --recursive`
`DEBUG=1 USE_CUDA=OFF USE_OPENMP=OFF USE_DISTRIBUTED=OFF USE_DEPLOY=ON BUILD_CAFFE2=OFF BUILD_CAFFE2_OPS=OFF python setup.py develop`
`python ~/local/pytorch/torch/csrc/deploy/example/generate_examples.py`
`./build/bin/test_deploy`
Remove all docstrings when python is running in optimization mode
Labels: module: performance, module: docs, triaged, better-engineering, module: python frontend

## Issue description
Follow up of #76034.

There are docstrings still present in *pyc files when python is run in optimization mode (`-OO`). 

## Code example

```python
$ /usr/local/bin/python3.9 -OO
Python 3.9.2 (default, Mar 26 2021, 21:58:27) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> help(torch.nn.Conv1d)
```

This has to be separately handled.

Current *pyc file size for pytorch:

1. Without optimization
```bash
$ find . -type f -name '*39.pyc' -exec du -ch {} + | grep total$ 
12M	total
```
2. With optimization (`-O`)
```bash
$ find . -type f -name '*39.opt-1.pyc' -exec du -ch {} + | grep total$
5.3M	total
```
3. With optimization (`-OO`)
```bash
$ find . -type f -name '*39.opt-2.pyc' -exec du -ch {} + | grep total$
3.9M	total
```

I don't see any noticeable time difference while importing torch.

cc @VitalyFedyunin @ngimel @brianjo @mruberry
Clarify dependency on NumPy (related to maskedtensor?)
Labels: triaged

### 🐛 Describe the bug

I installed torch cpu version on a new python3.8 ubuntu 20.04 system via pip. From what I understood, it did not install NumPy as dependency.

When I import torch, I get:
```
/usr/local/lib/python3.8/dist-packages/torch/_masked/__init__.py:223: UserWarning: Failed to initialize NumPy: numpy.core.multiarray failed to import (Triggered internally at  ../torch/csrc/utils/tensor_numpy.cpp:68.)
  example_input = torch.tensor([[-3, -2, -1], [0, 1, 2]])
```

This has a lot of confusing details. If NumPy is needed, it should be specified as pip dependency and installed automatically.

If it's not strictly required, I propose that either no warning is shown on import or the warning is more clear: remove the file paths, line number, example tensor and clearly explain why NumPy is needed + point to URL that explains all the details clearly.

I think that it's not strictly needed and that maybe something about this was forgotten during maskedtensor integration.

On testing note: there should be a test of PyTorch install and import on systems without numpy to check that all works 

cc @ezyang @gchanan @zou3519 @malfet

### Versions

1.11.0+cpu
[feature request] no-param sort to exploit parallelization
Labels: module: performance, module: cpu, triaged, module: multithreading, module: sorting and selection

### 🚀 The feature, motivation and pitch

'torch.sort' can not execute parallel when there is only one dimention of tensor. I hope you could make it faster when we try to sort 1 dimention tensors.


### Alternatives

Maybe you could implement merge sort or other parallel sorting algorithms

### Additional context

https://github.com/pytorch/pytorch/issues/76654
I have a report and code here
```
import torch
import torch.nn as nn
import torch.multiprocessing as mp
import torch.distributed as dist
import torch.nn.functional as F
from torchvision.models import mobilenet_v2
import time
input = torch.rand([64,3,224,224])
conv2d = torch.nn.Conv2d(3,32,(3,3),stride= (2,2), padding = (1,1),bias= False)
time_sort_avg = 0.0
time_conv_avg = 0.0
for i in range(2):
    print(i)
    start =time.time()
    output = conv2d(input)
    time_conv = time.time() -start
    output = output.view(-1)
    start =time.time()
    sorted,index = torch.sort(output)
    time_sort = time.time() -start
    time_sort_avg += time_sort
    time_conv_avg += time_conv
print("sort time avg:",time_sort_avg/2)
print("conv time avg:",time_conv_avg/2)
```

cc @VitalyFedyunin @ngimel
`torch.sort` does not exploit parallelization when invoked without the `dim` parameter.
Labels: module: performance, feature, module: cpu, triaged, module: sorting and selection

### 🐛 Describe the bug

At cpu, `torch.sort` is so slow that it is even slower than convulutional layers. I remembered that `torch.sort` is O(Nlog(N)) complexity. Convolutional layer should be heavier than that.
Here is the code
```
 
import torch
import torch.nn as nn
import torch.multiprocessing as mp
import torch.distributed as dist
import torch.nn.functional as F
from torchvision.models import mobilenet_v2
import time
input = torch.rand([64,3,224,224])
conv2d = torch.nn.Conv2d(3,32,(3,3),stride= (2,2), padding = (1,1),bias= False)
time_sort_avg = 0.0
time_conv_avg = 0.0
for i in range(2):
    print(i)
    start =time.time()
    output = conv2d(input)
    time_conv = time.time() -start
    output = output.view(-1)
    start =time.time()
    sorted,index = torch.sort(output)
    time_sort = time.time() -start
    time_sort_avg += time_sort
    time_conv_avg += time_conv
print("sort time avg:",time_sort_avg/2)
print("conv time avg:",time_conv_avg/2)
```
The reslut on my cpu is 
```
sort time avg: 3.3030941486358643
conv time avg: 0.06574046611785889
```
Why is this happening?

### Versions

pytorch 1.12.0


cc @VitalyFedyunin @ngimel
Add `ldl_unpack` functionality
Labels: feature, triaged, module: linear algebra

### 🚀 The feature, motivation and pitch

Recently LDL factorization and solver were added to `torch.linalg` with https://github.com/pytorch/pytorch/pull/69828. The factorization is stored in the packed format in a single matrix. Similar to `torch.lu` and `torch.lu_unpack` we need to unpack the `LD` factors into (possibly permuted) lower triangular matrix `L` and tridiagonal matrix `D` matrices so that it's easier to implement derivative rules.

In LAPACK there's a function for this conversion, `syconv`, ([netlib docs](http://www.netlib.org/lapack/explore-html/d3/db6/group__double_s_ycomputational_ga03e63781ce348cc17a7065520bc7dc1a.html#ga03e63781ce348cc17a7065520bc7dc1a), [mkl docs](https://www.intel.com/content/www/us/en/develop/documentation/onemkl-developer-reference-fortran/top/lapack-routines/lapack-auxiliary-routines/syconv.html)).
>The routine converts matrix A, which results from a triangular matrix factorization, into matrices L and D and vice versa. The routine returns non-diagonalized elements of D and applies or reverses permutation done with the triangular matrix factorization.

The proposal is to expose this functionality as `torch.linalg.ldl_unpack(Tensor LD, Tensor pivots, *, bool hermitian=False) -> (Tensor P, Tensor L, Tensor D)`

### Alternatives

_No response_

### Additional context

There's a SciPy function, [`scipy.linalg.ldl`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.ldl.html), that returns the uncompressed representation of the LDL factorization.

cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano
`torch.nn.HuberLoss` backwards unexpectedly fail
Labels: module: nn, module: loss, triaged, module: type promotion

### 🐛 Describe the bug

`torch.nn.HuberLoss` backwards unexpectedly fail with error message `Found dtype Float but expected Double`. 
For its documentation, "When delta is set to 1, this loss is equivalent to [SmoothL1Loss](https://pytorch.org/docs/stable/generated/torch.nn.SmoothL1Loss.html#torch.nn.SmoothL1Loss)". As a result, it should succeed to backward like `SmoothL1Loss`

```python
import torch
t1_tensor = torch.rand([67, 1], dtype=torch.float64)
t2_tensor = torch.rand([67, 1], dtype=torch.float32)

t1 = t1_tensor.clone().detach().requires_grad_().to('cpu')
t2 = t2_tensor.clone().detach().requires_grad_().to('cpu')
HL = torch.nn.HuberLoss()
HL_res = HL(t1, t2).sum()
try:
    HL_res.backward()
except Exception as e:
    print("HuberLoss Fail Backward:", e)

t1 = t1_tensor.clone().detach().requires_grad_().to('cpu')
t2 = t2_tensor.clone().detach().requires_grad_().to('cpu')
SL = torch.nn.SmoothL1Loss()
SL(t1, t2).sum().backward()
print("SmoothL1Loss Succeed Backward")

# HuberLoss Fail Backward: Found dtype Float but expected Double
# SmoothL1Loss Succeed Backward
```

### Versions

pytorch: 1.11.0

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @nairbv
Pytorch can't process special unicode
Labels: needs reproduction, module: serialization, triaged

### 🐛 Describe the bug

Trying to make use of a program that use pytorch i ended up with this issue:

"[E] UnicodeEncodeError: 'charmap' codec can't encode character \uff0f"
Which "uff0f" is " ／ " A special unicode.
My path folder has this unicode and therefor because of this the program wasn't able to work.
Result...
**The Bug: pytorch is unable to process path folder containing special unicodes**

### Versions

Lastest Version ( 1.11 )

cc @mruberry
WeightNorm: Add reset_parameters Linear override
Labels: feature, module: nn, triaged, module: norms and normalization

### 🚀 The feature, motivation and pitch

I have faced some problems when trying to **reset the parameters of a Linear module to which I have applied weight normalization**. I have realized that the `reset_parameters` instance method of the Linear module doesn't consider potential modifications of its parameters attributes names, i.e., `self.weights` might be split into `self.weight_g` and `self.weight_v` when using `weight_norm` for instance, so this instance method implementation become kind of useless in these cases.

### Alternatives

I guess that a proper solution may lead to some refactoring, but a naive solution could be just something like this with any luck:

https://github.com/a-canela/pytorch/pull/1/files

### Additional context

Reproducing the issue:

```
import torch

module = torch.nn.utils.weight_norm(torch.nn.Linear(20, 40), name='weight')
params_before = {k: p.mean() for k, p in module.named_parameters()}
module.reset_parameters()
params_after = {k: p.mean() for k, p in module.named_parameters()}
assert all([params_before[k] != params_after[k] for k in params_before])
```

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345
RestrictPtrTraits in CUDA potentially has no effect.
Labels: module: cuda, triaged, better-engineering

### 🐛 Describe the bug

Potentially `torch::PackedTensorAccessor32<T, N, torch::RestrictPtrTraits>` might be treated the same as `torch::PackedTensorAccessor32<T, N>` in CUDA given how `__restirct__` is handeled.

The pointer to the data is a class member [TensorAccessor.h#L158](https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/core/TensorAccessor.h#L158)
```cpp
// GenericPackedTensorAccessorBase and GenericPackedTensorAccessor are used on for CUDA `Tensor`s on the host
// and as
// In contrast to `TensorAccessor`s, they copy the strides and sizes on instantiation (on the host)
// in order to transfer them on the device when calling kernels.
// On the device, indexing of multidimensional tensors gives to `TensorAccessor`s.
// Use RestrictPtrTraits as PtrTraits if you want the tensor's data pointer to be marked as __restrict__.
// Instantiation from data, sizes, strides is only needed on the host and std::copy isn't available
// on the device, so those functions are host only.
template<typename T, size_t N, template <typename U> class PtrTraits = DefaultPtrTraits, typename index_t = int64_t>
class GenericPackedTensorAccessorBase {
public:
// code omitted for readability
protected:
  PtrType data_;
  index_t sizes_[N];
  index_t strides_[N];
  // code omitted for readability
};
```

However, this is not respected by `nvcc`: [As explained by Nvidia here](https://forums.developer.nvidia.com/t/restrict-seems-to-be-ignored-for-base-pointers-in-structs-having-base-pointers-with-restrict-as-kernel-arguments-directly-works-as-expected/154020/13?u=lkskstlr).

I made an example to show this in the [compiler explorer](https://godbolt.org/z/9c3z8GfT8).

I tried to make an example that shows an actual performance difference but have been struggling so far. It seems that caching can alleviate the performance problem in easy cases.

Thank you all for the amazing library and sorry if this is not a bug but a misunderstanding on my side :slightly_smiling_face: 

### Versions

Does not apply.

cc @ngimel
`Tensor.logit`'s signature in doc misses `eps` argument
Labels: module: docs, triaged, module: python frontend

### 📚 The doc issue

`Tensor.logit`'s signature in [doc](https://pytorch.org/docs/stable/generated/torch.Tensor.logit.html?highlight=logit#torch.Tensor.logit) misses `eps` argument.

### Suggest a potential alternative/fix

It should be `Tensor.logit(eps=None)`

cc @brianjo @mruberry
Improve error message for `unfold` when generating tensor with negative dimension
Labels: triaged, actionable, module: shape checking, module: viewing and reshaping

### 🐛 Describe the bug

`unfold` could generate tensor with negative dimension

```python
import torch
input_tensor = torch.rand([5, 5, 5], dtype=torch.complex128)
dimension = 2
size = -1
step = 2
res = input_tensor.unfold(dimension, size, step)
print(res.shape)
# torch.Size([5, 5, 4, -1])
```

### Versions

pytorch: 1.10.0

cc @ezyang @gchanan @zou3519
MeanVarianceNormalization
Labels: feature, module: nn, triaged, module: norms and normalization

### 🚀 The feature, motivation and pitch

[Onnx](https://github.com/onnx/onnx/blob/main/docs/Operators.md#MeanVarianceNormalization) lets you chose which axes you are normalizing over.

However, pytorch only supports normalization of a limited set of axis:

For input of shape `(N, C, L)`, batch norm lets you normalize over `N`, and `L`, layer norm lets you normalize over `C`, and `L`. 

However, there is no support for normalizing over `N` and `C`. Which is something you would want to do if your input has the channels in the L (`-1`) axis i.e. `(N, L, C)` input.

### Alternatives
```
def MeanVarianceNomralization(input, dims):
    standard_deviation = torch.std(input, dim=dims, keepdim=True)
    mean = torch.mean(input, dim=dims, keepdim=True)
    return (input - mean) / (standard_deviation + 0.0001)
```

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345
OpInfo incorrectly advertises lu_solve support on CUDA even when compiled without magma
Labels: module: tests, triaged, better-engineering, module: linear algebra

### 🐛 Describe the bug

```
ERROR: test_comprehensive_linalg_tensorsolve_cuda_complex128 (__main__.TestDecompCUDA)                     
...
RuntimeError: Calling torch.lu_solve on a CUDA tensor requires compiling PyTorch with MAGMA. Please rebuild
 with MAGMA.
```


### Versions

master

cc @mruberry @jianyuh @nikitaved @pearu @walterddr @IvanYashchuk @xwang233 @Lezcano
OpInfo CUDA bfloat16 support detection is buggy
Labels: module: tests, triaged, better-engineering

### 🐛 Describe the bug

On my CUDA enabled machine:

```
(/private/home/ezyang/pytorch-tmp-env) ezyang@devfair040:~/pytorch-tmp$ nvidia-smi
Fri Apr 29 19:12:09 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro GP100        On   | 00000000:81:00.0 Off |                    0 |
| 26%   33C    P0    31W / 235W |      1MiB / 16278MiB |      0%      Default |
|                               |                      |                  N/A |
```

OpInfo db says that, for example, addbmm supports bfloat16

```
           dtypesIfCUDA=floating_and_complex_types_and(torch.float16,
                                                       *[torch.bfloat16] if (CUDA11OrLater or TEST_WITH_ROCM) else []),
```

but my cublas does not support it

```
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasGemmStridedBatchedExFix(handle, o
pa, opb, (int)m, (int)n, (int)k, (void*)&falpha, a, CUDA_R_16BF, (int)lda, stridea, b, CUDA_R_16BF, (int)ld
b, strideb, (void*)&fbeta, c, CUDA_R_16BF, (int)ldc, stridec, (int)num_batches, CUDA_R_32F, CUBLAS_GEMM_DEF
AULT_TENSOR_OP)`
```

This is probably because my GPU is too old. What we probably need to do is test compute capability instead.

### Versions

master

cc @mruberry
AttributeError: '_thread._local' object has no attribute 'rel_tol' (cannot use TestCase.assertEqual from other threads)
Labels: module: tests, triaged, better-engineering

### 🐛 Describe the bug

In a test case, write:

```
    def test_foo(self):
        import threading
        def foo():
            self.assertEqual(torch.zeros(2), torch.zeros(2))
        t = threading.Thread(target=foo)
        t.start()
        t.join()
```

This will fail with

```
Traceback (most recent call last):
  File "/private/home/ezyang/pytorch-tmp-env/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/private/home/ezyang/pytorch-tmp-env/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "test/test_decomp.py", line 300, in foo
    self.assertEqual(torch.zeros(2), torch.zeros(2))
  File "/private/home/ezyang/pytorch-tmp/torch/testing/_internal/common_utils.py", line 2202, in assertEqual
    rtol_override=self.rel_tol,
  File "/private/home/ezyang/pytorch-tmp/torch/testing/_internal/common_device_type.py", line 302, in rel_tol
    return self._tls.rel_tol
AttributeError: '_thread._local' object has no attribute 'rel_tol'
```

A non-contrived situation this can occur is a `__torch_dispatch__` invocation inside of an autograd backwards thread; you are unable to do test asserts from inside this callback.

### Versions

master

cc @mruberry
support FSDP with AMP
Labels: oncall: distributed, module: fsdp

### 🚀 The feature, motivation and pitch

Replacing the DDP with FSDP results into a loss difference, where the AMP is enabled.

Do we have some document to help users enable both FSDP and AMP ?

Currently simply changing from DDP to FSDP can lead to some loss regression as below.
![image](https://user-images.githubusercontent.com/2343640/166081973-a0099713-3be0-4ff7-942b-cbb3eb661158.png)


### Alternatives

_No response_

### Additional context

_No response_

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang
`torch.linalg.cond` has different results for tensor requiring autograd
Labels: module: autograd, triaged, module: linear algebra

### 🐛 Describe the bug

`torch.linalg.cond` has different results for tensor requiring autograd if torch version is 1.11.0

```python
import torch
results = dict()
A_tensor = torch.rand([2485, 2485], dtype=torch.float32)

A1 = A_tensor.clone().detach()
results["res_1"] = torch.linalg.cond(A1)

A2 = A_tensor.clone().detach()
results["res_2"] = torch.linalg.cond(A2)

A3 = A_tensor.clone().detach().requires_grad_()
results["res_3_grad"] = torch.linalg.cond(A3)

print(results)
# {'res_1': tensor(1267001.1250), 'res_2': tensor(1267001.1250), 'res_3_grad': tensor(1266998.8750, grad_fn=<SqueezeBackward1>)}
```
They will output the same results when torch is 1.10.0

### Versions

torch: 1.11.0

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @jianyuh @mruberry @walterddr @IvanYashchuk @xwang233
Random Generator for Dropout
Labels: feature, triaged, module: random

### 🚀 The feature, motivation and pitch

I am doing some study on dropout, and want to keep the same randomness for dropout, but may change randomness in other parts of the code.
Can we add rng as an extra argument for random generator of Dropout (like JAX did) so that we can have better control of randomness under specific study?

### Alternatives

_No response_

### Additional context

_No response_

cc @pbelevich
[JIT][Autocast] Batchnorm folding pass during freezing doesn't preserve types
Labels: oncall: jit

### 🐛 Describe the bug

The optimize_numerics pass in the freezing pass interferes with the types used by the autocasting freezing pass.

Autocast() during freezing will cast weights to half precision; but then FoldFrozenConvBatchnorm turns them into full precision, causing errors when the model is run. 

```python
import torch
from typing import Final
net = torch.nn.Sequential(torch.nn.Conv2d(3, 64, (3, 3), 1, 1, bias=False),
                                      torch.nn.BatchNorm2d(64),
                                      torch.nn.PReLU(64)).cuda().eval()
img = torch.randn(3, 3, 112,112).cuda()
with torch.cuda.amp.autocast(True):
    net = torch.jit.trace(net, img, check_trace=False)
    net = torch.jit.freeze(net)
print(net.graph)
feature = net(img)  # forward pass
```

Workaround for now - you can disable optimize_numerics during freezing with `torch.jit.freeze(model, optimize_numerics=False)`

thanks to @xsacha who found the issue & provided the repro

### Versions

master
[jiterator] perf regression when jiterating few ops for complex dtype
Labels: module: performance, triaged, module: complex, module: jiterator

### 🐛 Describe the bug

Benchmarks (thanks @anjali411 for suggesting) of a few operators before and after jiterating for complex dtype shows that there is perf-regression in some cases (also jiterated version is faster in some cases).

Ref: https://gist.github.com/kshitij12345/6581f0aab6f2fc16648bbf0c4c3219fa

As can be seen from the benchmark, there is a noticeable regression for `sigmoid`. Diving a bit deeper for `sigmoid`, the regression is caused due to use of `operator/`. Root cause being jiterated version (based on llvm) uses a different implementation from non-jiterated version. (Thanks @anjali411 @ngimel @zasdfgbnm for figuring that out)

Hence going forward we should be more careful while jiterating an op for complex dtype and be sure to benchmark the same.

### Versions

master

cc @VitalyFedyunin @ngimel @ezyang @anjali411 @dylanbespalko @mruberry @Lezcano @nikitaved
Pip packaging and publishing improvements in pytorch wheels for better integration with poetry
Labels: oncall: binaries

### 🐛 Describe the bug

[Poerty](https://github.com/python-poetry/poetry) is a great build toolchain for python applications that lean on pip packaging. 

Please see a detailed thread here https://github.com/python-poetry/poetry/issues/4231#issuecomment-1109470128 on why the integration of applications written using PyTorch eco-system fails. As mentioned in the follow-up reply from @abn here https://github.com/python-poetry/poetry/issues/4231#issuecomment-1111400536 that some improvements in our PyTorch packages are published and indexed can greatly improve the experience and integration. To highlight the requests:

1. The lack of file hashes being exposed in the repository means that Poetry has to download each artifact unnecessarily. As per [PEP 503](https://peps.python.org/pep-0503/). this can be appended to the file URL as #<hashname>=<hashvalue>. This can be done easily by the index admins I believe.
2. While fixing (1) will reduce the number of required downloads, files still will be downloaded again when there is a need to inspect metadata (ie. Requires-Dist, etc. from the wheel). This can be avoided too if [PEP 658](https://peps.python.org/pep-0658/#specification) gets implemented. I suspect this will need to also be supported by Poetry as well.
3. Would be great if requesting a non-existant resource returns a 404 instead of 403. If I understand correctly, this is because they have not set s3:ListBucket permission for public users (assuming they use S3 for this).

I am not 100% sure this ticket is the best place on the main torch repo but I did not find any CI-specific repo either in [pytorc](https://github.com/pytorch). Please help direct if I got the main repo to report this wrong. 

### Versions

Has been tried with multiple versions including 1.11

cc @ezyang @seemethere @malfet
[numpy] Missing Tensor-Scalar support for multiple binary ops 
Labels: triaged, module: numpy, module: python frontend

### 🐛 Describe the bug

While trying to update the test sample in https://github.com/pytorch/pytorch/pull/76364, found that there are multiple binary ops in PyTorch which don't support Tensor and Scalar inputs.

```python
import torch
import numpy as np

t = torch.zeros(3, 3)
a = t.numpy()

# atan2 - Works on Numpy
np.arctan2(a, 1)
# torch.arctan2(t, 1)

# fmax - Works on Numpy
np.fmax(a, 1)
# torch.fmax(t, 1)

# fmin - Works on Numpy
np.fmin(a, 1)
# torch.fmin(t, 1)

# Similarly other ops that work on NumPy or SciPy
# heaviside
# hypot
# igamma - Works on Scipy (gammainc)
# igammac - Works on Scipy (gammaincc)
# isclose
# logical_and
# logical_or
# logical_xor
# maximum
# minimum
# nextafter

```

### Versions

master

cc @mruberry @rgommers
Better handling for exec
Labels: triaged, enhancement, oncall: pt2, module: dynamo

https://github.com/facebookresearch/torchdynamo/pull/184 introduces a band-aid fix to skip Dynamo on the `exec` frame. This issue is so that we don't forget to find a better solution.

For some reason, running the exec frame through Dynamo changes the `f_globals` dictionary. We can use the tests in the above PR to repro the failure.

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym @yanboliang @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
Non target rank receives result of 'reduce' op when backend is 'gloo'
Labels: oncall: distributed

### 🐛 Describe the bug

I run code using two rank.

```
import os
import torch
import torch.distributed as dist
from torch.multiprocessing import Process
os.environ['MASTER_ADDR'] = '127.0.0.1'
os.environ['MASTER_PORT'] = '29500'
backend = 'gloo'
dist.init_process_group(backend)
rank = dist.get_rank()
# simple dist
if rank == 0:
  print("pid is {}, rank is {}".format(os.getpid(), rank))
else:
  print("pid is {}, rank is {}".format(os.getpid(), rank))
if rank == 0:
  input = torch.tensor([[1, 2], [3, 4]], device="cpu")
else:
  input = torch.tensor([[4, 5], [6, 7]], device="cpu")

if rank == 0:
  dist.reduce(input, dst=0, async_op=False)
else:
  dist.reduce(input, dst=0, async_op=False)
print('Pid is ', os.getpid(), ', Rank ', rank, ', output is: ', input)
```

I got result as below:

> Pid is  3479174 , Rank  0 , output is:  tensor([[ 5,  7],
>         [ 9, 11]])
> Pid is  3479067 , Rank  1 , output is:  tensor([[ 4,  5],
>         [ 9, 11]])

It seems that non dst rank also receives part result ([9, 11])  when backend is 'gloo'.  But the document says only dst rank receive final result.  https://pytorch.org/docs/stable/distributed.html .   'nccl' backend has no this problem.
Is this right? 

### Versions

'1.12.0a0+git0aa3c39'

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang
`torch.clamp` does not distribute gradients as element-wise`min/max` do
Labels: module: autograd, triaged, actionable

### 🐛 Describe the bug

Since #59669 was merged, element-wise `max/min` evenly distribute gradients for all values that are equal in self and the other. However, `clamp`, which can be equivalently expressed using `max` and `min`, does not follow the way, resulting in different gradients between `clamp` and `min/max`. I suppose the similar fix with #59669 should be also applied to `clamp` so that it distributes gradients as `min/max` do.

```py
>>> import torch
>>> a = torch.tensor([-2.,-1.,0.,1.,2.], requires_grad=True)
>>> b = torch.full((5,), -1.); b.requires_grad = True
>>> c = torch.full((5,), 1.); c.requires_grad = True
>>> torch.clamp(a, b,c).sum().backward()
>>> a.grad, b.grad, c.grad
(tensor([0., 1., 1., 1., 0.]), tensor([1., 0., 0., 0., 0.]), tensor([0., 0., 0., 0., 1.]))
>>> a.grad, b.grad, c.grad = None, None, None
>>> torch.min(torch.max(a, b), c).sum().backward()
>>> a.grad, b.grad, c.grad
(tensor([0.0000, 0.5000, 1.0000, 0.5000, 0.0000]), tensor([1.0000, 0.5000, 0.0000, 0.0000, 0.0000]), tensor([0.0000, 0.0000, 0.0000, 0.5000, 1.0000]))
```


### Versions

```
Collecting environment information...
PyTorch version: 1.11.0+cpu
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.10.2
Libc version: glibc-2.27

Python version: 3.10.0 (default, Oct 15 2021, 11:40:42) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.15.0-169-generic-x86_64-with-glibc2.27
Is CUDA available: False
CUDA runtime version: Could not collect
GPU models and configuration: Could not collect

Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.11.0+cpu
```

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
Fix layout of masked output when all sparse dimensions are reduced
Labels: module: sparse, triaged

### 🐛 Describe the bug

[From discussion on #75454](https://github.com/pytorch/pytorch/pull/75454#discussion_r858083103), sparse reduction ops like torch.sparse.sum return a dense result if all sparse dimensions are reduced https://github.com/pytorch/pytorch/blob/a5bc02aeb20c484eb7aab045a8f5609700a75533/aten/src/ATen/native/sparse/SparseTensorMath.cpp#L1297-L1300
but `_masked` operators currently [expect the layout of the output to be the same as the layout of the input](https://github.com/pytorch/pytorch/blob/master/test/test_masked.py#L298-L303) and the current implementation of _masked.sum [reflects this](https://github.com/pytorch/pytorch/blob/master/torch/_masked/__init__.py#L701-L708). 

Creating this issue to track this


### Versions

N/A

cc @nikitaved @pearu @cpuhrsch