gopikrsmscs/torch-issues huggingface dataset:
Mismatch in clang toolchain lead to binary incompatibilities on M1 between torch and torchvision
Labels: oncall: binaries, module: ci, triaged, module: macos, module: arm

### 🐛 Describe the bug

The M1 unit-tests [fail](https://github.com/pytorch/vision/runs/6806004576?check_suite_focus=true) with the following error:
```
self = <OpOverloadPacket(op='torchvision.roi_align')>
args = (tensor([[[[ 8.2470e+00,  1.3783e+01,  1.3401e+01,  ...,  2.4769e+00,
            2.6093e+00,  6.0664e+00],
          ...68.9164],
        [  0.0000, 175.5915, 200.8642, 382.0600, 212.2028]],
       dtype=torch.float64), 0.25, 7, 7, 2, ...)
kwargs = {}
    def __call__(self, *args, **kwargs):
        # overloading __call__ to ensure torch.ops.foo.bar()
        # is still callable from JIT
        # We save the function ptr as the `op` attribute on
        # OpOverloadPacket to access it here.
>       return self._op(*args, **kwargs or {})
E       RuntimeError: torchvision::roi_align is not yet supported with named tensors. Please drop names via `tensor = tensor.rename(None)`, call the op with an unnamed tensor, and set names on the result of the operation.
```

~~The last time the tests run successfully was on the 6th of June at commit d4a03fc02d0566ec97341046de58160370a35bd2. Unfortunately the days after that our CI broke for unrelated issues, so I guess it could be a commit commits from Core between the 6th and 9th of June.~~

EDIT:
https://github.com/pytorch/vision/runs/6797676717?check_suite_focus=true is passing (June 8, 6:43 PM GMT+2), ee6f6ec214aca19d7dfb2f24bffd26e4e083216b  => it is passing on stable pytorch 1.11 (see https://github.com/pytorch/vision/runs/6797676717?check_suite_focus=true#step:3:88), `--pre` was added later: https://github.com/pytorch/vision/commit/e1f46d42f2acd1039e9a459316bc3daa92d3c983


### Versions

Latest main branch.

cc @ezyang @seemethere @malfet @pytorch/pytorch-dev-infra @albanD
_make_elementwise_unary_reference and other function factories in torch._refs don't set __name__ correctly
Labels: triaged, module: primTorch

### 🐛 Describe the bug

```
>>> import torch._refs
>>> torch._refs.abs.__name__
'_ref'
```

This is problematic for FX because FX uses the `__name__` to recompile the function back into Python, and cannot do so if it looks like this; it will silently miscompile the code.

functools.wraps does not help as there is no "real" function to wrap.

### Versions

master

cc @ezyang @mruberry @ngimel
PyTorch/XLA's DDP XLABackend is broken by upstream change
Labels: oncall: distributed, triaged, module: xla, module: ddp

### 🐛 PyTorch/XLA's DDP hangs

 In https://github.com/pytorch/xla/commit/841e0d3cab9555ce6ca03de5e0498c1310a01ba2 @hjm-aws introduced the `XLABackend` which enabled the DDP for PyTorch/XLA. However this is currently broken due to that subclassing of `Work` class(PyTorch/XLA has a `WorkXla`) is broken. The result is that when `wait` is being called `WorkXla::wait` is not correctly trigger and the default wait will be used and process will hang. 

More detail XLA issue can be found in https://github.com/pytorch/xla/issues/3625

### Versions

PyTorch: nightly
PyTorch/XLA: nightly

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501 @bdhirsh
Redundant info are saved when using torch.save to save part of torch.tensor
Labels: module: serialization, triaged

### 🐛 Describe the bug

I am trying to save part of a very large torch.tensor by indexing it. But the saved partial tensor is always as the same size as the original one. 

To reproduce:
```python
import torch

a = torch.ones([100,100])
torch.save(a, 'full.pt') #file full.pt is 40.7kB (size is reported by nautilus on Ubuntu 20.04)
torch.save(a[0], 'partial0.pt') #file partial0.pt is 40.7kB
torch.save(a[:1], 'partial1.pt') #file partial1.pt is 40.7kB


b = torch.ones([1,100])
torch.save(b, 'partial2.pt') #file partial2.pt is 1.1kB
```
But when I load the file ```partial0.pt``` and ```partial1.pt``` back to torch.tensor, they are of the size that I want. 


### Versions

PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.14.0-1042-oem-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.6.124
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3070
Nvidia driver version: 510.47.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] torch==1.11.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] libblas                   3.9.0            12_linux64_mkl    conda-forge
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0           py310h7f8727e_0  
[conda] mkl_fft                   1.3.1           py310hd6ae3a3_0  
[conda] mkl_random                1.2.2           py310h00e6091_0  
[conda] numexpr                   2.8.0           mkl_py310h22e654f_0    conda-forge
[conda] numpy                     1.22.3          py310hfa59a62_0  
[conda] numpy-base                1.22.3          py310h9585f30_0  
[conda] pytorch                   1.11.0          py3.10_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch


cc @mruberry
torchvision.models.mobilenetv3 can't save pre-trained model to custom dir?
Labels: triaged, module: vision

### 📚 The doc issue

import torchvision.models as models
mobile_v3 = models.mobilenet_v3_small(pretrained=True)
this method load the pre-trained model to defaul catch dir?
in this function  _mobilenet_v3_model
the line 
"state_dict = load_state_dict_from_url(model_urls[arch], progress=progress)
 model.load_state_dict(state_dict)"
if I want to save pre-trained model to custom dir .how to set the 'model_dir' to load_state_dict_from_url() this method?

### Suggest a potential alternative/fix

_No response_

cc @fmassa @vfdev-5 @pmeier
Hide or fuse TupleConstruct / TupleUnpack from tensorboard graph
Labels: triaged, module: tensorboard

### 🚀 The feature, motivation and pitch

Tensorboard graphs generated by Summary.add_graph are cluttered with successions of TupleUnpack and TupleConstruct when chaining modules that take and return several arguments. Tuple packing and unpacking is a basic and transparent programming pattern in python, I cannot think of a situation where we want to explicitly view it in the graph.

I would be more legible if forward arguments and tuple return values were not grouped into a single graph node.

### Alternatives

_No response_

### Additional context

Example:

```py
import torch
from torch import nn
from torch.utils.tensorboard import SummaryWriter


class Layer(nn.Module):
    def __init__(self):
        super().__init__()

        self.linear = nn.Linear(64, 64)

    def forward(self, a, b):
        return self.linear(a), self.linear(b)


class Model(nn.Module):
    def __init__(self):
        super().__init__()

        self.layer1 = Layer()
        self.layer2 = Layer()

    def forward(self, x):
        a, b = self.layer1(x, x)
        c, d = self.layer2(a, b)
        return a + b + c + d


model = Model()

summary = SummaryWriter(".")
summary.add_graph(model, input_to_model=(torch.rand([1, 64])))
```

![Capture d’écran du 2022-06-08 13-50-41](https://user-images.githubusercontent.com/3764009/172610553-8d943f15-0624-4823-9186-6d7920088147.png)
About the source code of 
Labels: triaged, topic: docs

### 📚 The doc issue

I am actually trying to find the source implementation of a very simple function - cosine similarity, due to some concerns from its computation scheme.

However, when I was looking at [the website](https://pytorch.org/docs/stable/generated/torch.nn.functional.cosine_similarity.html), I cannot locate the source code. I also failed to find the function from the original GitHub repository (or I am being too lazy to dig everything. I stopped after several migrating using F12 on VScode...).

I am aware that such "torch.nn.functional" functions are implemented not purely in Python but wrapped with Cython (correct me). But still I would like to see the inner working mechanism. Could you please kindly point me to where the actual implementation locates in the repo? Cython or even C/C++ code will be fine. Thanks!

### Suggest a potential alternative/fix

I actually would like to suggest to at least add one "source location" for the functions that can be browsed in the website. It does not have to be Python code or even code necessarily contained in the repo, but at least something making people feel whatever they will feel to open ideas or discussions.
Error with Named Tensors and multiple threads
Labels: triaged, module: named tensor

### 🐛 Describe the bug

When using Named Tensors I got an exception when the number of operations was too large.
Steps to reproduce:
```python
torch.randn((50000, 2), names=('K', 'M')).mean('M').mean('K')
```
We get the exception:
```
RuntimeError: aten::unsqueeze is not yet supported with named tensors. Please drop names via `tensor = tensor.rename(None)`, call the op with an unnamed tensor, and set names on the result of the operation.
```
However, if we instead run
```python
torch.randn((500, 2), names=('K', 'M')).logsumexp('M').mean('K')
```
we get no errors.
Moreover, if we run the original code but without multithreading, there is no error either:
```python
torch.set_num_threads(1)
torch.randn((50000, 2), names=('K', 'M')).mean('M').mean('K')
```


### Versions

```
Collecting environment information...
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.4 (x86_64)
GCC version: Could not collect
Clang version: 13.1.6 (clang-1316.0.21.2.5)
CMake version: version 3.11.1
Libc version: N/A

Python version: 3.9.7 (default, Sep  3 2021, 12:45:31)  [Clang 12.0.0 (clang-1200.0.32.29)] (64-bit runtime)
Python platform: macOS-10.16-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.4
[pip3] torch==1.11.0
[pip3] torchaudio==0.11.0
[pip3] torchvision==0.12.0
[conda] Could not collect
```

cc @zou3519
Improve PrimTorch testing for view consistency.
Labels: module: tests, triaged, module: viewing and reshaping, module: primTorch

### 🚀 The feature, motivation and pitch

Currently, we check view consistency by checking if `a._is_view` and `b._is_view` are both correct: https://github.com/pytorch/pytorch/blob/master/test/test_ops.py#L394

However, this is not particularly correct. As @ezyang notes (https://github.com/pytorch/pytorch/pull/78994#discussion_r891721598), view is an autograd concept. Thus, even though the ATen version of stack returns a view (https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/TensorShape.cpp#L2213), `a._is_view` returns False for the ATen implementation.

This is problematic for a variety of ops, including `rot90`: https://github.com/pytorch/pytorch/pull/78080#discussion_r885104613

cc: @mruberry @ngimel @ezyang 

### Alternatives

_No response_

### Additional context

_No response_

cc @mruberry @ezyang @ngimel
CI workflow creates too many tags in RSS feed
Labels: module: ci, triaged

Hi, I use GitHub's RSS feed to track the release of many repos. However, it seems that the PyTorch RSS feed is flooded by CI workflow (see https://github.com/pytorch/pytorch/releases.atom and https://github.com/pytorch/pytorch/tags). 

Is it possible that CI workflow is no longer tagged so that I receive only updates for important builds in the RSS feed? 

cc @seemethere @malfet @pytorch/pytorch-dev-infra
Multi-node, Multi-GPU set up tutorial for Slurm cluster
Labels: oncall: distributed, triaged

### 🚀 The feature, motivation and pitch

We need to have a tutorial/doc about setting up Multi-node, Multi-GPU environment for Slurm cluster. https://github.com/aivanou/disttraining/tree/main/slurm is a good starting point, but we need to furnish it and make sure it works and publish a tutorial/doc on it.

### Alternatives

_No response_

### Additional context

_No response_

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
inspect.signature.bind is not supported
Labels: triaged, oncall: pt2, module: dynamo

This is being used in elementwise_type_promotion_wrapper in PrimTorch which means I cannot nopython through PrimTorch refs.

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh @mlazos @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire @mruberry  maybe we just shouldn't use this API in PrimTorch
[MetaIssue] Investigate if we should be reusing primtorch formulas for `is_dynamic`
Labels: triaged, lazy

1. Investigate how hard it would be to integrate https://github.com/pytorch/pytorch/blob/1c1c68b1e31351030843c773e73fab736cbfaf1b/torch/_meta_registrations.py formulas to compute `is_dynamic` in LTC
2. Cross reference how many MaskRCNN ops that are not covered with existing SSA shape functions already have shape functions in https://github.com/pytorch/pytorch/blob/1c1c68b1e31351030843c773e73fab736cbfaf1b/torch/_meta_registrations.py and decomps in https://github.com/pytorch/pytorch/blob/master/torch/_decomp/decompositions.py
3. Sync with @ezyang about his timeline maybe it would be possible to prioritize some ops for MaskRCNN?
4. Benchmark JIT SSA vs meta shape propagations.
5. Communicate to Google if we decide to transition to meta functions
6. If we decide to transition to meta functions: E2E PoC. Hopefully this will build enough infrastructure.
7. If we decide to transition to meta functions:  Migration plan

This is based on @Gamrix 's doc: https://docs.google.com/document/d/1DhNxC9mtQdSXOktlSIoAKUCgN95Dx998FHcoPhJdJU8/edit#heading=h.l6gc8x2z7bv5
DDP Freezes w/ No Output for PyTorch Geometric GNN Multi-GPU Node Classification
Labels: oncall: distributed, triaged, module: ddp

### 🐛 Describe the bug

For some mysterious reason self.reducer._rebuild_buckets() from within the DDP forward call, hangs on the second iteration of calling the DDP module forward pass. It only happens on node property prediction for PyTorch Geometric, link prediction works with no issues. Originally the issue was found when predicting missing information in the Microsoft Academic Graph, but the issue has been isolated down to be reproducible with node classification on synthetic data. [Repro](https://github.com/puririshi98/debug_MAG_pyg/blob/main/synthetic_repro.py)
The lack of output when it freezes is making the debugging impossible for @rusty1s  as well.
Hacky fix:
```
            try:
                bool2 = self.reducer._rebuild_buckets()
            except:
                bool2 = False
            if torch.is_grad_enabled() and bool2:
                logger.info("Reducer buckets have been rebuilt in this iteration.")
                self._has_rebuilt_buckets = True
```
as opposed to:
```
            if torch.is_grad_enabled() and self.reducer._rebuild_buckets(:
                logger.info("Reducer buckets have been rebuilt in this iteration.")
                self._has_rebuilt_buckets = True
```

Not sure the cause or what the correct long term solution is.

### Versions

latest pytorch and pytorch-geometric installs should reproduce the issue

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
Add Autograd Support for Nested Tensor
Labels: triaged, module: nestedtensor

## Summary
This is the base issue with sub issues stored and tracked here.

Currently NestedTensors do not support autograd. The goal of this issue is to change that. There are a few parts to making this work and the sub components will be tracked in related issues referenced below.

#### Linked Issues:
- #79040
- #79447
- #79044
- #83773
Nice to have but there is a work around:
- #79046
- #79048

cc @cpuhrsch
Implement SymbolicIntNode interface for lazy (i.e. lazy::SymbolicIntNode)
Labels: triaged, lazy

These need to be implemented for lazy/xla tensors:

https://github.com/pytorch/pytorch/blob/eca474f93cfaa19786d665655378411edc370bbe/c10/core/SymbolicIntNode.h#L17-L63


by constructing IR nodes similar to https://github.com/pytorch/pytorch/blob/master/torch/csrc/lazy/core/tensor_impl.cpp#L91-L92

The IR is defined in https://github.com/pytorch/pytorch/blob/master/torch/csrc/lazy/ts_backend/dynamic_ir.h
Devirtualize `sym_sizes`. It still has to work for python tensor subclasses and LTC/Xla
Labels: triaged, lazy

The issue is as follows:

1. THPVariable_size now returns `sym_sizes`
2. Python tensors set `CustomSizes`
	1. Python tensors needs to run the default implementation of `sym_sizes`
	2. `CustomSizes` makes us use `sym_sizes_custom` which throws
3. LTC sets `CustomSizes`
	1. LTC needs to *reimplement* `sym_sizes`
4. Nested Tensors sets `CustomSizes`
	1. Nested Tensors needs to *reimplement* `sym_sizes`

I hacked around the issue by removing a check for the size policy in `sym_sizes` making it virtual and making it always call the default implementation. This inhibits compiler optimizations.
Random functions should infer device from user-specified Generator
Labels: triaged, enhancement, module: random

### 🚀 The feature, motivation and pitch

As of PyTorch 1.11.0, PyTorch's functions for drawing random numbers will not infer the appropriate device from a user-provided RNG:

```python
gen = tr.Generator(device="cuda:0")
tr.rand(1, generator=gen)   # RuntimeError: Expected a 'cpu' device type for generator but found 'cuda'
tr.rand(1, generator=gen, device=gen.device)  # OK
```

I am recommending that functions like `tr.rand` should infer their device from the RNG when `tr.rand(... device=None)`

```python
# pitch: these should all be OK

gen = tr.Generator(device="cuda:0")
tr.rand(1, generator=gen)
tr.rand(1, generator=gen, device=None)  
tr.rand(1, generator=gen, device=gen.device)
```



### Alternatives

_No response_

### Additional context

I have found that the current behavior makes it all too easy to write buggy code, since writing `tr.rand(1, generator=gen)` "feels" like it should always work. 

Furthermore, it is challenging to catch this potential bug in tests, unless you already anticipate this exact issue (it is not very likely that people would think to test cases where the generator is placed on various devices).

cc @pbelevich
Corner cases of ShardedTensor checkpoint when using TorchRec
Labels: oncall: distributed, module: checkpoint, triaged, sharded_tensor, release notes: distributed (sharded)

### 🐛 Describe the bug

Our team is trying torchrec in our business. It seems that torchrec does not support its own model save/load api that supports resharding, e.g. training with multiple gpus while inferencing with one. I've tried to use the recently introduced distributed checkpoint for ShardedTensor (https://github.com/pytorch/pytorch/pull/76123) according to the suggesting here: https://github.com/pytorch/torchrec/issues/346. It's really nice to have a unified way to store tensors and sharded tensors across machine, but I encountered some corner cases that are not working at the moment, for examples:

1. In torchrec, some embeddings (sharded tensors) only locate on part of the gpus, but we only stored the metadata of rank 0 now, which will trigger error when loading.
2. Some small embeddings are planned as data-paralleled when using multiple cards, which will be stored into a Tensor for each rank, but they are planned as row-wise when using 1 gpu, which will be stored as a ShardedTensor and cause type mismatch when loading.
3. We hope not to use the DMP (distributed model parallel) wrapper when doing inference so that the JIT tracing will work. However, all tensors in the model will be plain Tensor. Users of FSDP may also encounter this problem.

The first bug here can be solved separately and I will try to make a PR for it :) However, I think the last 2 is relevant to the design choices of pytorch and torchrec, for instance, shall we allow loading sharded tensor into a single tensor if the total size matches and shall the all tensors in torchrec in be labeled as data-parallel when training with only one gpu? I wonder if you could share some of your ideas for these problems. I would be great if all of these have be solved in the 'snapshot API' mentioned here: https://github.com/pytorch/torchrec/issues/346#issuecomment-1145203954. And if that's the case, could you share some of the progress of this API?

Our team love torchrec as it has brought us great flexibility of pytorch and the speed from the gpu native design! And of course we love pytorch! Therefore we hope to be able to use torchrec in practice soon :) I'd love to help if there is anything I can do~

Thank you for your time on this long issue and looking forward to your ideas!

gently ping @kumpera @dstaay-fb @colin2328 

### Versions

nightly

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
Multi30k can't be downloaded the destination domain can't be reached
Labels: triaged

### 🐛 Describe the bug

Could not get the file at http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/training.tar.gz. 

This website can't be reached!

Is there a substitution?

### Versions

torch                             1.11.0
torchaudio                        0.11.0
torchdata                         0.3.0
torchtext                         0.12.0
torchvision                       0.12.0
torchscript jit trace support custom op without  specific csrc and .so
Labels: oncall: jit

### 🚀 The feature, motivation and pitch

Despite of using a custom op in csrc way. sometimes, users just want wrap a block of lines into a single func, the using it as a single op, rather than tace them one by one.

For example, I have a block calucation like this:

```
a  = torch.sum(a*b, dim=1)
c = torch.einsum("ij,jk->ik", a, b)
```

I wanna using a single func as a Op like:

```
def my_dead_awesome_op(a, b):
    a  = torch.sum(a*b, dim=1)
    c = torch.einsum("ij,jk->ik", a, b)
```

and I wanna using jit trace this `my_dead_awesome_op` out, is there a way to do so?

### Alternatives

_No response_

### Additional context

_No response_
Doc on index of CPU Device seems wrong
Labels: module: cpp, triaged

### 🐛 Describe the bug

Comment on this line says "When the device type is CPU, the device index must be zero".

https://github.com/pytorch/pytorch/blob/bf4b6d0dce/c10/core/Device.h#L29

However, I run PyTorch and found a CPU tensor's device has index = -1. Maybe the comment is wrong.

### Versions

```
PyTorch version: 1.11.0+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 11 (bullseye) (x86_64)
GCC version: (Debian 10.2.1-6) 10.2.1 20210110
Clang version: 13.0.0
CMake version: version 3.18.4
Libc version: glibc-2.31

Python version: 3.9.2 (default, Feb 28 2021, 17:03:44)  [GCC 10.2.1 20210110] (64-bit runtime)
Python platform: Linux-5.10.0-13-amd64-x86_64-with-glibc2.31
Is CUDA available: False
CUDA runtime version: 11.4.120
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] torch==1.11.0
[conda] Could not collect
```

cc @jbschlosser
TorchInductor failing inference models tracker
Labels: triaged, oncall: pt2, module: inductor

This is meta issue to track multiple issues with TorchInductor inference models in TorchBench.

Note all the models tracked by pytorch/torchdynamo#331 also don't work with TorchInductor.

Reproduce with:
`./torchbench.py -dcuda --inductor --float16 --no-skip -k <name>`

Updated list as of 8/26/2022:

- [x] pytorch/pytorch#93788
- Accuracy Error:
  - [x] pytorch/torchdynamo#622
  - [x] pytorch/torchdynamo#838
  - [x] pytorch/torchdynamo#634
  - [x] pytorch/torchdynamo#630 
  - [x] pytorch/torchdynamo#1039

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @mlazos @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire
[forwardAD] torch.no_grad has no effect under forward_ad
Labels: triaged, module: forward ad

### 🐛 Describe the bug

```python
import torch
import torch.autograd.forward_ad as fwAD

primal = torch.ones(10, 10)
tangent = torch.ones(10, 10)

def fn(x):
    return x ** 3

with fwAD.dual_level():
    dual_input = fwAD.make_dual(primal, tangent)
    with torch.no_grad():
        dual_output = fn(dual_input)

    jvp = fwAD.unpack_dual(dual_output).tangent
    torch.testing.assert_close(jvp, torch.ones(10, 10))  # Error

```

cc: @soulitzer @zou3519 

### Versions

master
Can we have Additive Attention?
Labels: triaged, oncall: transformer/mha

### 🚀 The feature, motivation and pitch

Like MultiheadAttention which was proposed in **`Transformer`** paper similarly, can we have Additive attention
 proposed in `**Neural Machine Translation by Jointly Learning to Align and Translate**`.

### Alternatives

`nn.MultiheadAttention`

### Additional context

_No response_

cc @jbschlosser @bhosmer @cpuhrsch
library libshm.dylib is missing
Labels: high priority, triage review, oncall: binaries, triaged, module: macos

i have basically the same problem  but the library libshm.dylib is missing.

```
dlopen(/usr/local/lib/python3.9/site-packages/torch/_C.cpython-39-darwin.so, 2): Library not loaded: @loader_path/libshm.dylib
  Referenced from: /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_python.dylib
  Reason: image not found

```
all answers i found to fix that say to[ install libomp](https://github.com/pytorch/pytorch/issues/20030), which i reinstalled but it didnt fix the issue, though for others [that seemed to work in previous years](https://discuss.pytorch.org/t/importerror-on-mac-with-pytorch-1-1/44305). 

i cant find the library libshm in any lib folders on my machine. Im wondering if its included somewhere in libomp and how to point pytorch to it, or if it can be downloaded from somewhere else?

i have the current torch version installed 1.11.0 with Python 3.9 on Mac Big Sur.

Found the [libshm library](https://github.com/afeldman/libshm) required for sharedmemory alloc on linux machines.
Does PyTorch require a bugfix to include this dylib? Or is there any other workaround? @malfet

_Originally posted by @jskye in https://github.com/pytorch/pytorch/issues/36941#issuecomment-1146030734_

cc @ezyang @gchanan @zou3519 @seemethere @malfet @albanD
Add type() support for mps backend
Labels: triaged, actionable, module: mps

As reported in https://discuss.pytorch.org/t/m1-macos-12-3-torchvision-ops-nms-error/152887/3?u=alband

Calling `Tensor.type()` on an mps Tensor returns `torch.mps.FloatTensor` which the parser code does not handle (as it is special cased for cpu/cuda.
We could update the parsing logic to handle mps at https://github.com/pytorch/pytorch/blob/129d9dbb1581ead52061f140cba200662373574c/torch/csrc/utils/tensor_types.cpp#L48

But isn't type() deprecated? Do we want to add more to it? cc @kulinseth @albanD @ezyang
Strange tracing result with torchscript
Labels: oncall: jit

### 🐛 Describe the bug

I have a model, which weights are loaded one by one from standalone bin files, so I using nn.Parameters to load then.

But the trace to torchscript, it gives me weired result:

![image](https://user-images.githubusercontent.com/21303438/172114996-45119083-4997-467f-b394-be3bbfc00f37.png)

this is not expected, since these are not inputs, in onnx, it works normal:

![image](https://user-images.githubusercontent.com/21303438/172115187-ee2382a3-be5b-4b9d-8590-67dd544b7e14.png)


how should make torchscript traced model more user friendly?

### Versions

1.11
TorchInductor missing ops tracker
Labels: triaged, oncall: pt2

The following ops are using `ir.FallbackKernel` via `make_fallback()` in [lowering.py](https://github.com/pytorch/torchdynamo/blob/main/torchinductor/lowering.py#L894) and appear in benchmarks.  We should rewrite them to use decomps or lowerings.


- [x] pytorch/pytorch#93650
- [ ] aten.grid_sampler_2d_backward (higher priority)
- [ ] aten.upsample_bilinear2d_backward (higher priority)
- [ ] aten._adaptive_avg_pool2d_backward
- [ ] aten.upsample_bicubic2d_backward
- [ ] aten._fused_moving_avg_obs_fq_helper
- [x] aten.upsample_nearest3d (needed for FAIR model)
- [ ] aten.avg_pool3d (needed for FAIR model)
- [x] aten.bucketize (needed for internal model) - not targeting for codegen, do a fallback
- [x] aten.prod (needed for research model) - not targeting for codegen, do a fallback

Might not be possible (in a performant way), but currently use fallbacks:
- [ ] aten.convolution_backward (might need to hold of on this if perf doesn't match)
- [ ] aten._cudnn_rnn (might need to hold of on this if perf doesn't match) - not targeting for codegen, do a fallback
- [ ] aten._cudnn_rnn_backward (might need to hold of on this if perf doesn't match) - not targeting for codegen, do a fallback
- [ ] aten._embedding_bag (may have a template internally) (Attempted with https://github.com/pytorch/pytorch/pull/84235, but it's very hard to make it performant and inductor-friendly) 
- [ ] pytorch/pytorch#93631 not targeting for codegen, do a fallback
- [ ] torchvision.roi_align (need to sort of decomps for domain libs)

Done:
- [x] aten._adaptive_avg_pool2d
- [x] aten.nll_loss_forward ([PR](https://github.com/pytorch/pytorch/pull/78491))
- [x] aten.grid_sampler_2d (could use the version [from here](https://github.com/pytorch/pytorch/issues/34704))
- [x] aten.upsample_bilinear2d https://github.com/pytorch/pytorch/pull/80964
- [x] aten.reflection_pad2d_backward (higher priority)
- [x] aten.native_batch_norm_backward (https://github.com/pytorch/pytorch/pull/81522)
- [x] aten.avg_pool2d_backward
- [x] aten.expand_as
- [x] aten.glu_backward (https://github.com/pytorch/pytorch/pull/78919)
- [x] aten.masked_fill_
- [x] aten.max_pool2d_with_indices_backward
- [x] aten.select_scatter
- [x] aten.upsample_nearest2d_backward
- [x] aten.baddbmm
- [x] aten.log1p
- [x] aten.bernoulli_
- [x] aten.conj_physical
- [x] pytorch/torchdynamo#885
- [x] aten.im2col_backward
- [x] aten.native_group_norm_backward (https://github.com/pytorch/pytorch/pull/84037)
- [x] aten.im2col (https://github.com/pytorch/pytorch/pull/84303)
- [x] aten.binary_cross_entropy_with_logits (needed for internal model)
- [x] aten.upsample_bicubic2d (#1349)
- [x] aten.unfold https://github.com/pytorch/pytorch/pull/85629
- [x] aten.unfold_backward https://github.com/pytorch/pytorch/pull/85629
- [x] aten.col2im https://github.com/pytorch/pytorch/pull/85541
- [x] aten.mse_loss_backward (higher priority)
- [x] aten.softplus (needed for research model)
- [x] aten.softplus_backward (needed for research model)

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
torch.fx deepcopy does not copy attributes added to GraphModule or Nodes
Labels: triaged, module: fx

### 🐛 Describe the bug

Hi !

Calling deepcopy on GraphModules does not copy added attributes.
Here is a MWE :
```Python
#imports
import torch
import torch.nn
import torch.fx
import copy

#MWE
net=torch.nn.Linear(0,0)
gm=torch.fx.symbolic_trace(net)
gm.myattr="registered"
gm2=copy.deepcopy(gm)
print(gm2.myattr)
```
which returns `AttributeError: 'GraphModule' object has no attribute 'myattr'`, showing that attribute `myattr` has not been copied.

However, what lead me to this is that I wanted to mark nodes for transformations, and copy the GraphModule before applying the different transformations in order to be able to train transformed copies separately.
Here is an example with the nodes :
```Python
#imports
import torch
import torch.nn
import torch.fx
import copy

#MWE on nodes
net=torch.nn.Linear(0,0)
gm=torch.fx.symbolic_trace(net)
for node in gm.graph.nodes:
    node.is_marked=True
gm2=copy.deepcopy(gm)
for node in gm2.graph.nodes:
    print(node.is_marked)
```
which returns `AttributeError: 'Node' object has no attribute 'is_marked'`.

I think the same holds with attributes added to `gm.graph` etc.



### Versions

PyTorch version: 1.11.0+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Famille
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.10 (tags/v3.9.10:f2f3f53, Jan 17 2022, 15:14:21) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.18363-SP0
Is CUDA available: True
CUDA runtime version: 11.6.124

GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2070
Nvidia driver version: 511.65
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.3
[pip3] torch==1.11.0+cu113
[pip3] torchaudio==0.11.0+cu113
[pip3] torchvision==0.12.0+cu113
[conda] Could not collect

cc @ezyang @SherlockNoMad
[distributed_test.py] Improve `test_barrier`
Labels: oncall: distributed, module: tests, better-engineering

### 🚀 The feature, motivation and pitch

Right now, `test_barrier` tests in `distributed_test.py` work by having rank 0 sleep for a certain amount of time, then run barrier on all ranks, and verify all ranks waited for at least this amount of time. Asserts based on timing / sleep of a certain process have historically been very flaky and hard to get right, so we should refactor this test. 

One suggestion, based on the goal of the test to ensure no ranks pass barrier until all ranks have called into it:
- Maintain a count starting with 0 on all ranks
- All ranks increment the count on each rank by 1
- Barrier()
- Validate each rank count == world_size
- If some rank passed barrier before others called into it, then the validation at the end could be incorrect.

### Alternatives

_No response_

### Additional context

_No response_

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501 @mruberry
Cannot build master on AWS cluster: error: ‘__fatDeviceText’ was not declared in this scope
Labels: module: build, module: cuda, triaged

### 🐛 Describe the bug

```
FAILED: third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/nccl/gloo_cuda_generated_nccl.cu.o /fsx/users/ezyang/pytorch-tmp/build/
third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/nccl/gloo_cuda_generated_nccl.cu.o
cd /fsx/users/ezyang/pytorch-tmp/build/third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/nccl && /fsx/users/ezyang/conda/bin/cmak
e -E make_directory /fsx/users/ezyang/pytorch-tmp/build/third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/nccl/. && /fsx/users/ez
yang/conda/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=Release -D generated_file:STRING=/fsx/users/ezyang/pyto
rch-tmp/build/third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/nccl/./gloo_cuda_generated_nccl.cu.o -D generated_cubin_file:STRI
NG=/fsx/users/ezyang/pytorch-tmp/build/third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/nccl/./gloo_cuda_generated_nccl.cu.o.cub
in.txt -P /fsx/users/ezyang/pytorch-tmp/build/third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/nccl/gloo_cuda_generated_nccl.cu.
o.Release.cmake
In file included from /tmp/tmpxft_0000ff6b_00000000-5_nccl.cudafe1.stub.c:9:0,
                 from tmpxft_0000ff6b_00000000-5_nccl.cudafe1.stub.c:1:
/tmp/tmpxft_0000ff6b_00000000-2_nccl.fatbin.c:30:14: error: ‘__fatBinC_Wrapper_t’ does not name a type; did you mean ‘__nv_hdl
_wrapper_t’?
 static const __fatBinC_Wrapper_t __fatDeviceText __attribute__ ((aligned (8))) __attribute__ ((section (__CUDAFATBINSECTION))
)=
              ^~~~~~~~~~~~~~~~~~~
              __nv_hdl_wrapper_t
In file included from /tmp/tmpxft_0000ff6b_00000000-5_nccl.cudafe1.stub.c:8:0,
                 from tmpxft_0000ff6b_00000000-5_nccl.cudafe1.stub.c:1:
/tmp/tmpxft_0000ff6b_00000000-5_nccl.cudafe1.stub.c: In function ‘void __sti____cudaRegisterAll()’:
/tmp/tmpxft_0000ff6b_00000000-5_nccl.cudafe1.stub.c:13:44: error: ‘__fatDeviceText’ was not declared in this scope
 static void __sti____cudaRegisterAll(void){__cudaRegisterBinary(__nv_cudaEntityRegisterCallback);}
                                            ^
/tmp/tmpxft_0000ff6b_00000000-5_nccl.cudafe1.stub.c:13:44: note: suggested alternative: ‘cuDeviceGet’
CMake Error at gloo_cuda_generated_nccl.cu.o.Release.cmake:277 (message):
  Error generating file
  /fsx/users/ezyang/pytorch-tmp/build/third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/nccl/./gloo_cuda_generated_nccl.cu.o
```

I used the standard instructions at https://www.internalfb.com/intern/wiki/PyTorch/PyTorchDev/Workflow/PyTorch_environment_setup/pytorch_aws_setup/

### Versions

master

cc @malfet @seemethere @ngimel
fatal_signal_asan_no_sig_test in current master hang.
Labels: module: cpp, triaged, module: deadlock, module: testing

### 🐛 Describe the bug

Our docker build with pytorch timed out recently and we found the fatal_signal_asan_no_sig_test always hangs.
It occupies ~700% CPU on a 36-thread system, so I guess something's spinning.

```
[root@15f31169863c build]# bin/fatal_signal_asan_no_sig_test 
Running main() from ../third_party/googletest/googletest/src/gtest_main.cc
[==========] Running 7 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 7 tests from fatalSignalTest
[ RUN      ] fatalSignalTest.SIGABRT8
```

This issue on exists on CentOS 7.
Debian 11 seems fine.
Old version of pytorch is also fine.

The build is a CUDA build on non-CUDA system.

### Versions

Current master of pytorch.
rh-python38 scl on CentOS 7.


cc @jbschlosser
Improving clarity in the docs of different losses
Labels: module: docs, module: nn, triaged, module: python frontend

### 📚 The doc issue

Looking at the following losses [BCELoss](https://pytorch.org/docs/stable/generated/torch.nn.functional.binary_cross_entropy.html#torch.nn.functional.binary_cross_entropy) vs [CrossEntropyLoss](https://pytorch.org/docs/stable/generated/torch.nn.functional.cross_entropy.html#torch.nn.functional.cross_entropy) it seems that there's not clear indication of the type of data expected for `input` and `target`. 

We see that for other variables it clearly indicates if its `int`, `bool`, `float`, etc, but not for `input` and `target`.

Also we see this discrepancy where `BCELoss` expects the `target` to be of type `float` instead `CrossEntropyLoss` expects it to be `int`.

Why `BCELoss` doesn't accept `target` of type `int` and expects `float`?

Another thing that really would allow more clarity is to have links from the doc directly to the source code of the forward and backward calls for each loss.

I know that you might say but there are these links, for instance like the following `SOURCE`:


<img width="856" alt="image" src="https://user-images.githubusercontent.com/2902390/171992728-e4835a16-4075-4a09-98cd-dd8b5166697d.png">


But following the `SOURCE` link brings us to the following definition:

<img width="352" alt="image" src="https://user-images.githubusercontent.com/2902390/171992840-177cf94f-4ef8-4629-a6a4-e78a9681760c.png">


With the following dispatch code:

<img width="600" alt="image" src="https://user-images.githubusercontent.com/2902390/171992880-9fa7b750-1940-4f71-94a7-a73f758546b8.png">


As we can see to find the actual implementation we'll have to go and find the definition of `torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)`

This prevents folks from directly retrieving relevant pieces of information on how different parts work directly from the docs. One would have to spend a would chunk of time searching through the github code base until they reach what they are looking for.

### Suggest a potential alternative/fix

1. The types of data inputs should be clearly indicated for all arguments and there should be some consistency in terms of the types for the first two inputs `input` and `target`.

2. Improve navigation from docs to source code, linking directly each method to its actual forward/backward pass instead to some dispatch function.


cc @svekars @holly1238 @albanD @mruberry @jbschlosser @walterddr @kshitij12345
Remove const from function return type if returning const value
Labels: module: cpp, triaged

### 🚀 The feature, motivation and pitch

It is better to avoid the `const` qualifier on function return type if the function returns by value. The `const` qualifier may prevent move semantics if the type is movable. https://quuxplusone.github.io/blog/2022/01/23/dont-const-all-the-things/#return-types-never-const

From [Scott Meyer](https://www.aristeia.com/BookErrata/ec++3e-errata.html):
> Declaring by-value function return values const will prevent their being bound to [rvalue references in C++0x](http://www.artima.com/cppsource/rvalue.html). Because rvalue references are designed to help improve the efficiency of C++ code, it's important to take into account the interaction of const return values and the initialization of rvalue references when specifying function signatures.

### Alternatives

Leave it as it is if the `const` qualifiers are there for some reasons.

### Additional context

I'd like to merge [this commit](https://github.com/nglee/pytorch/commit/10bdb79a0a4c1b0ae3e3dc1ba61b5929982994b4).

cc @jbschlosser
[Profiler] Capture more information about inputs
Labels: triaged, oncall: profiler

### 🚀 The feature, motivation and pitch

One of the most important options in profiler is `record_shapes`. It records the shape and dtype of inputs. This information is essential to for analyzing traces as without it there is no way to asses if an op has reasonable performance. However there are several things that we do not collect:
1) The identities of Tensors. Without this it is not possible to do data flow analysis.
2) The strides and layout of Tensors. Highly strided Tensors suffer from pathalogical memory/cache behavior and often kernes are not well optimized for these cased. Without strides two ops may be identical to the profiler but have very different runtime characteristics.
3) Scalar values. We capture that an input is a Scalar, but not the value itself. These values often parameterize computation (`dim`, `groups`, etc.) so proper analysis requires these values. (And they're scalars, so they should be cheap to store.)
4) More exotic types like `TensorList` or `NestedTensor` are important performance abstractions, but not handled by the profiler.

For (1), we can access the `TensorImpl*` and `StorageImpl*` when we encounter a Tensor. For (2), we already have an efficient packed sizes and strides representation that we use inside TensorImpl so it's mostly a matter of interfacing that with the profiler. (3) should be quite straightforward.

The input capture machinery in profiler was designed for extensibility (https://github.com/pytorch/pytorch/blob/master/torch/csrc/profiler/collection.h#L172-L202) so hopefully it will just be a matter of working through each case. And of course if you think of other things that might be useful feel free to add them as well.

### Alternatives

_No response_

### Additional context

_No response_

cc @ilia-cher @robieta @chaekit @gdankel @bitfort @ngimel @nbcsm @guotuofeng @guyang3532 @gaoteng-git
[RecordFunction] Hold a durable schema reference
Labels: triaged, oncall: profiler

### 🚀 The feature, motivation and pitch

When an op is called through the dispatcher we hold a reference to the schema in the RecordFunction object for the duration of the call: https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/record_function.h#L372 We would like to keep the schema until post processing and expose it to users as that would enable certain analyses like FLOP counting. However it is prohibitively expensive to copy the schema while profiling. We should be able to guarantee the lifetime of the FunctionSchema in the Dispatcher and then just store a pointer in RecordFunction, however work needs to be done to ensure that all the machinery is safe.

### Alternatives

_No response_

### Additional context

_No response_

cc @ilia-cher @robieta @chaekit @gdankel @bitfort @ngimel @nbcsm @guotuofeng @guyang3532 @gaoteng-git
MPS: Adding int64 tensor does not work on AMD GPU
Labels: triaged, module: correctness (silent), module: mps

### 🐛 Describe the bug

Consider following simple example:
```
python3 -c "import torch;x=torch.arange(6); print(x+(1<<32), x.to('mps')+(1<<32))"
```

It yields correct results on my M1 MacMini:
```
% python3 -c "import torch;x=torch.arange(6); print(x+(1<<32), x.to('mps')+(1<<32))"
tensor([4294967296, 4294967297, 4294967298, 4294967299, 4294967300, 4294967301]) tensor([4294967296, 4294967297, 4294967298, 4294967299, 4294967300, 4294967301],
       device='mps:0')
```
But fails on x86 MacBook Pro:
```
 % python3 -c "import torch;x=torch.arange(6); print(x+(1<<32), x.to('mps')+(1<<32))"
tensor([4294967296, 4294967297, 4294967298, 4294967299, 4294967300, 4294967301]) tensor([0, 1, 2, 3, 4, 5], device='mps:0')
```

### Versions

Nightly

cc @kulinseth @albanD