gopikrsmscs/torch-issues huggingface dataset:
Incorrect documentation in ``gumble_softmax`` function.
Labels: module: docs, module: nn, triaged

### 📚 The doc issue

[Gumbel-Softmax documentation](https://pytorch.org/docs/stable/generated/torch.nn.functional.gumbel_softmax.html) states that the ``logits`` argument should be unnormalized. However, this is not how the formula is defined in the [original article](https://arxiv.org/abs/1611.01144), but apparently it's how it is coded in the accompanying notebook. The issue was referenced in this [PyTorch forum post](https://discuss.pytorch.org/t/gumbel-softmax-function-logits/22797).

The problem is that unnormalized log-probabilities may be to large w.r.t. the values introduced by the random samples from the Gumbel distribution, which defeats the purpose of the layer. This in turn leads to poor training performance.

### Suggest a potential alternative/fix

Two possible fixes: 
1. Change the documentation to state that the ``logits`` should be normalized.
2. Normalize the logits inside the function using ``log_softmax``.

cc @svekars @holly1238 @albanD @mruberry @jbschlosser @walterddr @kshitij12345
Building from source results in broken __version__
Labels: oncall: binaries, module: build, triaged

### 🐛 Describe the bug

I'm doing 
```bash
git clone --recursive https://github.com/pytorch/pytorch
cd pytorch
git submodule update --init --recursive --jobs 0
# I'm not trying to build main, just latest release
git checkout tags/v1.11.0 -b v1.11.0-branch
# I've previously installed all the dependencies
python setup.py install --prefix ~/.local
```
and I'm seeing
```
Building wheel torch-1.11.0a0+gitbc2c6ed
-- Building version 1.11.0a0+gitbc2c6ed
```
so even though I'm building v1.11.0, the `__version__` is all screwed up.

I did this for v1.10.0 and I'm trying again with v1.11.0 because maybe I did something wrong?

This becomes a pretty big problem when I need to install packages that require torch like sentence_transformers. The `torch>=1.6.0` dependency of sentence_transformers sees `1.10.0a0+git71f889c` and doesn't know what the heck it is, so it starts downloading torch, like I didn't go to all the trouble of installing CUDNN and building the package from source lol
```
Collecting torch>=1.6.0
  Downloading torch-1.11.0-cp38-cp38-manylinux1_x86_64.whl (750.6 MB)
     ╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.9/750.6 MB 5.4 MB/s eta 0:02:16
ERROR: Operation cancelled by user
```
I've also tried to checkout the tag/<release> without create a new branch, just in detached HEAD state. Still gives the <version>a0+git934ac58 tag when building.

Really prefer not to pass `--no-deps` when installing packages that require torch then `pip show` so I can install the dependencies by hand. What am I doing wrong?

### Versions

Collecting environment information...
PyTorch version: 1.10.0a0+git71f889c
Is debug build: False
CUDA used to build PyTorch: 11.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.22.2
Libc version: glibc-2.31

Python version: 3.8.10 (default, Mar 15 2022, 12:22:08)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.13.0-40-generic-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: 11.4.152
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3070 Laptop GPU
Nvidia driver version: 470.82.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.761
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.3
[pip3] torch==1.10.0a0+git71f889c
[pip3] torch-geometric==2.0.3
[pip3] torch-scatter==2.0.9
[pip3] torch-sparse==0.6.12
[conda] Could not collect


cc @ezyang @seemethere @malfet
Stop calling sizes/numel/dim/is_contiguous on undefined tensors
Labels: module: internals, triaged

### 🐛 Describe the bug

In https://github.com/pytorch/pytorch/pull/77036 I accidentally made these error and it caused a large number of test failures

https://gist.github.com/ezyang/d7eb3b8440268bd8bac8df2f66eaa39b

but they shouldn't be called in the first place

### Versions

master

cc @ezyang @bhosmer @smessmer @ljk53 @bdhirsh
torch.stack test_conj_view and test_neg_view are failing after 77043
Labels: triaged, module: complex, module: viewing and reshaping, module: primTorch

See https://github.com/pytorch/pytorch/pull/77043, which extended the sample inputs for stack and added xfails for these two tests.

cc @ezyang @anjali411 @dylanbespalko @mruberry @Lezcano @nikitaved @ngimel
Where is fx2trt fx to tensorrt tool?
Labels: triaged, module: fx

### 📚 The doc issue

I found there are some PR:

https://github.com/jerryzh168/pytorch/tree/fb09fd4ab4ba618db148f9dfc035be589efb9355/torch/fx/experimental/fx2trt

which persist of fx2trt tool, where does it goes in main stream pytorch code?

### Suggest a potential alternative/fix

_No response_
[bug] `NATIVE` and `OMP` `parallel_for` implementations are inconsistent.
Labels: triaged, module: openmp, module: multithreading

### 🐛 Describe the bug

https://github.com/pytorch/pytorch/pull/72710 and parallel CIs uncovered that.
Simply build with `ATEN_THREADING=NATIVE python setup.py develop` and run `pytest -sv test/test_sparse.py -k index_select_par`.
Error messages could be inspected here: https://github.com/pytorch/pytorch/runs/6324324431?check_suite_focus=true.
These tests pass when built with `ATEN_THREADING=OMP`.

I was not able to test `TBB` as it is currently impossible to compile due to https://github.com/pytorch/pytorch/issues/76953.

The issue is in how backends decide on chunk size and the number of threads. I will submit a fix for that soon.

### Versions

Current master.
DISABLED test_comprehensive_linalg_ldl_factor_ex_cuda (__main__.TestDecompCUDA)
Labels: triaged, module: linear algebra, skipped

Platforms: linux

Same as #76961, but disabling test_comprehensive_linalg_ldl_factor_ex_cuda_complex128 and test_comprehensive_linalg_ldl_factor_ex_cuda_complex64

cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano
When using Rsqrt, the output of the 1/x process is very likely to have nan/inf
Labels: needs reproduction, triaged

### 🐛 Describe the bug

When using Rsqrt, the output of the 1/x process is very likely to have nan/inf. In some cases, the operator will not show output-nan/inf, but will show output-nan/inf

**For example：**
structure
```
[{'name': 'Slice', 'params': {'tensor_space': 4, 'begin': [0, 9, 1, 1], 'size': [50, 7, 15, 1]}}, {'name': 'ReduceMax', 'params': {'keep_dims': True, 'dim': 1, 'tensor_space': 4}}, {'name': 'Rsqrt', 'params': {}}, {'name': 'Flatten'}, {'name': 'Dense', 'params': {'in_features': 15, 'out_features': 200}}, {'name': 'Softmax'}]

[{'name': 'ReduceMean', 'params': {'keep_dims': True, 'dim': 3, 'tensor_space': 4}}, {'name': 'ReduceMean', 'params': {'keep_dims': False, 'dim': 1, 'tensor_space': 4}}, {'name': 'Flatten', 'params': {}}, {'name': 'Rsqrt', 'params': {}}, {'name': 'Dense', 'params': {'in_features': 16, 'out_features': 200}}, {'name': 'Softmax'}]

[{'name': 'ReduceSum', 'params': {'keep_dims': True, 'dim': 1, 'tensor_space': 4}}, {'name': 'Rsqrt', 'params': {}}, {'name': 'Flatten'}, {'name': 'Dense', 'params': {'in_features': 48, 'out_features': 200}}, {'name': 'Softmax'}]
```

### Versions

Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-74-generic x86_64)
Pytorch1.8
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
NVIDIA-SMI 460.80 Driver Version: 460.80 CUDA Version: 11.2
When using Lambda, the output of the 1/x process is very likely to have nan/inf
Labels: needs reproduction, triaged

### 🐛 Describe the bug

When using Lambda, the output of the 1/x process is very likely to have nan/inf.  In some cases, the operator will not show output-nan/inf, but will show output-nan/inf. Scenarios, where this happens with lambda, are generally: when the lambda expression carried in the lambda operator has certain risks (the operator does not impose certain constraints on its expression).

**For example：**
structure
`[{'name': 'ReduceMax', 'params': {'keep_dims': True, 'dim': 1, 'tensor_space': 4}}, {'name': 'Upsample2d', 'params': {'scale_factor': 2, 'mode': 'nearest'}}, {'name': 'Lambda', 'params': {'output_shape': [2, 32, 3], 'function': 'lambda x: 1 / x**0.5 + 1.0'}}, {'name': 'Flatten'}, {'name': 'Dense', 'params': {'in_features': 192, 'out_features': 200}}, {'name': 'Softmax'}]`

### Versions

Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-74-generic x86_64)
Pytorch1.8
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
NVIDIA-SMI 460.80 Driver Version: 460.80 CUDA Version: 11.2
GlobalAvgPool2d causes the inconsistency of output between frameworks
Labels: needs reproduction, triaged

### 🐛 Describe the bug

Operators with GlobalAvgPool2d will cause inconsistency between frameworks. And for this inconsistency, only changing the input may cause its nature to change.
（Inconsistency: When testing the three frameworks, the inconsistency here refers to comparing the output of the GlobalAvgPool2d layer with the other two frameworks and computing their absolute values.）

**structure：**
`[{'name': 'MaxPool2d', 'params': {'pool_size': [3, 2], 'stride': [4, 3], 'padding': 'same'}}, {'name': 'GlobalMaxPool2d', 'params': {}}, {'name': 'Dense', 'params': {'in_features': 1, 'out_features': 10}}, {'name': 'Softmax'}]`

### Versions

Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-74-generic x86_64)
Pytorch1.8
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
NVIDIA-SMI 460.80 Driver Version: 460.80 CUDA Version: 11.2
GaussianNoise causes the inconsistency of output between frameworks
Labels: needs reproduction, triaged

### 🐛 Describe the bug

Operators with GaussianNoise will cause inconsistency between frameworks. And for this inconsistency, only changing the input may cause its nature to change.
（Inconsistency: When testing the three frameworks, the inconsistency here refers to comparing the output of the GaussianNoise layer with the other two frameworks and computing their absolute values.）

**structure：**
```
[{'name': 'GaussianNoise', 'params': {'stddev': 0.0050545493115078935}}, {'name': 'Flatten'}, {'name': 'Dense', 'params': {'in_features': 784, 'out_features': 10}}, {'name': 'Softmax'}]

[{'name': 'StridedSlice', 'params': {'tensor_space': 4, 'begin': [0, 1, 1, 0], 'end': [25, 22, 10, 1], 'stride': [1, 1, 1, 1]}}, {'name': 'GaussianNoise', 'params': {'stddev': 0.0031597719267489033}}, {'name': 'Flatten'}, {'name': 'Dense', 'params': {'in_features': 189, 'out_features': 10}}, {'name': 'Softmax'}]
```

### Versions

Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-74-generic x86_64)
Pytorch1.8
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
NVIDIA-SMI 460.80 Driver Version: 460.80 CUDA Version: 11.2
ReduceSum causes the inconsistency of output between frameworks
Labels: needs reproduction, triaged

### 🐛 Describe the bug

Operators with ReduceSum will cause inconsistency between frameworks. And for this inconsistency, only changing the input may cause its nature to change.
（Inconsistency: When testing the three frameworks, the inconsistency here refers to comparing the output of the ReduceSum layer with the other two frameworks and computing their absolute values.）

**structure：**
```
[{'name': 'Transpose', 'params': {'output_shape': [0, 2, 3, 1], 'tensor_space': 4}}, {'name': 'ReduceSum', 'params': {'keep_dims': False, 'dim': 1, 'tensor_space': 4}}, {'name': 'Flatten'}, {'name': 'Dense', 'params': {'in_features': 28, 'out_features': 10}}, {'name': 'Softmax'}]

[{'name': 'ReduceSum', 'params': {'keep_dims': True, 'dim': 1, 'tensor_space': 4}}, {'name': 'Flatten'}, {'name': 'Dense', 'params': {'in_features': 28, 'out_features': 10}}, {'name': 'Softmax'}]

[{'name': 'LeakyReLU', 'params': {'alpha': 0.08195527217568771}}, {'name': 'Transpose', 'params': {'output_shape': [0, 3, 1, 2], 'tensor_space': 4}}, {'name': 'ReduceSum', 'params': {'keep_dims': True, 'dim': 2, 'tensor_space': 4}}, {'name': 'Flatten'}, {'name': 'Dense', 'params': {'in_features': 28, 'out_features': 10}}, {'name': 'Softmax'}]
```


### Versions

Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-74-generic x86_64)
Pytorch1.8
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
NVIDIA-SMI 460.80 Driver Version: 460.80 CUDA Version: 11.2
Private API for accessing all "internal" attributes on Tensors
Labels: module: internals, triaged

### 🐛 Describe the bug

c10::TensorImpl has a bunch of internal properties which are not directly accessible from Python. This can make debugging certain types of problems more difficult; e.g., you suspect the dispatch key set isn't set correctly but you can't just print it out directly. There should be a private, unstable API for querying this information (I don't think we should allow mutating the info) that can be used to assist in debugging.

### Versions

master

cc @ezyang @bhosmer @smessmer @ljk53 @bdhirsh
NVFuser opinfos - check for CudaFusionGroup in the graph
Labels: triaged, module: nvfuser

### 🚀 The feature, motivation and pitch

If an NVFuser opinfo fails but the JIT graph doesn't contain a CudaFusionGroup node, then we should append this information to the error, since it indicate that the problem is likely with JIT, not with NVFuser. This would make debugging these types of errors faster.

### Alternatives

_No response_

### Additional context

_No response_
[RFC] Upstream current implementation of ssd_offload from fairscale FSDP to Torch Distributed FSDP
Labels: oncall: distributed, module: fsdp

### 🚀 The feature, motivation and pitch

# Goal
Upstream the currently experimental ssd_offload feature from Fairscale FSDP to torch distributed FSDP.

ssd_offload is a capability in similar FSDP-like systems (for example zero-infinity by Microsoft)

# API changes:
Logically from the outside ssd_offload looks very similar to cpu_offload. So merging the options probably makes sense (even though in the implementation they will look quite different.

fsdp constructor: cpu_offload argument -> offload_config
torch.distributed.fsdp.CPUOffload -> ... OffloadConfig

```
class OffloadConfig:
    offload_type: Optional(str) = None # "cpu" or "ssd" only supported for now, maybe make into an enum
    offload_directory: Optional(str) = None # directory where tensors offloaded to disk reside (only valid with offload_type="ssd". In the future this will probably be extended to support some sort of wildcards (e.g. a single machine has N GPUs and N SSDs, you may want to have a dedicated per GPU)
    offload_params: bool = False
```

# Parts of Implementation
 1) Efficient Chunked Disk I/O functions
 2) Tensor Subclass implementation that exposes Tensors that can migrate back and forth between CPU and disk (SsdTensorHandle and SsdParameter). This also includes some optimizations for pickling (torch.save-ing) SsdTensorHandles (and its descendants) allowing them to serialize to disk and back w/out having multiple entire copies of the tensor in memory at a time.
 3) Implementation of Flattened Parameter (SsdFlatParameter/SsdFlatParameterView) and ancillary code to support them. Current implementation uses non-chainable derivative of pytorch parameterization I've coined propertization (turning an nn.Module attribute into a property -- necessary for updating views from the flattened tensor due to .data modification) I'm also looking into another possible alternative that would utilize a transparently wrapped SsdFlatParameterView object (similar to wrapt, but much simpler).

### Alternatives

_No response_

### Additional context

# Future Works/Desires
* Creating Asynchronous disk I/O calls that can be called by an prefetcher/offloader to pipeline disk accesses.
* Simplifying SsdTensorHandle by removing .data overriding within FSDP
* Benchmarking + Profiling both time and memory usage of ssd_offloaded tensors when used in realistic models.
   * Optimizations to improve performance. 

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang
Avoid Self-loops on Module Creation
Labels: module: nn, triaged

### 🚀 The feature, motivation and pitch

I have two models `Model1` and `Model2`.
`Model2` will be a sub-module of `Model1`.

I need to use some parameters and methods of `Model1` in `Model2`, thus I want to link `Model2` to its father `Model1`.

```python
import torch

class Model1(torch.nn.Module):
    def __init__(self):
        torch.nn.Module.__init__(self)
        self.submodel = Model2()
        self.submodel.link(self)

class Model2(torch.nn.Module):
    def link(self, model):
        self.supmodel = model

model1 = Model1()
model1.eval()
```

Writing in the above way will cause self-loop, and raise maximum recursion error.

I know that I can write a third module containing both models to avoid such error, but I think directly support above implementation makes creation easier.
Is it possible to support such convention?

### Alternatives

_No response_

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345
Pytorch return TCPStore( RuntimeError: Connection reset by peer) 
Labels: oncall: distributed, module: c10d

I amtrying to run Cosmic Tagger pytorch benchmark. It runs file up to 256 nodes(1024 ranks). However, when I try to run on higher number of nodes 384 nodes(1536 ranks) it runs fine occasionally. Most of the time it fails

## Issue description

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
```
Error executing job with overrides: ['data.data_directory=/lus/grand/projects/Polaris/CosmicTagger/data/', 'run.minibatch_size=1536', 'run.id=at-2022-05-05_384nodes', 'run.iterations=10', 'data.downsample=0']
Traceback (most recent call last):
  File "/home/ranand/bare_metal_Cosmic_tagger/CosmicTagger/bin/exec.py", line 254, in main
    s = exec(cfg)
  File "/home/ranand/bare_metal_Cosmic_tagger/CosmicTagger/bin/exec.py", line 47, in __init__
    self.train()
  File "/home/ranand/bare_metal_Cosmic_tagger/CosmicTagger/bin/exec.py", line 100, in train
    self.make_trainer()
  File "/home/ranand/bare_metal_Cosmic_tagger/CosmicTagger/bin/exec.py", line 181, in make_trainer
    self.trainer = distributed_trainer.distributed_trainer(self.args)
  File "/home/ranand/bare_metal_Cosmic_tagger/CosmicTagger/src/utils/torch/distributed_trainer.py", line 139, in __init__
    torch.distributed.init_process_group(
  File "//home/ranand/bare_metal_Cosmic_tagger/pytorch/torch/distributed/distributed_c10d.py", line 595, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
  File "//home/ranand/bare_metal_Cosmic_tagger/pytorch/torch/distributed/rendezvous.py", line 259, in _env_rendezvous_handler
    store = _create_c10d_store(master_addr, master_port, rank, world_size, timeout)
  File "//home/ranand/bare_metal_Cosmic_tagger/pytorch/torch/distributed/rendezvous.py", line 191, in _create_c10d_store
    return TCPStore(
RuntimeError: Connection reset by peer
```


Software installed
Cudatoolkit/11.4
gcc-11.2
pytorch version 1.12.0

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang
torch.nn.functional.linear sometimes incorrectly accepts arguments of the different type
Labels: module: nn, triaged, module: type promotion

```
In [14]: a=torch.randn(10,10,10,device="cuda", dtype=torch.half)

In [15]: w=torch.randn(10,10, device="cuda", dtype=torch.half)

In [16]: b=torch.randn(10,10, device="cuda")

In [17]: torch.nn.functional.linear(a.transpose(1,2), w, b).dtype
Out[17]: torch.float16

In [18]: torch.nn.functional.linear(w, b).dtype
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-18-23a27726c7de> in <module>
----> 1 torch.nn.functional.linear(w, b).dtype

RuntimeError: expected scalar type Half but found Float
```
We should disallow this case always. 

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @nairbv
Multiprocessing DataLoader hangs on exception inside iterator when using a simple queue and a producer thread
Labels: module: multiprocessing, module: dataloader, triaged

### 🐛 Describe the bug

I'm using a thread to read data from a pipe which can only be accessed by a single thread (hence the quirky setup). So the thread reads data continuously from the stream, and inserts the data frames into a multiprocessing SimpleQueue as they become available.

Then I have an `IterableDataset` that has access to this queue, and tries to read the data frame and then do some "heavy" processing on the data. I use a multiprocessing `DataLoader`.

However if something happens to raise an exception inside an dataset instance during the dataloader iteratator iteration the **python main process hangs after printing the exception rather than exiting as I would expect**. I think this is a bug, unless you can spot to multiprocessing mistake in my code example below.

I stumbled upon this ticket but I don't know it it's related: https://github.com/pytorch/pytorch/issues/48666

### Minimal example
```python
import threading
import time

import torch
import torch.multiprocessing as mp
from torch.utils.data import IterableDataset, DataLoader

class ProducerThread(threading.Thread):
    def __init__(self):
        super().__init__()
        self.queue = mp.SimpleQueue()

    def run(self):
        while True:
            data = '...'
            time.sleep(0.01)
            self.queue.put(data)


class Dataset(IterableDataset):
    def __init__(self, queue):
        self.queue = queue

    def __iter__(self):
        while True:
            data = self.queue.get()
            data = self.process(data)

            yield data

    def process(self, data):
        raise RuntimeError()


if __name__ == '__main__':
    batch_size = 4
    num_workers = 3
    producer = ProducerThread()
    dataset = Dataset(producer.queue)
    dataloader = DataLoader(dataset, batch_size, num_workers=num_workers)

    producer.start()
    for batch in dataloader:
        pass
```

### Versions

Collecting environment information...
PyTorch version: 1.10.2
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.2.1 (arm64)
GCC version: Could not collect
Clang version: 13.0.0 (clang-1300.0.29.30)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.6 (default, Feb 15 2022, 14:18:12)  [Clang 13.0.0 (clang-1300.0.27.3)] (64-bit runtime)
Python platform: macOS-12.2.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] torch==1.10.2
[pip3] torchvision==0.11.3
[conda] Could not collect

cc @VitalyFedyunin @SsnL @ejguan @NivekT
PrimTorch binary refs do not handle CUDA + CPU scalar tensors correctly
Labels: triaged, module: primTorch

### 🐛 Describe the bug

```
import torch
import torch._refs as ref

print(torch.add(torch.randn(2, device='cuda'), torch.randn(())))
print(ref.add(torch.randn(2, device='cuda'), torch.randn(())))
```

gives

```
tensor([0.7217, 0.5864], device='cuda:0')
Traceback (most recent call last):
  File "a.py", line 5, in <module>
    print(ref.add(torch.randn(2, device='cuda'), torch.randn(())))
  File "/private/home/ezyang/pytorch-tmp/torch/_refs/__init__.py", line 716, in add
    result = prims.add(a, b)
  File "/private/home/ezyang/pytorch-tmp/torch/_prims/__init__.py", line 173, in _prim
    meta(*args, **kwargs)
  File "/private/home/ezyang/pytorch-tmp/torch/_prims/__init__.py", line 202, in _elementwise_meta
    utils.check_same_device(*args, allow_cpu_scalar_tensors=True)
  File "/private/home/ezyang/pytorch-tmp/torch/_prims/utils.py", line 314, in check_same_device
    raise RuntimeError(msg)
RuntimeError: Tensor on device cpu is not on the expected device cuda:0!
```

The CPU scalar tensor behavior in stock ATen is intentional and we need to replicate it.

### Versions

master

cc @ezyang @mruberry @ngimel
Object-base collectives create tensors at unexpected devices
Labels: high priority, oncall: distributed, better-engineering, module: c10d

### 🐛 Describe the bug

torch.distributed.gather_object (and similar object based collectives) rely on default Tensor pickling behavior of deserializing tensors to their original devices.

I ran into this issue when using ShardedTensor::gather as it leads to all processes occupying all GPUs.

This causes deadlocks on test/distributed/_shard/checkpoint/test_file_system_checkpoint.py -- test_load_rowwise_to_colwise and test_load_with_different_shard_plan.

Those deadlocks only happens on the VMs used by PyTorch CI (M60).

This behavior is quite surprising as the docs sort of imply that gathered tensors would all land `torch.cuda.current_device()`.


The following script will print this on a 4 GPU host:
`[0] :: cuda:0 cuda:1 cuda:2 cuda:3`


```python
import random
import torch.multiprocessing as mp

import torch
import torch.distributed as dist
import traceback

def run(rank):
    dst = 0
    obj = { "x": torch.rand(10).cuda(rank) }

    gather_list = [None] * dist.get_world_size()
    dist.gather_object(
        obj,
        gather_list if rank == dst else None,
        dst=dst)

    if rank == dst:
        devices = " ".join(str(l["x"].device) for l in gather_list)
        print(f"[{rank}] :: {devices}")

def worker(rank, init_method, world_size):
    init_comms(init_method, rank, world_size)
    try:
        run(rank)
    except BaseException as e:
        print(f"[{rank}] :: raised {e}")
        traceback.print_last()
    destroy_comms()

def init_pg(init_method, rank, world_size):
    torch.distributed.init_process_group(
        backend="nccl",
        world_size=world_size,
        rank=rank,
        init_method = init_method
    )
    torch.cuda.set_device(rank)

def init_comms(file_name, rank, world_size):
    init_pg(file_name, rank, world_size=world_size)

def destroy_comms():
    dist.barrier()
    dist.destroy_process_group()

if __name__ == "__main__":
    print("main")
    port = random.randint(10000, 20000)
    init_method = f"tcp://localhost:{port}"
    world_size = torch.cuda.device_count()

    mp.spawn(
        fn=worker,
        args=(init_method,world_size),
        nprocs=world_size,
        join=True,
    )
```

### Versions

main

cc @ezyang @gchanan @zou3519 @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang
Feature requests for optimizer overlapping
Labels: high priority, triage review, oncall: distributed, module: ddp, module: fsdp

### 🚀 The feature, motivation and pitch

After chatting with users who are interested in using optimizer overlap with DDP (and eventually FSDP) have a couple of feature requests:

1. Support an argument `set_grads_to_None`, as users would like to simply pass in this flag which will make their calls to optimizer.step() functionally a noop, instead of manually having to either to this themselves on their gradients or removing their optimizer step, and 
2. Add a getter to expose the fully initialized fused optimizer on the DDP / FSDP module. This is required for things like learning late schedulers that take the optimizer as a ctor argument, as well as for use with things such as torchrec's `KeyedOptimizer` which wraps a vanilla torch.optim.Optimizer, as well as additional usability for things such as optimizer checkpointing. 
3. Compatibility with LR schedulers.

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @gchanan @zou3519 @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang
Inconsistent results between Pow and Float Pow with their numpy references for complex types
Labels: triaged, module: complex, module: NaNs and Infs

### 🐛 Describe the bug

Added numpy references to Pow and Float_Pow Opinfos. For `torch.complex64`, `torch.complex128` test_reference_numerics - (small, large, extremal) were skipped do to errors of the form: `Greatest absolute difference: nan at index`. 

Followed from working on these issues: #76483, #74279
### Versions

Not version specific 

cc @ezyang @anjali411 @dylanbespalko @mruberry @Lezcano @nikitaved
Unify torch.ops argument parsing code with PythonArgParser
Labels: triaged, module: __torch_dispatch__, module: python frontend

### 🐛 Describe the bug

They are implementations of the same thing; we should use the same codepath for each. Relevant for https://github.com/pytorch/pytorch/pull/76835 where scalar arguments were not accepted into wrapped tensors.

### Versions

master

cc @Chillee @ezyang @zou3519 @albanD @samdow
Windows CUDA TTS tracking task
Labels: module: ci, triaged

Our goal for CI TTS started at 3hrs this half and was not far from there at the start of 2022. However, now we are noticing some pretty large regressions where our Windows jobs have a TTS of 4hrs (per shard!). At peak times, the TTS goes higher due to stressed queuing times. 

Actionable steps:
- Increase our pool of available Windows runners --> if feasible, this would be the ideal solution.
- Aggressively shard --> this is fine but there is an overhead we'd be eating @clee2000 
- Remove Windows CUDA job from PRs --> this would not be ideal as Windows CUDA build signal is valuable to have on pull requests. We used to have smoke tests to not test so much on Windows GPU machines, but those were not worth maintaining as the process of maintaining them was mostly manual.
- Be able to delineate TTS on the job level so we can confirm results in our metrics (compared to spot checks)

cc @seemethere @malfet @pytorch/pytorch-dev-infra
TYPEIGNORE lint run locally disagrees with CI
Labels: module: lint, triaged, module: macos

### 🐛 Describe the bug

https://github.com/pytorch/pytorch/pull/76735 is failing TYPEIGNORE lint https://github.com/pytorch/pytorch/runs/6291285337?check_suite_focus=true but when I check out the package and run it locally the TYPEIGNORE lint passes. This is on OSX. cc @malfet @albanD @suo

### Versions

master
There is a bug with latest stable torch version and the following Nightly versions related to `optimize_for_mobile`
Labels: oncall: mobile

### 🐛 Describe the bug

There is a bug in `optimize_for_mobile` with latest torch version and the following nightly versions. Please look at this [discussion](https://discuss.pytorch.org/t/torch-utils-mobile-optimizer-optimize-for-mobile-is-resulting-different-output-than-torch-model-and-jit-model/150183) for full details.

**The short problem description:** the model is giving different result after `optimize_for_mobile`

The problem started  with stable torch version 1.11.0 and the following Nightly versions. But it is working correctly with previous torch versions. 

Here is the minimal code snippet to reproduce the error:

```
import torch
import torchvision
from torch.utils.mobile_optimizer import optimize_for_mobile

model = torchvision.models.resnet50(pretrained=True)
model.eval()


out1 = model(torch.ones([1, 3, 224, 224], dtype=torch.float))
out2 = traced_script_module(torch.ones([1, 3, 224, 224], dtype=torch.float))
out3 = traced_script_module_optimized(torch.ones([1, 3, 224, 224], dtype=torch.float))
```
and results:

```
tensor([[-3.0806e-01,  7.9845e-02, -1.1900e+00, -1.4837e+00, -5.1359e-01,...]])
tensor([[-3.0806e-01,  7.9845e-02, -1.1900e+00, -1.4837e+00, -5.1359e-01,...]])
tensor([[-5.2194e+00,  1.4939e+00, -2.2703e+00, -5.4212e+00, -2.0551e+00,...]])
```

### Versions

 stable torch version 1.11.0 and the following Nightly versions.
torch.Tensor.__rdiv__ long x scalar float type promotion is incorrect
Labels: triaged, module: type promotion, module: primTorch

```
a = torch.tensor((1, 2, 3))
b = torch.tensor((1.), dtype=torch.double)
torch.Tensor.__rdiv__(a, b)
: tensor([1.0000, 0.5000, 0.3333])
```

The result should be have dtype torch.double:

```
b / a
: tensor([1.0000, 0.5000, 0.3333], dtype=torch.float64)
```

If we pursue this we should consider it in the context of https://github.com/pytorch/pytorch/issues/74616. I think type promotion works as expected in scenarios where the dunder is expected to be used:

```
1 / torch.tensor((1.), dtype=torch.double)
tensor(1., dtype=torch.float64)
```


cc @nairbv @mruberry @ezyang @ngimel
torch.add bool x bool allows integer alpha, inconsistent with other dtype type checking
Labels: triaged, module: primTorch

torch.add has an alpha kwarg which must be a scalar. Typically this kwarg must have a Python type that is weakly lower than the corresponding Python type of operation's computation dtype. For example, if adding two integer tensors alpha cannot be a float:

```
a = torch.tensor((1, 2, 3))
b = torch.tensor((4, 5, 6))
torch.add(a, b, alpha=.5)
: RuntimeError: For integral input tensors, argument alpha must not be a floating point number.
```

However for some reason we support integer alpha when adding bool tensors:

```
torch.add(torch.tensor((True, False, True, False)), torch.tensor((True, False, False, True)), alpha=1)
: tensor([ True, False,  True,  True])
```

The integer alpha works fine, as does specifying a boolean alpha:

```
torch.add(torch.tensor((True, False, True, False)), torch.tensor((True, False, False, True)), alpha=False)
: tensor([ True, False,  True, False])

torch.add(torch.tensor((True, False, True, False)), torch.tensor((True, False, False, True)), alpha=True)
tensor([ True, False,  True,  True])
```

It's just inconsistent with our type checking for torch.add. 



cc @ezyang @mruberry @ngimel
`gradcheck` for `torch.solve` may trigger INTERNAL ASSERT FAILED
Labels: needs reproduction, module: autograd, triaged

### 🐛 Describe the bug

`gradcheck` for `torch.solve` may trigger INTERNAL ASSERT FAILED randomly.

```python
import torch

input = torch.rand([0, 2, 5, 4], dtype=torch.float64, requires_grad=True)
A = torch.rand([1, 5, 5], dtype=torch.float64, requires_grad=True)

torch.autograd.gradcheck(torch.solve, (input, A))
# RuntimeError: falseINTERNAL ASSERT FAILED at "/Users/distiller/project/pytorch/aten/src/ATen/native/LinearAlgebraUtils.h":328, please report a bug to PyTorch. linalg.solve: (Batch element 0): Argument 1998362383 has illegal value. Most certainly there is a bug in the implementation calling the backend library.
```

### Versions

pytorch: 1.11.0

cc @ezyang @gchanan @zou3519 @albanD @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
`cumprod, prod` will backward fail if `dtype` argument is different than the dtype of input tensor
Labels: module: autograd, triaged, module: complex, complex_autograd

### 🐛 Describe the bug

`cumprod, prod` will backward fail if `dtype` argument is different than the dtype of input tensor

```python
import torch

def fn(input):
    dim = -1
    dtype = torch.complex128
    fn_res = torch.cumprod(input, dim, dtype=dtype)
    return fn_res
input = torch.rand([5, 5, 5], dtype=torch.float64, requires_grad=True)
fn(input).sum().backward()

# RuntimeError: Expected isFloatingType(grad.scalar_type()) || (input_is_complex == grad_is_complex) to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
```

```python
import torch
input = torch.rand([3, 3], dtype=torch.float64, requires_grad=True)
def fn(input):
    dtype = torch.complex64
    fn_res = torch.prod(input, dtype=dtype, )
    return fn_res

res = fn(input)
res.backward()
# RuntimeError: Expected isFloatingType(grad.scalar_type()) || (input_is_complex == grad_is_complex) to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
```

### Versions

pytorch: 1.11.0

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @anjali411 @dylanbespalko @mruberry
`addr, baddmm, dist, l1_loss` will backward fail when input tensors have different dtypes
Labels: module: autograd, triaged, module: complex, actionable, complex_autograd

### 🐛 Describe the bug

`addr, baddmm, dist, l1_loss` will backward fail when input tensors have different dtypes

```python
import torch
input = torch.rand([3, 2], dtype=torch.float64, requires_grad=True)
vec1 = torch.rand([3], dtype=torch.float64, requires_grad=True)
vec2 = torch.rand([2], dtype=torch.complex128, requires_grad=True)

res = torch.addr(input, vec1, vec2)
res2 = res.sum()
res2.backward()
# RuntimeError: expected scalar type ComplexDouble but found Double
```

```python
import torch
results = dict()
input = torch.rand([10, 3, 5], dtype=torch.float64, requires_grad=True)
batch1 = torch.rand([10, 3, 4], dtype=torch.complex128, requires_grad=True)
batch2 = torch.rand([10, 4, 5], dtype=torch.complex128, requires_grad=True)

res = torch.baddbmm(input, batch1, batch2)
res2 = res.sum()
res2.backward()
# RuntimeError: Expected isFloatingType(grad.scalar_type()) || (input_is_complex == grad_is_complex) to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
```

```python
import torch
input = torch.rand([5, 3], dtype=torch.complex128, requires_grad=True)
other = torch.rand([5, 3], dtype=torch.float64, requires_grad=True)

res = torch.dist(input, other)
res.sum().backward()
# RuntimeError: Expected isFloatingType(grad.scalar_type()) || (input_is_complex == grad_is_complex) to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
```

```python
import torch
input = torch.rand([3, 5], dtype=torch.complex128, requires_grad=True)
target = torch.rand([3, 5], dtype=torch.float64, requires_grad=True)

res = torch.nn.functional.l1_loss(input, target)
res.backward()
# RuntimeError: result type ComplexDouble can't be cast to the desired output type Double
```

All of them succeed to forward but fail to backward. However, some other APIs will backward successfully even if input tensors have different dtypes, for example

```python
import torch
input = torch.rand([5, 3], dtype=torch.complex128, requires_grad=True)
other = torch.rand([5, 3], dtype=torch.float64, requires_grad=True)

res = torch.nn.functional.poisson_nll_loss(input, other)
res.sum().backward()
# succeed
```

### Versions

pytorch: 1.11.0

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @anjali411 @dylanbespalko @mruberry
`gradcheck` fails for `torch.trace`
Labels: module: autograd, triaged, actionable

### 🐛 Describe the bug

`gradcheck` fails for `torch.trace` when `dim1 > dim0 + 1`

```python
import torch
input = torch.rand([5, 3], dtype=torch.float64, requires_grad=True)

torch.autograd.gradcheck(torch.trace, (input))

# torch.autograd.gradcheck.GradcheckError: Jacobian mismatch for output 0 with respect to input 0,
# numerical:tensor([[1.0000],
#         [0.0000],
#         [0.0000],
#         [0.0000],
#         [1.0000],
#         [0.0000],
#         [0.0000],
#         [0.0000],
#         [1.0000],
#         [0.0000],
#         [0.0000],
#         [0.0000],
#         [0.0000],
#         [0.0000],
#         [0.0000]], dtype=torch.float64)
# analytical:tensor([[1.],
#         [0.],
#         [0.],
#         [0.],
#         [1.],
#         [0.],
#         [0.],
#         [0.],
#         [1.],
#         [0.],
#         [0.],
#         [0.],
#         [1.],
#         [0.],
#         [0.]], dtype=torch.float64)
```

I think the `numerical:tensor` is right since `dim0 = 3` and there are 3 elements of the diagonal. Plus, when `dim1 = 4`, it will succeed. 

### Versions

pytorch: 1.11.0

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
`gradcheck` should support the comparison of NaN
Labels: module: autograd, triaged, enhancement, actionable

### 🐛 Describe the bug

`gradcheck` should support the comparison of NaN, like adding an argument that whether `NaN == NaN` in the check.

```python
import torch
input = torch.rand([4], dtype=torch.float64, requires_grad=True)
torch.autograd.gradcheck(torch.acosh, (input))
# torch.autograd.gradcheck.GradcheckError: Jacobian mismatch for output 0 with respect to input 0,
# numerical:tensor([[nan, nan, nan, nan],
#         [nan, nan, nan, nan],
#         [nan, nan, nan, nan],
#         [nan, nan, nan, nan]], dtype=torch.float64)
# analytical:tensor([[nan, nan, nan, nan],
#         [nan, nan, nan, nan],
#         [nan, nan, nan, nan],
#         [nan, nan, nan, nan]], dtype=torch.float64)
```

### Versions

pytorch: 1.11.0

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
Strange warning from `matmul(..., out=...)`
Labels: triaged, module: linear algebra

### 🐛 Describe the bug

The torch.matmul `out=` parameter seems to reshape its parameter down to two dimensions, which leads to a strange UserWarning:
```
U = torch.randn(2, 4, 3)
W = torch.randn(3, 5)
temp = torch.empty((U @ W).shape)
torch.matmul(U, W, out=temp)
UserWarning: An output with one or more elements was resized since it had shape [2, 4, 5], which does not match the required output shape [8, 5].This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at  /Users/distiller/project/pytorch/aten/src/ATen/native/Resize.cpp:24.)
  torch.matmul(U, W, out=temp)
```
It seems like `temp` tensor should clearly have the right shape [2, 4, 5] and not [8, 5] like the warning says.

### Versions

Collecting environment information...
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.3.1 (x86_64)
GCC version: Could not collect
Clang version: 13.1.6 (clang-1316.0.21.2.3)
CMake version: version 3.21.4
Libc version: N/A

Python version: 3.9.12 (main, Apr 11 2022, 01:47:47)  [Clang 13.1.6 (clang-1316.0.21.2)] (64-bit runtime)
Python platform: macOS-12.3.1-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] functorch==0.1.1
[pip3] numpy==1.19.5
[pip3] torch==1.11.0
[pip3] torchvision==0.12.0
[conda] Could not collect


cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano
`torch.addmv` backward fails
Labels: module: autograd, triaged, module: complex, actionable

### 🐛 Describe the bug

`torch.addmv` backward fails when `input` is `float64`, `mat` and `vec` are `complex128`

```python
import torch
input = torch.rand([2], dtype=torch.float64, requires_grad=True)
mat = torch.rand([2, 3], dtype=torch.complex128, requires_grad=True)
vec = torch.rand([3], dtype=torch.complex128, requires_grad=True)

res = torch.addmv(input, mat, vec)
print("addmv SUCCEED!")

res_2 = res.sum()
print("sum SUCCEED!")

res_2.backward()
# addmv SUCCEED!
# sum SUCCEED!
# RuntimeError: Expected isFloatingType(grad.scalar_type()) || (input_is_complex == grad_is_complex) to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
```

However, when `input` is `complex128`, `mat` and `vec` are `float64`, it will succeed to backward

### Versions

pytorch: 1.11.0

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @anjali411 @dylanbespalko @mruberry
Support Positional-only Arguments in JIT
Labels: oncall: jit

### 🚀 The feature, motivation and pitch

From Python 3.8, we have positional-only arguments for function definition which gives a better control of code behaviors.
However, it seems that PyTorch (1.11) JIT (`torch.jit.script`) does not support it.
For instance, if we replace the example code
```python
import torch
@torch.jit.script
def foo(x, y):
    if x.max() > y.max():
        r = x
    else:
        r = y
    return r
```
to positional-only definition
```python
import torch
@torch.jit.script
def foo(x, y, /):
    if x.max() > y.max():
        r = x
    else:
        r = y
    return r
```
It will raise error:
```
RuntimeError: 
undefined value x:
  File "/scratch-data/gao462/Coding/Ray/fretboard.py", line 4
@torch.jit.script
def foo(x, y, /):
    if x.max() > y.max():
       ~ <--- HERE
        r = x
    else:
```
Can we support this in future update?

### Alternatives

_No response_

### Additional context

_No response_
[typing] distribution.lazy_property is not typed
Labels: module: distributions, triaged

### 🐛 Describe the bug

`lazy_property` is not typed, but widely used in distribution attributes, e.g., `bernoulli_distn.probs`:
https://github.com/pytorch/pytorch/blob/8473173c36a171c9ad896804ab36049f80a99073/torch/distributions/utils.py#L95

This makes writing typed code w/ distributions ugly (e.g., having to use `cast(..)`).

Please consider making it typed, e.g., 
```py
T = TypeVar('T')


class lazy_property(Generic[T]):
    r"""
    Used as a decorator for lazy loading of class attributes. This uses a
    non-data descriptor that calls the wrapped method to compute the property on
    first call; thereafter replacing the wrapped method into an instance
    attribute.
    Derived from:
      https://github.com/pytorch/pytorch/blob/556c8a300b5b062f3429dfac46f6def372bd22fc/torch/distributions/utils.py#L92
    """

    def __init__(self, wrapped: Callable[[Any], T]):
        self.wrapped = wrapped
        functools.update_wrapper(self, wrapped)

    def __get__(self, instance: Any, obj_type: Any = None) -> T:
        if instance is None:
            return self  # typing: ignore
        value = self.wrapped(instance)
        setattr(instance, self.wrapped.__name__, value)
        return value
```

### Versions

all versions

cc @fritzo @neerajprad @alicanb @nikitaved