gopikrsmscs/torch-issues huggingface dataset:
fx: cannot find module <built-in method matmul> when using apex.amp
Labels: triaged, module: fx

### 🐛 Describe the bug

I camp up with the following issue when using `torch.fx` and [apex.amp](https://nvidia.github.io/apex/amp.html) together.
If a model has `torch.matul` operations and uses `apex.amp (opt_level=O1)` for mixed precision optimization, then fx will fail to recompile the model graph and will report the following error:
```text
File "/xxxxxxxxxxxxx/dist-packages/torch/fx/node.py", line 40, in _find_module_of_method
    raise RuntimeError(f'cannot find module for {orig_method}, {name}')
RuntimeError: cannot find module for <built-in method matmul of type object at 0x7f6c5fd9fe40>
```

A simple toy module with `torch.matmul` can reproduce the error.
```python
import torch
from torch import fx

class ToyMod(torch.nn.Module):
    def __init__(self, in_features=768, out_features=768):
        super().__init__()
        self.linear = torch.nn.Linear(in_features=in_features, out_features=out_features)

    def forward(self, X, other):
        result = torch.matmul(self.linear(input=X), other)
        return result

def test():
    from apex import amp
    mod = ToyMod().cuda()
    optimizer = torch.optim.SGD(mod.parameters(), lr=1e-3)

    # Allow Amp to perform casts as required by the opt_level
    model, optimizer = amp.initialize(mod, optimizer, opt_level="O1")
    graph : fx.Graph = fx.Tracer().trace(model)
    for node in graph.nodes:
        # do something
        pass
    graph.lint()
    return fx.GraphModule(model, graph)
```

By checking [fx/node.py](https://github.com/pytorch/pytorch/blob/v1.10.0/torch/fx/node.py), it seems `torch.matmul` is missing in `torch`. I added `print(torch.matmul)` at https://github.com/pytorch/pytorch/blob/v1.10.0/torch/fx/node.py#L37, and got 
```text
torch.matmul: <function _VariableFunctionsClass.matmul at 0x7f377820cb70>
```
The object address of matmul in `torch` has been changed. 

Any suggestions about this issue are appreciated.


### Versions

PyTorch version: 1.10.0
CUDA used to build PyTorch: 11.3

OS: Debian GNU/Linux 10 (buster) (x86_64)
GCC version: (Debian 8.3.0-6) 8.3.0

Python version: 3.7.3
Is CUDA available: True
CUDA runtime version: 11.3.109

[pip3] numpy==1.21.5
[pip3] torch==1.10.0
[pip3] torchaudio==0.10.0+cu113
[pip3] torchvision==0.11.1+cu113

apex: Version 0.1



cc @ezyang @SherlockNoMad
Ensure custom Function are correct in double backward setting
Labels: module: autograd, triaged, enhancement, actionable

### Problem statement
Today, if intermediary Tensors are saved during the forward and used during the backward, the double backward will be silently wrong because their contribution will be ignored.

A small repro is:
```python
import torch
from torch.autograd import Function, gradcheck, gradgradcheck


class MyFn(Function):
    @staticmethod
    def forward(ctx, inp):
        temp = inp * 2
        out = temp * 3
        ctx.temp = temp
        ctx.inp = inp
        return out

    @staticmethod
    def backward(ctx, gO):
        return gO * torch.full_like(gO, 3) * ctx.temp / ctx.inp


inp = torch.rand(10, dtype=torch.float64, requires_grad=True)

gradcheck(MyFn.apply, inp)
gradgradcheck(MyFn.apply, inp)
```

### Proposed solution
Note that I am not sure this is the best way to go about this, we might want to do something else in practice?

Option 1) Add a `ErrorNode` as the grad_fn for everything that gets saved on the ctx and via save_for_backward if it does not already have a grad_fn.
Since the backward runs in no grad_mode when you're not doing double backward, it will just be ignored for most users. But in the double backward case, it will be attached to the created graph. And so if the user tries to backward through that particular Tensor, we can provide a nice error message.
If this is a false positive for some users (they want to ignore that intermediary from the double backward computation), then they can just "detach()" it to recover the previous behavior.

Option 2) Add these error nodes on the fly during the backward function only if grad_mode is enabled. This avoid mutating the object when the user does `ctx.foo = foo` which is better? And only the accessed one is changed? Unfortunately, we cannot do this out of place as some users rely on the fact that `ctx.foo is foo`.

Option 3) ??



cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
Allow `torch.fx` tracing on TorchScript models
Labels: oncall: jit, module: fx

### 🚀 The feature, motivation and pitch

It would be really nice if we could use `torch.fx.symbolic_trace` on a TorchScript model. We use TorchScript extensively to save models since it's a hermetically sealed artifact that no longer requires a Python dependency. We'd like to be able to then `torch.fx.symbolic_trace` these TorchScript models so we can use the Graph FX quantization API on pre-trained models -- right now, if you try to trace a TorchScript model you get an error like:

```python
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [47], in <module>
      1 original = nn.Linear(1, 1)
      2 scripted = torch.jit.script(original)
----> 3 torch.fx.symbolic_trace(scripted)

File /opt/conda/envs/ctrldev/lib/python3.8/site-packages/torch/fx/_symbolic_trace.py:907, in symbolic_trace(root, concrete_args, enable_cpatching)
    858 """
    859 Symbolic tracing API
    860 
   (...)
    904     GraphModule: a Module created from the recorded operations from ``root``.
    905 """
    906 tracer = Tracer(enable_cpatching=enable_cpatching)
--> 907 graph = tracer.trace(root, concrete_args)
    908 name = root.__class__.__name__ if isinstance(root, torch.nn.Module) else root.__name__
    909 return GraphModule(tracer.root, graph, name)

File /opt/conda/envs/ctrldev/lib/python3.8/site-packages/torch/fx/_symbolic_trace.py:559, in Tracer.trace(self, root, concrete_args)
    557 if isinstance(root, torch.nn.Module):
    558     self.root = root
--> 559     fn = type(root).forward
    560     self.submodule_paths = {mod: name for name, mod in root.named_modules()}
    561 else:

File /opt/conda/envs/ctrldev/lib/python3.8/site-packages/torch/jit/_script.py:298, in _CachedForward.__get__(self, obj, cls)
    297 def __get__(self, obj, cls):
--> 298     return self.__getattr__("forward")

AttributeError: '_CachedForward' object has no attribute '__getattr__'
```

This would be super convenient because it would decouple the code that produced the original pre-trained model and the code used for quantization.

### Alternatives

Ultimately, what we want is to be able to quantize a model without having access to the original code that generated the model (so no `pickle`, for instance). AFAICT, TorchScript is the best way to serialize a model without needing the original Python dependency, but if there's an alternative serialization format that plays nice with `torch.fx` then I think we'd be happy to use that instead (e.g. is there a way to serialize a `torch.fx.Graph` in such a hermetically sealed way?).

### Additional context

_No response_
Many dispatch keys do not print to string correctly
Labels: triaged, module: dispatch

### 🐛 Describe the bug

I've been spending a lot of time with the "dispatch key implementation not found" error and I noticed that there are a lot of `UNKNOWN_TENSOR_TYPE_ID` in the output:

```
NotImplementedError: Could not run 'aten::_foreach_minimum.List' with arguments from the 'Meta' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::_foreach_minimum.List' is only available for these backends: [Dense, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, SparseCPU, SparseCUDA, SparseHIP, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, SparseXPU, UNKNOWN_TENSOR_TYPE_ID, SparseVE, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, NestedTensorCPU, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID, UNKNOWN_TENSOR_TYPE_ID
```

cc @bdhirsh

### Versions

master
at::real and at::imag as methods
Labels: triaged, module: complex

### 🚀 The feature, motivation and pitch

It'd make formulae easier to read, as you could read them left-to-right. E.g.
```c++
t.diagonal(0, -2, -1).real().mul_(0.5);
```
rather than
```c++
at::real(t.diagonal(0, -2, -1)).mul_(0.5);
```

I could submit a PR if people are fine with this.

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @anjali411 @dylanbespalko @mruberry @Lezcano @nikitaved
test_wishart_log_prob fails locally for me
Labels: module: distributions, module: tests, triaged

### 🐛 Describe the bug

```
======================================================================
ERROR: test_wishart_log_prob (__main__.TestDistributions)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/users/ezyang/pytorch-tmp/test/distributions/test_distributions.py", line 2309, in test_wishart_log_prob
    expected = ref_dist.logpdf(x.transpose(0, 2).numpy())
  File "/home/ezyang/local/pytorch-tmp-env/lib/python3.9/site-packages/scipy/stats/_multivariate.py", line 2258, in logpdf
    out = self._dist._logpdf(x, self.dim, self.df, self.scale,
  File "/home/ezyang/local/pytorch-tmp-env/lib/python3.9/site-packages/scipy/stats/_multivariate.py", line 1851, in _logpdf
    _, log_det_x[i] = self._cholesky_logdet(x[:, :, i])
  File "/home/ezyang/local/pytorch-tmp-env/lib/python3.9/site-packages/scipy/stats/_multivariate.py", line 2221, in _cholesky_logdet
    c_decomp = scipy.linalg.cholesky(scale, lower=True)
  File "/home/ezyang/local/pytorch-tmp-env/lib/python3.9/site-packages/scipy/linalg/_decomp_cholesky.py", line 88, in cholesky
    c, lower = _cholesky(a, lower=lower, overwrite_a=overwrite_a, clean=True,
  File "/home/ezyang/local/pytorch-tmp-env/lib/python3.9/site-packages/scipy/linalg/_decomp_cholesky.py", line 37, in _cholesky
    raise LinAlgError("%d-th leading minor of the array is not positive "
numpy.linalg.LinAlgError: 3-th leading minor of the array is not positive definite

----------------------------------------------------------------------
```

I have scipy                    1.8.0 installed

### Versions

master

cc @fritzo @neerajprad @alicanb @nikitaved @mruberry @jianyuh @pearu @walterddr @IvanYashchuk @xwang233 @Lezcano
Replace `RuntimeError` by custom exception for unsupported ONNX operators during export
Labels: module: onnx, triaged, onnx-triaged, topic: improvements

### 🐛 Describe the bug

During `torch.onnx.export()`, `RuntimeError` [is](https://github.com/pytorch/pytorch/blob/master/torch/onnx/symbolic_helper.py#L360-L387) raised when the ONNX converter need to express that a specific operator is not supported.

`RuntimeError` can also be raised due to a legitimate user (within training script) error or an implementation bug within a symbolic operator implementation. Up to today, both scenarios are mixed, and it is up to the user to distinguish whether this is a PyTorch (and report a bug) or their own (and fix it). Possibly a PyTorch bug would be filed, causing unnecessary noise to all. 

After https://github.com/pytorch/pytorch/pull/74759, when `operator_export_type=ONNX_ATEN_FALLBACK` and a `RuntimeError` is raised for whatever reason, ONNX exporter will interpret `RuntimeError` as an unsupported operator and will emit ATen operator (instead of raising a possible implementation bug). 

### Versions

All pytorch versions
Coverage test is only checking packages and not all submodules
Labels: module: docs, triaged, module: python frontend

The check for `ispkg` should be removed from this check: https://github.com/pytorch/pytorch/blob/81722f66306aef05334c27bb2a64137d8fa6493a/docs/source/conf.py#L469 The allowlist need to be updated to reflect that.



cc @brianjo @mruberry
test_license_for_wheel always fails on my local dev copy
Labels: module: tests, triaged

### 🐛 Describe the bug

```
python test/test_license.py
sE
======================================================================
ERROR: test_license_for_wheel (__main__.TestLicense)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/users/ezyang/pytorch-tmp/test/test_license.py", line 27, in test_license_for_wheel
    create_bundled('third_party', current)
  File "/data/users/ezyang/pytorch-tmp/third_party/build_bundled.py", line 41, in create_bundled
    collected = collect_license(d)
  File "/data/users/ezyang/pytorch-tmp/third_party/build_bundled.py", line 19, in collect_license
    raise ValueError('could not identify license file '
ValueError: could not identify license file for third_party/cudnn_frontend

----------------------------------------------------------------------
Ran 2 tests in 0.168s

FAILED (errors=1, skipped=1)
```

this seems to pass on master CI so I guess it's a problem with my local dev env... would be nice to make this test more robust

### Versions

master

cc @mruberry
run_test.py option to write out failed tests
Labels: module: tests, triaged, better-engineering

### 🐛 Describe the bug

Sometimes I need to make a wide ranging change to PyTorch and run the entire test suite, and I only expect a few tests scattered here and there to fail. It would be nice to get a list of failing tests, which I could then pipe into the runner, and only execute those tests to see if my fix worked.

### Versions

master

cc @mruberry
Deprecation warning from SequentialLR
Labels: high priority, triaged, module: deprecation, module: LrScheduler

### 🐛 Describe the bug

This is probably from sub-scheduler when SequentialLR calls them with epoch arg.

```
/usr/local/lib/python3.6/dist-packages/torch/optim/lr_scheduler.py:154: UserWarning: The epoch parameter in `scheduler.step()` was not necessary and is being deprecated where possible. Please use `scheduler.step()` to step the scheduler. During the deprecation, if epoch is different from None, the closed form is used instead of the new chainable form, where available. Please open an issue if you are unable to replicate your use case: https://github.com/pytorch/pytorch/issues/new/choose.
  warnings.warn(EPOCH_DEPRECATION_WARNING, UserWarning)
```

Related code:
- https://github.com/pytorch/pytorch/blob/29a45f00097664d12921c8d13e387bee4b835665/torch/optim/lr_scheduler.py#L154
- https://github.com/pytorch/pytorch/blob/29a45f00097664d12921c8d13e387bee4b835665/torch/optim/lr_scheduler.py#L632

Comment should also be fixed:
- https://github.com/pytorch/pytorch/blob/29a45f00097664d12921c8d13e387bee4b835665/torch/optim/lr_scheduler.py#L1276

For compatibility with 3rd-party sub-scheduler who's still relying on epoch arg, you may:
- Add a member switch to temporarily suppress this warnings when called directly by SequentialLR, and revert it at the end of step().
- Or whitelist pytorch's own types with warnings in SequentialLR to use non-epoch signature.
You may also need to check the type of `self` before printing that warning, to keep it quiet on derived types.

### Versions

Commit 29a45f00097664d12921c8d13e387bee4b835665

cc @ezyang @gchanan @zou3519
Allow any operation that takes a Storage to also take a contiguous Tensor instead
Labels: feature, triaged, module: python frontend

### 🐛 Describe the bug

We don't like storages, right? So we should make it possible to do stuff in Python userland without having to explicitly refer to storage. Allowing all operations on storages to instead accept contiguous tensors is a good start; e.g., `set_(..., storage_offset, size, strides)` should take a contiguous tensor.

One fly in the ointment is that even if a tensor is contiguous, it could refer to the inside of a storage (non-zero storage offset).  This should probably also be disallowed when using the API in this way, or accounted for (in the case of `set_`, you can just implicitly adjust `storage_offset` and it will work out)

### Versions

master
Eliminate uses of deprecated `FindCUDA.cmake`
Labels: module: build, module: cuda, triaged, better-engineering

`find_package(CUDA)` is the legacy way of using CMake to build CUDA code and was deprecated in CMake 3.10. #62445 transitioned over to `enable_language(CUDA)` which gives first class support for `CUDA` as a language in CMake.  However, `find_package(CUDA)` is still being used to locate the CUDA toolkit libraries. This is potentially dangerous because we have two different mechanisms searching for the CUDA installation, each with their own configuration variables. So, it would be best to stop using `FindCUDA.cmake` entirely.

The modern CMake equivalent for library targets is `find_package(CUDAToolkit)` which requires CMake 3.17, so we can't assume it's present yet. However, it might also be possible to vendor `FindCUDAToolkit.cmake` and use it anyway.

There are also dependencies in `third_party` which still use `find_package(CUDA)`
- [ ] [gloo](https://github.com/facebookincubator/gloo/blob/c22a5cfba94edf8ea4f53a174d38aa0c629d070f/cmake/Cuda.cmake#L133)
- [ ] [tensorpipe](https://github.com/pytorch/tensorpipe/blob/52791a2fd214b2a9dc5759d36725909c1daa7f2e/tensorpipe/CMakeLists.txt#L237)
- [ ] [kineto](https://github.com/pytorch/kineto/blob/b2b48c00c6e5bd8e807e2231adb229db6a1d1c22/libkineto/CMakeLists.txt#L181)

cc @malfet @seemethere @ngimel
HIPFFT_EXEC_FAILED when using AMD GPU run FFT
Labels: module: cuda, triaged, module: fft

### 🐛 Describe the bug

When running the following function in my code, it reports an error:

The code is:
```
import torch
import torch.nn as nn
import torch.fft
import torch.cuda

...

def fft_loss(patterson, electron_density):
    patterson = patterson[0,0,:,:,:]
    electron_density = electron_density[0,0,:,:,:]
    f1 = torch.fft.fftn(electron_density)
    f2 = torch.fft.fftn(torch.roll(torch.flip(electron_density, [0, 1, 2]), shifts=(1, 1, 1), dims=(0, 1, 2)))
    f3 = torch.mul(f1,f2)
    #f3 = torch.mul(f1.real,f1.real)
    f4 = torch.fft.ifftn(f3)
    f4 = f4.real

    vx = f4 - torch.mean(f4)
    vy = patterson - torch.mean(patterson)

    cost = (torch.sum(vx * vy) / (torch.sqrt(torch.sum(torch.square(vx)) + epsilon) * torch.sqrt(torch.sum(torch.square(vy)) + epsilon)))
    return cost
```

The patterson and electron_density are all tensors.

The error is:
Traceback (most recent call last):
  File "20220418_draft_seblock_20.py", line 274, in <module>
    loss5 = fft_loss(x, yhat)
  File "20220418_draft_seblock_20.py", line 187, in fft_loss
    f1 = torch.fft.fftn(electron_density)
RuntimeError: cuFFT error: HIPFFT_EXEC_FAILED

I'm doubting the AMD GPU doesn't support some of the FFT module? The same script runs successfully on NVIDIA GPUs.

Best,
Sky

### Versions

PyTorch version: 1.8.0a0+56b43f4
Is debug build: False
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: 4.2.21155-37cb3a34

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 11.2.0
Clang version: 3.4.2 (tags/RELEASE_34/dot2-final)
CMake version: version 2.8.12.2
Libc version: glibc-2.17

Python version: 3.8.9 (default, Jan 26 2022, 00:42:48)  [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] (64-bit runtime)
Python platform: Linux-3.10.0-1127.19.1.el7.x86_64-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: Vega 20
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: 3.27.5
MIOpen runtime version: 2.11.0
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.1
[pip3] torch==1.8.0a0+56b43f4
[pip3] torchvision==0.9.0a0+01dfa8e
[conda] Could not collect

cc @ngimel @mruberry @peterbell10
NVFuser microbenchmark classifier - hash on memory formats
Labels: triaged, module: nvfuser

### 🚀 The feature, motivation and pitch

in https://github.com/pytorch/benchmark/blob/main/torchbenchmark/util/classify_graphs.py, we should consider memory formats (e.g. stride orders) as part of the hash. This will separate fusion groups that have different memory formats.

### Alternatives

_No response_

### Additional context

_No response_
RuntimeError: bucket_count == per_bucket_sizes.size() INTERNAL ASSERT FAILED
Labels: module: cuda, triaged, module: ddp

### 🐛 Describe the bug

When training with 4GPUs by using DDP, my program have the following errors: 
```
  File "anaconda3/envs/tnt2/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 993, in forward
    if torch.is_grad_enabled() and self.reducer._rebuild_buckets():
RuntimeError: bucket_count == per_bucket_sizes.size() INTERNAL ASSERT FAILED at "../torch/csrc/distributed/c10d/reducer.cpp":963, please report a bug to PyTorch
```
Other information:
1. If I train the model with batch size 1 on each GPU, the model will throw the above error
2. If I train the model with batch size 10 on each GPU, the model will be blocked at `loss.backward()` with 100% CPU utilization.
3. If I do not use DDP, the model can be trained successfully with 1 single GPU. 


### Versions

Collecting environment information...
PyTorch version: 1.12.0.dev20220419+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 10 (buster) (x86_64)
GCC version: (Debian 8.3.0-6) 8.3.0
Clang version: 7.0.1-8+deb10u2 (tags/RELEASE_701/final)
CMake version: version 3.13.4
Libc version: glibc-2.28

Python version: 3.8.0 (default, Nov  6 2019, 21:49:08)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-5.4.173.1.amd64-smp-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.2.67
GPU models and configuration: 
GPU 0: Quadro RTX 8000
GPU 1: Quadro RTX 8000
GPU 2: Quadro RTX 8000
GPU 3: Quadro RTX 8000

Nvidia driver version: 470.86
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.12.0.dev20220419+cu113
[pip3] torchaudio==0.12.0.dev20220419+cu113
[pip3] torchvision==0.13.0.dev20220419+cu113
[conda] numpy                     1.19.5                   pypi_0    pypi
[conda] torch                     1.12.0.dev20220419+cu113          pypi_0    pypi
[conda] torchaudio                0.12.0.dev20220419+cu113          pypi_0    pypi
[conda] torchvision               0.13.0.dev20220419+cu113          pypi_0    pypi


cc @ngimel
SSL certificate error: urlopen error [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1091)>
Labels: triaged

### 🐛 Describe the bug

I'm facing SSL error while downloading the 'resnext101_32x16d_swsl'  model from torch.hub.load('facebookresearch/semi-supervised-ImageNet1K-models', 'resnext101_32x16d_swsl') 

```
error message: URLError: <urlopen error [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1091)>
```

### Versions


[pip3] numpy==1.21.5
[pip3] torch==1.10.0
[pip3] torchaudio==0.10.0+cu111
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.11.0
[pip3] torchvision==0.11.1+cu111
[RFC] NPU device for PyTorch
Labels: oncall: distributed

### 🚀 The feature, motivation and pitch

NPU is device abstraction of Ascend Neural Networks Process Units.
NPU works as a normal torch device like CPU, GPU and XPU, so PyTorch users can easily migrate model scripts to perform model training/inference on NPU.

The enabling of NPU contains two parts:
1. registering a new 'NPU' device type to PyTorch, which is the focus of this RFC;
2. implementation of PyTorch ops and runtime of NPU, which has be supported via a PyTorch extension, i.e. Ascend extension for PyTorch(https://github.com/Ascend/pytorch). PyTorch users can load this extension to enable the full functionality of NPU, similar to what other PyTorch extensions do.

### Alternatives

_No response_

### Additional context

Work with NPU: 
PyTorch users interface the NPU device as a normal device but need to import torch_npu for full functionality.

```
import torch 
import torch_npu # Ascend Extension for PyTorch 

npu = torch.device('npu') 

input = torch.randn([100]).to(npu) 

model.to(npu) 

output = model(input)
```

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang
__torch_function__ and generator input hazard
Labels: triaged, module: __torch_function__

### 🐛 Describe the bug

Generators in Python are a one-shot business: you can iterate through them once, and then that's it; they'll silently return an empty list upon subsequent iterations. Generators cause trouble for `__torch_function__`, where we may need to iterate through a list multiple times to detect Tensor subclass arguments.

In general, list-accepting PyTorch functions that are bound from C don't accept iterable arguments. However, poorly written userland code can accidentally iterate over iterable arguments multiple times. I committed this mistake with the implementation of torch.autograd.grad:

```
    t_outputs = cast(Tuple[torch.Tensor, ...], (outputs,) if is_tensor_like(outputs) else tuple(outputs))
    t_inputs = cast(Tuple[torch.Tensor, ...], (inputs,) if is_tensor_like(inputs) else tuple(inputs))
    overridable_args = t_outputs + t_inputs
    if has_torch_function(overridable_args):
        return handle_torch_function(
            grad,
            overridable_args,
            outputs,
            inputs,
            grad_outputs=grad_outputs,
            retain_graph=retain_graph,
            create_graph=create_graph,
            only_inputs=only_inputs,
            allow_unused=allow_unused,
            is_grads_batched=is_grads_batched,
        )
```

It is necessary to iterate over the inputs/outputs to detect tensor subclasses, but doing so will destroy an iterator. Thus, we must use the result of the iterator to actually implement the function in question; we cannot preserve the arguments as is.

### Versions

master

cc @hameerabbasi @rgommers @peterbell10
Computer using CPU instead of GPU nvidia with CUDA
Labels: module: cuda, triaged

### 🐛 Describe the bug

Using the YoloV5, it seems that it is using entirely the CPU and not the GPU, and I can't seem to understand why...
I have already opened an issue in the YoloV5 repository (https://github.com/ultralytics/yolov5/issues/7277), but it seems to be a PyTorch problem... Can someone help me?
Here is the problem and some videos.
I have been working with yolov5 for quite some time now, and I always used google colab. But since they are implementing limitations, I have decided to buy an eGPU and an Nvidia 3080ti to train my networks. I successfully installed the drivers and can use de GPU for other software. I can also use the GPU for running a trained network, using yolo detection.py and even using my code based on the PyTorch library.
For some reason, I can't figure out why I can't seem to use my GPU for training. When I use the train.py from yolov5 it doesn't use my Cuda Nvidia GPU. I have tried multiple versions of python, multiple versions of Cuda, and multiple versions of PyTorch (LTS, stable, nightly) and I still can't figure it out.
When I run the train.py file it detects that I have a 3080ti, but when it starts the epochs, my GPU usage is 0% and my CPU usage jumps to 90%, so it is using my CPU.

And here I leave a couple of videos demonstrating the problem. Even thought I'm using YoloV5 the problem seems to be in pytroch, can someone please help me?



### Versions

Collecting environment information...
PyTorch version: 1.11.0+cpu
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.12 (tags/v3.9.12:b28265d, Mar 23 2022, 23:52:46) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19044-SP0
Is CUDA available: False
CUDA runtime version: 11.3.58
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] torch==1.11.0
[pip3] torchaudio==0.11.0+cu113
[pip3] torchvision==0.12.0
[conda] Could not collect


cc @ngimel
Dirichlet with small concentration
Labels: module: distributions, triaged

### 🐛 Describe the bug

When sampling from Dirichlet distribution with a low (<<1) concentration, it returns in about 1/K of the times (K=number of categories) the vector 1/K*(1,1,1,...) i.e (1/3,1/3,1/3) for 3 categories. This shouldn't be the case as the samples are supposed to
be concentrated on the vertices of the simplex. 


Example code:

```python
import torch
from torch import distributions

a = torch.ones(3)*1e-2
b = torch.ones(3)*1e-3
f=distributions.Dirichlet(a)
g=distributions.Dirichlet(b)
f.sample((10,)).max(-1)
g.sample((10,)).max(-1) # here is the bug
```

### Versions

--2022-04-19 12:19:37--  https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
Resolving raw.githubusercontent.com... 185.199.108.133, 185.199.110.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17051 (17K) [text/plain]
Saving to: ‘collect_env.py’

collect_env.py                                        100%[======================================================================================================================>]  16.65K  --.-KB/s    in 0s

2022-04-19 12:19:37 (54.9 MB/s) - ‘collect_env.py’ saved [17051/17051]

cc @fritzo @neerajprad @alicanb @nikitaved
Mobile assets upload could break third party mirrors due to binary data size
Labels: oncall: mobile

### 🐛 Describe the bug

https://github.com/pytorch/pytorch/pull/74947 added mobile assets which seem to be used in mobile unit testing via https://github.com/pytorch/pytorch/pull/74793.
The total size increase using these binary files is ~88MB, which sounds undesirable in a source code repository, given these files are used for testing (please correct me if I'm wrong about their usage).

`GitLab` has a default max. size for binary files of 10MB and will break if you try to rebase your fork with the current PyTorch master. 
A workaround could be to rewrite the git history via a similar command to:
```
git filter-branch --force --index-filter \
  'git rm --cached --ignore-unmatch path/to/ceo.jpg' \
  --prune-empty --tag-name-filter cat -- --all
```
which is also problematic as a simple rebase is not possible anymore and you might end up re-writing the history in each rebase.

Another workaround is to increase the size limit of your mirror to be able to fit the largest binary file of ~45MB.

Could we please host model checkpoints in 3rd party mirrors as is done in e.g. `torchvision` in `https://download.pytorch.org/models/` or any other location outside of this repository?

Reverting this commit or removing the files without re-writing the git history would not solve the original issue, as these files would still be in the history and `git push` would still fail. 

CC @malfet @atalman @linbinyu 

### Versions

All versions after https://github.com/pytorch/pytorch/pull/74947
A bug in instructions for building PyTorch with ASAN
Labels: module: docs, module: ci, triaged, topic: bug fixes

Following instructions in https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#building-pytorch-with-asan , the build fails at the CMake step:
```
<snip>
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX -- compiled but failed to run
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX -- compiled but failed to run
CMake Error at third_party/benchmark/CMakeLists.txt:283 (message):
  Failed to determine the source files for the regular expression backend
```

The problem is in
https://github.com/pytorch/pytorch/blob/bd7e99cbb9e7980b89c26ae3fa5596f6e4aaebc4/CONTRIBUTING.md?plain=1#L1150
where `CXX_FLAGS` is spelled wrong and `-shared-libasan` is missing.

A fix is to replace the above line with:
```
    CXXFLAGS="-shared-libasan -pthread" \
```

On the other hand, this seems to work on CI
https://github.com/pytorch/pytorch/blob/bd7e99cbb9e7980b89c26ae3fa5596f6e4aaebc4/.jenkins/pytorch/build-asan.sh#L39
but [speculations follow] that could be related to the misspelling of `CXX_FLAGS` and the build scripts picks up `CXXFLAGS` from the environment..

cc @brianjo @mruberry @seemethere @malfet @pytorch/pytorch-dev-infra
Jit torchscript for prediction is missing 'forward' when using forward hooks
Labels: oncall: jit

### 🐛 Describe the bug

I'm using forward hooks to extract layer values from a pre-trained CNN and use them as features for my model. I also want to use torchscript for inference. The problem is that when I try to export any other method than 'forward' I get an error that 'forward' is missing for the registered forward hooks. I have a minimal example:

```python
from typing import Iterable, Callable, Tuple
from torch import Tensor, nn, ones, jit, empty
from torchvision.models import resnet50

class FeatureExtractor(nn.Module):
    def __init__(self, model: nn.Module, layers: Iterable[str]):
        super().__init__()
        self.model = model
        self.layers = layers

        for layer_id in layers:
            layer = dict([*self.model.named_modules()])[layer_id]
            layer.register_forward_hook(self.save_outputs_hook(layer_id))

    def save_outputs_hook(self, layer_id: str) -> Callable:
        def fn(_, input: Tuple[Tensor], output):
            print('Hi')
        return fn

    def forward(self, x: Tensor):
        return self.model(x)

    @jit.export
    def predict(self, x: Tensor):
        return self.model(x)

if __name__ == '__main__':
    dummy_input = ones(10, 3, 224, 224)
    resnet_features = FeatureExtractor(resnet50(), layers=["layer4", "avgpool"])
    features = resnet_features(dummy_input)
    script = jit.trace(resnet_features, dummy_input)
```

This fails with:

```
RuntimeError: Couldn't find method: 'forward' on class: '__torch__.torch.nn.modules.container.___torch_mangle_141.Sequential (of Python compilation unit at: 0x7fdc5a676da8)'
```

If I deregister the hooks or export forward instead of predict this of course runs without problem.

### Versions

PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 11.6 (x86_64)
GCC version: Could not collect
Clang version: 13.0.0 (clang-1300.0.29.30)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.8 (main, Nov 10 2021, 09:21:22)  [Clang 13.0.0 (clang-1300.0.29.3)] (64-bit runtime)
Python platform: macOS-11.6-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] pytorch-lightning==1.6.0
[pip3] torch==1.11.0
[pip3] torchmetrics==0.7.3
[pip3] torchvision==0.12.0
[conda] Could not collect
Numerical instability: matrix multiplication got different results on cpu and gpu 
Labels: triaged, module: tf32

### 🐛 Describe the bug

matrix multiplication got different results on cpu and gpu, `5.5665e-06` difference.

Here is a minimal example

```python
import torch

srf = [[0.00784314, 0.00492611, 0.02788845],
       [0.00392157, 0.00492611, 0.03984064],
       [0.00392157, 0.00492611, 0.05976096],
       [0.00392157, 0.00492611, 0.07569721],
       [0.00392157, 0.00492611, 0.09960159],
       [0.00392157, 0.00492611, 0.11553785],
       [0.        , 0.00985222, 0.11952191],
       [0.        , 0.01970443, 0.11553785],
       [0.        , 0.02955665, 0.10756972],
       [0.        , 0.03940887, 0.0876494],
       [0.        , 0.05418719, 0.06374502],
       [0.        , 0.07881773, 0.03585657],
       [0.        , 0.09359606, 0.00796813],
       [0.        , 0.10344828, 0.],
       [0.        , 0.09852217, 0.],
       [0.        , 0.08866995, 0.],
       [0.00784314, 0.07881773, 0.],
       [0.02352941, 0.06896552, 0.],
       [0.04313725, 0.05418719, 0.],
       [0.06666667, 0.03448276, 0.],
       [0.08235294, 0.02463054, 0.00398406],
       [0.08627451, 0.01477833, 0.00398406],
       [0.08235294, 0.00985222, 0.00398406],
       [0.07843137, 0.00985222, 0.00398406],
       [0.07843137, 0.00492611, 0.00398406],
       [0.0745098 , 0.00492611, 0.00398406],
       [0.0745098 , 0.00985222, 0.00398406],
       [0.07058824, 0.00985222, 0.00398406],
       [0.07058824, 0.00985222, 0.00398406],
       [0.06666667, 0.00985222, 0.00398406],
       [0.06666667, 0.00985222, 0.00398406],]

srf_cpu = torch.tensor(srf)
srf_cuda = torch.tensor(srf).cuda()

a = srf_cpu @ srf_cpu.T
b = srf_cuda @ srf_cuda.T

b = b.cpu()

print(torch.allclose(a,b)) # False
print((a-b).abs().max()) # tensor(5.5665e-06)
```


### Versions

pytorch 1.10.1
cudatoolkit 11.3.1 

cc @zasdfgbnm @ptrblck
test_jit.py TestWarn.test_warn and friends doesn't work under pytest
Labels: oncall: jit

### 🐛 Describe the bug

```
pytest test/test_jit.py -k test_warn
```

gives

```
__________________________ TestWarn.test_warn_once_per_func ___________________________
Traceback (most recent call last):                                                       File "/data/users/ezyang/pytorch-tmp/test/jit/test_warn.py", line 93, in test_warn_on
ce_per_func                                
    FileCheck() \
RuntimeError: Expected to find "UserWarning: I am warning you" but did not find it
Searched string:                                                                       From CHECK-COUNT-2: UserWarning: I am warning you
                                           
______________________ TestWarn.test_warn_once_per_func_in_loop _______________________
Traceback (most recent call last):                                                     
  File "/data/users/ezyang/pytorch-tmp/test/jit/test_warn.py", line 117, in test_warn_once_per_func_in_loop
    FileCheck() \                          
RuntimeError: Expected to find "UserWarning: I am warning you" but did not find it
Searched string:                                                                       
From CHECK-COUNT-2: UserWarning: I am warning you                                                           
____________________________ TestWarn.test_warn_only_once _____________________________
Traceback (most recent call last):
  File "/data/users/ezyang/pytorch-tmp/test/jit/test_warn.py", line 50, in test_warn_on
ly_once                                                                                    FileCheck() \
RuntimeError: Expected to find "UserWarning: I am warning you" but did not find it
Searched string:
From CHECK-COUNT-1: UserWarning: I am warning you
```

The warnings then show up in warnings summary so probably there's something going on with pytest's installed warning handler


### Versions

master
Build a default NVFuser comparison callback, e.g. for use with torchbench
Labels: triaged, module: nvfuser

### 🚀 The feature, motivation and pitch

A default comparison callback would make it a lot easier to start using the callback.

Callback functionality was added in https://github.com/pytorch/pytorch/pull/74361.

### Alternatives

_No response_

### Additional context

_No response_
DISABLED test_zero_model_parallel_parameters_as_bucket_view_True (__main__.TestZeroRedundancyOptimizerDistributed)
Labels: oncall: distributed, module: rocm, skipped

Platforms: rocm

This test was disabled because it is failing flakily on master ([recent examples](http://torch-ci.com/failure/test_zero_model_parallel_parameters_as_bucket_view_True%2C%20TestZeroRedundancyOptimizerDistributed)).

Culprit could be related to https://github.com/pytorch/pytorch/pull/75753 @pritamdamania 

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @jeffdaily @sunway513 @jithunnair-amd @ROCmSupport @KyleCZH
API to determine if a torch.return_type is a "structseq"
Labels: feature, triaged

## Motivation

A "structseq" is this weird Python-C API thing used to create namedtuple-like types from C++.

https://github.com/pytorch/pytorch/pull/75915 introduces some logic that ideally would grab all "structseq" classes from torch.return_types and register them as PyTree nodes. However, we didn't know how to actually check if something is a structseq or not (the [documentation](https://docs.python.org/3/c-api/tuple.html?highlight=structseq#struct-sequence-objects) doesn't provide any helper functions for that).

If possible we should write an API that can check if a type is a "structseq" and then use it in https://github.com/pytorch/pytorch/pull/75915
jit/_trace.py", line 71, in _unique_state_dict     filtered_dict[k] = v.detach() AttributeError: 'torch.dtype' object has no attribute 'detach'
Labels: oncall: jit

### 🐛 Describe the bug

Try exporting torch.fx quantized model to onnx get this error:

```jit/_trace.py", line 71, in _unique_state_dict
    filtered_dict[k] = v.detach()
AttributeError: 'torch.dtype' object has no attribute 'detach'

```

From discussion forum, it was claimed quantized model export to onnx already supported, but I still get this error on a single res18 model from torchvision.



### Versions

torch.1.11
FSDP remove the requirement of all trainable parameters  
Labels: high priority, oncall: distributed, triaged, module: fsdp

### 🚀 The feature, motivation and pitch

There are some cases that the model may have some fixed/untrainable parameters which should not be updated (.

In this case, the FSDP will raise the exception at https://github.com/pytorch/pytorch/blob/0df2e863fbd5993a7b9e652910792bd21a516ff3/torch/distributed/fsdp/flatten_params_wrapper.py#L363

assert (len(set(p.requires_grad for p in params)) == 1
        ), "expects all parameters to have same requires_grad"

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @gchanan @zou3519 @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @pietern @SciPioneer
AllGather with backward support async_op=True
Labels: oncall: distributed

### 🚀 The feature, motivation and pitch

I has one project which used all gather contrastive loss, but  _AllGather[https://github.com/pytorch/pytorch/blob/master/torch/distributed/nn/functional.py#L276] does not support async_op, so it cannot overlap computation and communication.

### Alternatives

_No response_

### Additional context

_No response_

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang
torch.jit.trace error when custom autograd function used in the model
Labels: oncall: jit

### 🐛 Describe the bug

custom autograd function defined in the model, jit.trace reports error

```
class GradientMultiplyLayer(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input, mask_bw):
        ctx.save_for_backward(mask_bw)
        return input

    @staticmethod
    def backward(ctx, grad_output):
        mask_bw, = ctx.saved_tensors
        return grad_output.mul(mask_bw), None

...
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("model_for_libtorch.pt")

Could not export Python function call 'GradientMultiplyLayer'. Remove calls to Python functions before export. Did you forget to add @script or @script_method annotation? If this is a nn.ModuleList, , add it to __constants__:
```

I can't solve this problem. thanks for your help

### Versions

Collecting environment information...
PyTorch version: 1.11.0+cu113
Is debug build: False
*CUDA used to build PyTorch: 11.3
*ROCM used to build PyTorch: N/A
OS: Microsoft Windows 10 
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A
Python version: 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.18362-SP0
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1660 Ti
Nvidia driver version: 511.79
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.11.0+cu113
[pip3] torchaudio==0.11.0+cu113
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.12.0+cu113
[conda] Could not collect
Disable TracerWarnings on NVFuser opinfo tests
Labels: oncall: jit, module: bootcamp, triaged

### 🚀 The feature, motivation and pitch

If you run the nvfuser opinfo tests, you'll see a lot of tracer warnings like this:

```
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the da
ta flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
```

(you can repro with `$ PYTORCH_TEST_WITH_SLOW=1 gpui python test/test_jit_cuda_fuser.py -k test_nvfuser_correctness__masked_log_softmax_cuda_f
loat64`, for example)

We can disable these with the [NoWarn](https://github.com/pytorch/pytorch/blob/aa51ee2345022db1f318b4adf460abe461e38bc5/torch/csrc/jit/frontend/tracer.h#L179-L194) guard, but currently it doesn't have python bindings - so you'll have to add python bindings first. You can model this after InsertPointGuard which has both a [c++ implementation](https://github.com/pytorch/pytorch/blob/aa51ee2345022db1f318b4adf460abe461e38bc5/torch/csrc/jit/ir/ir.h#L1444-L1456) and a [python implementation](https://github.com/pytorch/pytorch/blob/aa51ee2345022db1f318b4adf460abe461e38bc5/torch/jit/_ir_utils.py#L4-L18)

### Alternatives

_No response_

### Additional context

_No response_
autogen-58 microbenchmark fails on NNC gpu fusion
Labels: NNC

### 🐛 Describe the bug

autogen-58 (from https://github.com/pytorch/benchmark/pull/801) fails on NNC.

Repro is [here](https://gist.github.com/davidberard98/12fdd498e5284a543f2daa55addc8445). Output is:

```
$ gpui python 58-repro.py
Traceback (most recent call last):
  File "/fsx/users/dberard/benchmark/58-repro.py", line 261, in <module>
    log_extract.run_nnc(ir, inputs, dynamic=False)
  File "/fsx/users/dberard/pytorch/torch/utils/jit/log_extract.py", line 105, in run_nnc
    return run_test(ir, inputs)
  File "/fsx/users/dberard/pytorch/torch/utils/jit/log_extract.py", line 75, in run_test
    graph(*inputs)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: CUDA driver error: too many resources requested for launch

srun: error: dev-st-p4d24xlarge-1: task 0: Exited with exit code 1
```

cc @ZolotukhinM @huiguoo 

### Versions

A100 (AWS cluster), master.