gopikrsmscs/torch-issues huggingface dataset:
[Prims+NVFuser] nvFuser running into "Tensors of type SparseTensorImpl do not have strides"
Labels: triaged, module: nvfuser

After enabling aten2aten decomps: 

===== dlrm_backward_0 ======
Generating testing data...
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.threshold_backward.default <function threshold_backward at 0x7f1c8b8dd310>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.threshold_backward.default <function threshold_backward at 0x7f1c8b8dd310>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.threshold_backward.default <function threshold_backward at 0x7f1c8b8dd310>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.detach.default <function nop_decomposition at 0x7f1c8b87d430>
aten2aten decomp:  aten.threshold_backward.default <function threshold_backward at 0x7f1c8b8dd310>
aten2aten decomp:  aten.slice_backward.default <function slice_backward at 0x7f1c8b8e78b0>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f1c8b87d5e0>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f1c8b87d5e0>
aten2aten decomp:  aten.transpose.int <function transpose_int at 0x7f1c8b87d5e0>
**dlrm_backward_0 failed! Tensors of type SparseTensorImpl do not have strides**
Nested tensor: Support Noncontiguous Buffer
Labels: triaged, topic: not user facing, release notes: nested tensor

Current implementation of nested tensor assume the buffer memory to be contiguous. However, some operations can break that assumption:
* reshape
* transpose
* slice

To deal with discontinuity, metadata strides and offsets are introduced in [#79831](https://github.com/pytorch/pytorch/pull/79831) and [#80981](https://github.com/pytorch/pytorch/pull/80981). 

Let's update existing nested tensor functionalities that assumed continuity to embrace the discontinuity and make use of the new metadata. For example, the way to access unbinded tensors from discontinuous buffer is `buffer.as_strided(size, stride, offset)`
[Prims+NVFuser] aten.to.dtype refs impl causing segfault
Labels: triaged, module: nvfuser

model: pytorch_struct_forward_0
Condition: enable aten2aten decomps 

Partitions proposed:
partition 0 ['to_dtype_45', 'exp_default_2', 'sum_dim_int_list_2', 'log_default_2', 'sub_tensor_5', 'sub_tensor_4']
partition 2 ['relu_default_7', 'to_dtype_43', 'add_tensor_12', 'mul_tensor_19', 'to_dtype_40', 'mul_tensor_18', 'add_tensor_13']
partition 4 ['to_dtype_39', 'relu_default_6', 'to_dtype_36', 'add_tensor_11', 'mul_tensor_16', 'to_dtype_41', 'mul_tensor_17']
partition 6 ['add_tensor_10', 'to_dtype_35', 'relu_default_5', 'to_dtype_37', 'mul_tensor_15', 'to_dtype_32', 'mul_tensor_14', 'add_tensor_9']
partition 8 ['to_dtype_33', 'mul_tensor_13', 'add_tensor_8', 'to_dtype_31', 'to_dtype_28', 'relu_default_4', 'mul_tensor_12']
partition 10 ['to_dtype_27', 'add_tensor_7', 'mul_tensor_11', 'mul_tensor_10', 'to_dtype_29', 'to_dtype_24']
partition 13 ['log_default_1', 'sum_dim_int_list_1', 'exp_default_1', 'sub_tensor_2', 'to_dtype_23', 'sub_tensor_3']
partition 15 ['sub_tensor_1', 'log_default', 'sum_dim_int_list', 'exp_default', 'sub_tensor', 'to_dtype_21']
partition 17 ['to_dtype_16', 'add_tensor_6', 'relu_default_3', 'to_dtype_19', 'add_tensor_5', 'mul_tensor_9', 'mul_tensor_8']
partition 19 ['mul_tensor_6', 'to_dtype_17', 'mul_tensor_7', 'add_tensor_4', 'to_dtype_12', 'to_dtype_15', 'relu_default_2']
partition 21 ['mul_tensor_5', 'to_dtype_13', 'to_dtype_8', 'add_tensor_2', 'to_dtype_11', 'relu_default_1', 'add_tensor_3', 'mul_tensor_4']
partition 23 ['mul_tensor_2', 'to_dtype_9', 'mul_tensor_3', 'add_tensor_1', 'to_dtype_7', 'to_dtype_4', 'relu_default']
partition 25 ['mul_tensor', 'mul_tensor_1', 'to_dtype', 'add_tensor', 'to_dtype_3', 'to_dtype_5']

**srun: error: a100-st-p4d24xlarge-6: task 0: Segmentation fault (core dumped)**

Workaround: 
Remove "torch.ops.aten.to.dtype" from operator_support list. 

My guess:

Following ref implementation has some issue. 


def aten_to_dtype(self, dtype: torch.dtype, **kwargs):
    if len(kwargs) > 0 or not dtype:
        raise RuntimeError("No support for other to.dtype() formats other than to.dtype(self, dtype)")
    return torch._prims.convert_element_type(self, dtype)
[ONNX] Tool to find mismatch in exported ONNX model
Labels: module: onnx, triaged, onnx-triaged, onnx-needs-info

### 🚀 The feature, motivation and pitch

Output mismatch between exported ONNX and PyTorch models has been a hard to debug problem. Developers and data scientists usually encounter this problem in the following scenarios. 

* A new kind of model exports successfully to ONNX, but outputs don’t match. 
* An ONNX model that works correctly on a given set of inputs (usually the ones it was originally exported with), but outputs don’t match for other set of inputs. 
* An ONNX model that matched results previously, but due to package version changes the results start to differ. 

So far there hasn’t been any effective/automatic tools to help with investigation. Developers had to manually compare results between subgraph of ONNX and PyTorch. 

Usually, the root cause of mismatch problem is not hard to fix, after it was discovered. It could be any one of the following scenarios: 

* The operator use case is new and not exported correctly. 
* Dynamic value (sometimes could be configuration or hyperparameter) exported as constant. 
* Dynamic graph (controlflow) exported as static graph. 
* Discrepancies between PyTorch and ONNX operator spec/implementation.

## Proposal

Proposing a new API under `torch.onnx` module to try export the model and automatically find the first operator/node that produces mismatch results. 

## Goals 

* To find the first operator/node in ONNX model, and corresponding line in PyTorch that starts the mismatch. 
* The first version may support only tracing based export. 

## Non-Goals 

* This tool does not fix the mismatch. 
* This tool requires successful export as pre-condition: it does not fix and cannot work under export failures. 

### Alternatives

_No response_

### Additional context

_No response_
[Prims+NVFuser] Aten2Aten decomp hurting performance
Labels: triaged, module: nvfuser, module: primTorch

Following aten2aten decomps are not latter fused by nvFuser, thus hurting the performance. 

- [ ] aten.native_layer_norm_backward.default

  | Module Name | Eager Time (ms) | WithAtenDecomp Time (ms) | nvFuser Time (ms)
-- | -- | -- | -- | --
with | hf_T5_backward_0 | 159.1049759 | 3191.585663 | 3186.037783
without | hf_T5_backward_0 | 183.5727645 | 170.5960212 | 165.7386031
with | hf_T5_backward_8 | 237.4839457 | 3832.827996 | 3867.255308
without | hf_T5_backward_8 | 256.4443965 | 256.3290102 | 249.5397562



- [ ] aten.embedding_dense_backward.default

  | Module Name | Eager Time (ms) | AtenDecomp Time | nvFuser Time (ms)
-- | -- | -- | -- | --
with | hf_DistilBert_backward_0 | 13.35794386 | 1523.864605 | 31.73270822
without | hf_DistilBert_backward_0 | 13.90568446 | 13.46321963 | 13.67696188




cc @ezyang @mruberry @ngimel
ExpandedWeights sometimes fail silently and doesn't compute .grad_sample attribute
Labels: module: nn, triaged

### 🐛 Describe the bug

Normally `call_for_per_sample_grads` detects inputs where it can't compute per sample gradients and throws user-friendly exception with the explanation.
However, in some scenarios (see below) it just skips certain parameters and doesn't do anything with them. As such, after the backward pass some parameters would have only `.grad_sample` attribute, and some - only `.grad` attribute. All this happens silently to the user and the user would only notice the issue when they try to access `.grad_sample` attribute.

I haven't done a detailed investigation as to when this occurs, but it might have something to do with recurrent nets. Below is the example using opacus custom-written RNN implementation.

```python
import torch
import torch.nn as nn
from torch.nn.utils._per_sample_grad import call_for_per_sample_grads

from opacus.layers import DPRNN

N=1
T=1
D=4
H=8
num_layers=1
bias=False
batch_first=True
bidirectional=False
using_packed_sequences=False
packed_sequences_sorted=False

rnn = DPRNN(
    D,
    H,
    num_layers=num_layers,
    batch_first=batch_first,
    bias=bias,
    bidirectional=bidirectional,
)
x = torch.randn([N, T, D])

# out, _ = rnn(x)
out, _ = call_for_per_sample_grads(rnn, x.shape[0], x)
out.sum().backward()

rnn.l0.ih.weight.grad_sample
```

Error
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/var/folders/_r/hzffvgfd23sc3w0gr577_qmc0000gn/T/ipykernel_71302/2553106598.py in <module>
     26 out.sum().backward()
     27 
---> 28 rnn.l0.ih.weight.grad_sample

AttributeError: 'Parameter' object has no attribute 'grad_sample'
```

### Versions

PyTorch version: 1.12.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.4 (x86_64)
GCC version: Could not collect
Clang version: 12.0.5 (clang-1205.0.22.9)
CMake version: Could not collect
Libc version: N/A

Python version: 3.7.5 (default, Oct 22 2019, 10:35:10)  [Clang 10.0.1 (clang-1001.0.46.4)] (64-bit runtime)
Python platform: Darwin-21.5.0-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.910
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.4
[pip3] pytorch-lightning==1.5.3
[pip3] torch==1.12.0
[pip3] torchaudio==0.12.0
[pip3] torchmetrics==0.6.0
[pip3] torchvision==0.13.0
[conda] Could not collect

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
Wrong example of sliced computation in doc page Numerical Accuracy
Labels: module: docs, triaged, module: numerical-reproducibility

### 📚 The doc issue

This issue is related to the page [Numerical Accuracy](https://pytorch.org/docs/stable/notes/numerical_accuracy.html#batched-computations-or-slice-computations)
(Accessed July 6 2022)
(The source of the page in the repository is [here](https://github.com/pytorch/pytorch/blob/master/docs/source/notes/numerical_accuracy.rst#batched-computations-or-slice-computations). But I'm not sure which commit).

Quote from the page:

> let `A` be a 2-dimentional tensor. `A.sum(-1)[0]` is not guaranteed to be bitwise equal to `A[:,0].sum()`.

Viewing `A` as a matrix, notice that `A.sum(-1)[0]` is the sum of the first row, whereas `A[:,0].sum()` is the sum of the first column. These are trivially different, and this difference is not related to numerical accuracy.

See for example:
```python
import torch
A = torch.ones(4,5)
x = A.sum(-1)[0] # equals 5
y = A[:,0].sum() # equals 4
```

I suspect that the intention in this example was either to use `A.sum(0)[0]` for the first calculation or `A[0].sum()` for the second.

### Suggest a potential alternative/fix

I believe that this sentence: 
> `A.sum(-1)[0]` is not guaranteed to be bitwise equal to `A[:,0].sum()`.

should be replaced with one of the following:
> `A.sum(0)[0]` is not guaranteed to be bitwise equal to `A[:,0].sum()`.

> `A.sum(-1)[0]` is not guaranteed to be bitwise equal to `A[0].sum()`.

(which one? depends on the numerical accuracy issue here...)

cc @svekars @holly1238
[jit] script backward wrong gradient
Labels: oncall: jit

### 🐛 Describe the bug

Consider the following simple pnorm implementation
```py
def pnorm(x: torch.Tensor, p: torch.Tensor) -> torch.Tensor:
    return x.abs().pow(p).sum().pow(1 / p)

script_pnorm = torch.jit.script(pnorm)
```

When `x` contains any entry of *zero*, gradient of `p` will become `NaN` for script version, but works normally (not `NaN`) for eager version.

See this gist for full example.
https://gist.github.com/SsnL/07f58873766a2e8456741b1281e04c24

### Versions

```
PyTorch version: 1.11.0+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.22.5
Libc version: glibc-2.26

Python version: 3.7.13 (default, Apr 24 2022, 01:04:09)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: False
CUDA runtime version: 11.1.105
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.11.0+cu113
[pip3] torchaudio==0.11.0+cu113
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.12.0
[pip3] torchvision==0.12.0+cu113
[conda] Could not collect
```
Position embedding aware global circular convolution
Labels: feature, module: nn, triaged

### 🚀 The feature, motivation and pitch

Recently, we propose a new basic operation position embedding aware global circular convolution (GCC). Differing from previous convolution operations, the proposed GCC has global receptive field.  Experimental results show that GCC  uniformly
improves performance of various typical models.  

This work has been accpted by ECCV 2022, we hope the proposed GCC can be used by other researchers. Someone may refer to https://arxiv.org/abs/2203.03952 to find more details. 





### Alternatives

We will prepare a CUDA optimized version GCC following tutorial in https://pytorch.org/tutorials/advanced/cpp_extension.html.

We also want to propose a pull request to merge our code into Pytorch.  Will such a pull request be accpted ? If it is acceptable, is there anyone can provide us a demo of proposing this kind of pull request to Pytorch?



### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
Interpolation artifacts when using nn.interpolate, trilinear mode for 3D label images
Labels: module: nn, triaged, module: interpolation

### 📚 The doc issue

It seems a functional bug. I have use nn.interpolate trilinear to umsample a 2 classes(and background) 3D image. The function tend to interpolate a "1" between label 2 and the background using trilinear. I could be solved by transfer the label image into one-hot format and than resample. Or i can also use nearest mode of nn.interpolate, its ok with label image.
It seems trilinear is not a suitable method for label image. If i use trilinear to upsample the training images, and use nearest to upsample the labels. If there could be some errors between images and labels because of the different upsample method?


### Suggest a potential alternative/fix

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
make_fx doesn't work with truly dynamic argument functions (e.g. fx.Interpreter)
Labels: triaged, module: fx

### 🐛 Describe the bug

sample which breaks:

```
make_fx(lambda *args: torch.cat(args))(torch.randn(2), torch.randn(2))
```

Fails with

```
  File "test/test_dynamo_cudagraphs.py", line 145, in cudagraphs
    make_fx(lambda *args: torch.cat(args))([torch.randn(2), torch.randn(2)])
  File "/raid/ezyang/pytorch-scratch2/torch/fx/experimental/proxy_tensor.py", line 281, in wrapped
    t = dispatch_trace(wrap_key(f, args), tracer=fx_tracer, concrete_args=tuple(phs))
  File "/raid/ezyang/pytorch-scratch2/torch/fx/experimental/proxy_tensor.py", line 177, in dispatch_trace
    graph = tracer.trace(root, concrete_args)
  File "/raid/ezyang/torchdynamo/torchdynamo/eval_frame.py", line 88, in _fn
    return fn(*args, **kwargs)
  File "/raid/ezyang/pytorch-scratch2/torch/fx/_symbolic_trace.py", line 667, in trace
    fn, args = self.create_args_for_root(
  File "/raid/ezyang/pytorch-scratch2/torch/fx/_symbolic_trace.py", line 524, in create_args_for_root
    raise RuntimeError(
RuntimeError: Tracing expected 0 arguments but got 1 concrete arguments
```

The problem is FX knows how to see through varargs if there is an inner function its wrapping that has a true argument list, but cannot deal with truly dynamic lists. This is fine for vanilla FX as FX is unwilling to burn in the number of tensors in the list but for AOTAutograd/ProxyTensor we should burn in the list count and trace anyway.

cc @ezyang @Chillee @zdevito @jamesr66a 

### Versions

master
slow test infra cannot handle nested suites
Labels: module: ci, triaged

### 🐛 Describe the bug

We currently categorize a selection of slow test cases in our repo to run on a particular config. We query for tests slower than a certain threshold and put these tests into a json every night. The slow tests json generated by our infra looks like: https://github.com/pytorch/test-infra/blob/generated-stats/stats/slow-tests.json.

The code in our CI that parses the JSON to detect whether or not to run the test is https://github.com/pytorch/pytorch/blob/master/torch/testing/_internal/common_utils.py#L1544. Note that we perfectly match a test name with the keys of the JSON. 

This is almost always correct, except not all test suites proceed with a `__main__`, which we assume in our query here https://github.com/pytorch/test-infra/blob/main/torchci/rockset/commons/__sql/slow_tests.sql#L57.

### Solution
Let's change the logic in the test selection side to not care about `__main__` when figuring out if a test is a match.

### Versions

CI related

cc @seemethere @malfet @pytorch/pytorch-dev-infra
C++ extensions inject a bunch of compilation flags
Labels: oncall: binaries, module: cpp-extensions, triaged, better-engineering

### 🐛 Describe the bug

Discovered by @vfdev-5 

This is possibly intentional, but I've noticed that C++ extensions, while building on linux, inject a bunch of compilation flags. For example, in functorch, we do not specify either of ['-g -NDEBUG'](https://github.com/pytorch/functorch/blob/b22d52bb15276cd919815cde01a16d4b3d8f798e/setup.py#L84-L88), but those appear in [our build logs](https://github.com/pytorch/functorch/runs/7140654131?check_suite_focus=true).

The end result of `-g` is that our linux binaries are a lot larger than they need to be (the debug symbols are 10s of mb) while our mac and windows binaries are very small < 1mb. I would expect the domain libraries to have the same issue.

## The diagnosis

In C++ extensions we adopt [certain flags from distutils](https://github.com/pytorch/pytorch/blob/f7678055033045688ae5916c8df72f5107d86a4a/torch/utils/cpp_extension.py#L558).

These flags come from Python's built-in sysconfig module (https://github.com/python/cpython/blob/ec5e253556875640b1ac514e85c545346ac3f1e0/setup.py#L476), which contains a list of flags used to compile Python. These include -g, among other things.

This is probably intentional (I wrote these lines by copy-pasting them from distutils) but I don't have a good understanding of why these flags are necessary or if only some of them are.

### Versions

main

cc @ezyang @seemethere @malfet @zou3519
[BE] Refactor FSDP Unit Tests
Labels: triaged, better-engineering, module: fsdp

## Motivation & Goals
1. In H2, we plan to land new FSDP features such as non-recursive wrapping and multiple parameter group support within one `FlatParameter`. Since these features correspond to distinct code paths, the core unit tests need to parameterize over them. This means an increase in the time-to-signal (TTS) by up to 4x and may increase with further features (e.g. native XLA support).
**Goal 1: Cut down the existing TTS while maintaining feature coverage**.
2. The test files have become fragmented. Despite the existence of `common_fsdp.py`, test files often define their own models and training loops, leading to redundancy, and the methods that do exist in `common_fsdp.py` are not well-documented, leading to confusion.
**Goal 2: Refactor common models and boilerplate to `common_fsdp.py`.**
3. For the non-recursive wrapping code path, the model construction differs from the existing recursive wrapping code path. As a result, much of the existing tests cannot be directly adapted to test the non-recursive path by simply changing a single `FullyShardedDataParallel` constructor call.
**Goal 3: Refactor model construction to enable simpler testing for the non-recursive wrapping path.**

## Status Quo
On the AI AWS cluster with 2 A100 GPUs, the current TTS is approximately **4967.06 seconds = 82.78 minutes = 1.38 hours**[0]. PyTorch Dev Infra has asked to have our multi-GPU tests run in < 75 minutes in CI, which uses P60 GPUs.

The largest contributors to the TTS are `test_fsdp_core.py` (2191.89 seconds = 36.53 minutes) and `test_fsdp_mixed_precision.py` (1074.42 seconds  = 17.91 minutes), representing almost 2/3 of the total. Since these test the core FSDP runtime, newly-added code paths will target these tests.

[0]This is a point estimate from running each test file once and excludes recent changes to the test files (my stack was rebased on a commit from 6/21).

## Approach

I will proceed with a series of PRs. The order will be to address Goal 3 -> Goal 1 -> Goal 2.
- For Goal 3, I will introduce a common interface `FSDPTestModel`.
https://github.com/pytorch/pytorch/pull/80873
- For Goal 1, I will use `self.subTest()` with `dist.barrier()` to avoid the expensive process spawn and `dist.init_process_group()` for each parameterization.
https://github.com/pytorch/pytorch/pull/80908
https://github.com/pytorch/pytorch/pull/80915

https://blog.ganssle.io/articles/2020/04/subtests-in-python.html
> There are, however, occasionally situations where the subTest form factor offers some advantages even in parameterization. For example, if you have a number of tests you'd like to perform that have an expensive set-up function that builds or acquires an immutable resource that is used in common by all the subtests:
- For Goal 2, I will perform a deep comb through the existing test suite. 

## Related Issues
https://github.com/pytorch/pytorch/issues/80872
https://github.com/pytorch/pytorch/issues/78277
https://github.com/pytorch/pytorch/issues/67288

cc @zhaojuanmao @mrshenli @rohan-varma @ezyang
Compatibility List
Labels: oncall: binaries, module: docs

### 🚀 The feature, motivation and pitch

Have an easily to find (e.g. in the previous-versions page) compatibility list. If it has to be in the form of commands and comments ( just like in the previous-versions page), instead of a proper table, then this would be fine, too. 

However I believe many people wont know what version to use. and a proper tabulation would help. 

### Alternatives

wasting good nights sleep to find out it doesn't work, after jumping through all the hoops was not a good alternative. 

### Additional context

this is no "help request" or anything, but simply a feature request for a general professional table (or some list of arbitrary commands)

for context: 

Even though my pytorch version and "CUDA SDK version" are valid and below the max supported driver version as reported by smi (max = 11.4, installed 11.3) ( `pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3` ) pytorch slams the GC max supported "CUDA version" as 3.7 ... nothing you can actually find out until after the fact based on looking at the installation information 


cc @ezyang @seemethere @malfet @svekars @holly1238
[DDP] doesn't support multiple backwards when static_graph=True
Labels: oncall: distributed, module: ddp

### 🐛 Describe the bug

```python
import torch
import torch.distributed as dist
import torch.nn as nn
from torch.nn.parallel import DistributedDataParallel as DDP

class ToyModel(nn.Module):
    def __init__(self, in_dim=10, out_dim=5):
        super(ToyModel, self).__init__()
        self.dense1 = nn.Linear(in_dim, in_dim)
        self.dense2 = nn.Linear(in_dim, out_dim)
    
    def forward(self, x):
        x = self.dense1(x)
        return self.dense2(x)


dist.init_process_group("nccl")
rank = dist.get_rank()

model = ToyModel().to(rank)
ddp_model = DDP(model, device_ids=[rank], find_unused_parameters=True)
ddp_model._set_static_graph()

x = torch.randn(5, 10)
y = torch.randn(5, 5).to(rank)
loss_fn = nn.MSELoss()

output = ddp_model(x)
loss = loss_fn(output, y)

loss.backward(retain_graph=True)
output.backward(torch.zeros_like(output))
```

It will report the error:
```
SystemError: <built-in method run_backward of torch._C._EngineBase object at 0x7fb37a2ec200> returned NULL without setting an error
```
But if I call the prepare_for_backward before backward (like #47260),
```python
ddp_model.reducer.prepare_for_backward(loss)
loss.backward(retain_graph=True)
ddp_model.reducer.prepare_for_backward(output)
output.backward(torch.zeros_like(output))
```
It works and the output seems ok. But I don't know if this is correct or potentially risky?

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501 @ezyang @gchanan @zou3519 @bdhirsh @agolynski @mrzzd @xush6528


### Versions

PyTorch 1.10.0
Can torchscript dump backward graph?
Labels: oncall: jit

### 🚀 The feature, motivation and pitch

Can torchscript dump backward graph?
I am interested in tracing through both the forward and backward graph using TorchScript and dumping the IR, for full graph optimization in a separate framework.

Currently,  can torchscript dump backward graph?

### Alternatives

_No response_

### Additional context

_No response_
Ne op does not behaves as expected with nan
Labels: high priority, needs reproduction, triaged

### 🐛 Describe the bug

When  Nan values are given as input torch.ne op behaves in a strange manner

```
import torch 
import numpy as np
x= torch.tensor([  -np.inf, 0.3516, 0.3719, 0.5452, 0.4024, 0.9232, 0.6995, 0.0805, 0.7434,0.1871, 0.3802, 0.5379, 0.1533,    np.nan, 0.8519, 0.7572,    np.inf, 0.4675, 0.4702, 0.2297, 0.5905, 0.6923, 0.2628,   -np.inf,   -np.inf, 0.6335, 0.9912,0.9256, 0.0237, 0.4891,    np.nan, 0.9731])
x.ne__(45)
# when x.size is greater than or equal to 32, ne  op returns 0 in the place of NAN

x= torch.tensor([  -np.inf, 0.9232, 0.6995, 0.0805, 0.7434,0.1871, 0.3802, 0.5379, 0.1533,    np.nan, 0.8519, 0.7572,    np.inf, 0.4675, 0.4702, 0.2297, 0.5905, 0.6923, 0.2628,   -np.inf,   -np.inf, 0.6335, 0.9912,0.9256, 0.0237, 0.4891,    np.nan, 0.9731])
x.ne__(45)
# when x.size is lesser than 32, ne  op returns 1 in the place of NAN


```


### Versions

Collecting environment information...
PyTorch version: 1.11.0+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.22.5
Libc version: glibc-2.26

Python version: 3.7.13 (default, Apr 24 2022, 01:04:09)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: False
CUDA runtime version: 11.1.105
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.11.0+cu113
[pip3] torchaudio==0.11.0+cu113
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.12.0
[pip3] torchvision==0.12.0+cu113
[conda] Could not collect

cc @ezyang @gchanan @zou3519
Add typing support to ModuleList and ModuleDict
Labels: module: typing, triaged

### 🚀 The feature, motivation and pitch

Currently, the containers `nn.ModuleList` and `nn.ModuleDict` are typing-unaware, i.e. given this:

```python
class A(nn.Module):
    def __init__(self):
        self.my_modules = nn.ModuleList([nn.Linear(1, 1) for _ in range(10)])
```

`self.my_modules[i]` is treated as `nn.Module`, not as `nn.Linear`. For example, VSCode complains about snippets like `function_that_expects_tensor(self.my_modules[i].weight`), because it thinks that `.weight` can be both `Tensor` and `nn.Module`.

What I propose:

```python
from collections.abc import MutableSequence
from typing import TypeVar

ModuleListValue = TypeVar('ModuleListValue', bound=nn.Module)

class ModuleList(Module, MutableSequence[ModuleListValue]):
    # now, some methods can be typed, e.g.:    
    def __getitem__(...) -> ModuleListValue:
        ...
    ...
```

For `nn.DictModule`, it is a bit more complicated, since there are two different patterns:
- (A) `dict`-like: the set of keys is not fixed, all values are modules of the same type (e.g. `nn.Linear`)
- (B) `TypedDict`-like: the set of keys is fixed, the values can be of different types (e.g. `{'linear': nn.Linear(...), 'relu': nn.ReLU}`).

(A) can be implemented similarly to the previous example:

```python
...
class ModuleDict(Module, MutableMapping[str, ModuleDictValue]):
    ...
```

In fact, this can cover (B) as well in a very limited way (by setting `ModuleDictValue=nn.Module`). And it is unclear to me how to implement the fully functioning (B), it looks like we need something like `TypedMutableMapping`, but there is no such thing in typing. So I would start with `MutableMapping` and add TypedModuleDict when it becomes technically possible.

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @malfet @rgommers @xuzhao9 @gramster
torch.einsum results in segfault
Labels: high priority, triage review, oncall: binaries, module: crash, module: openmp, module: multithreading

### 🐛 Describe the bug

I experience a segfault when running a simple script like:
```python
import torch

As = torch.randn(3, 2, 5)
Bs = torch.randn(3, 5, 4)

torch.einsum("bij,bjk->bik", As, Bs)
```
Running this results in:
```console
$ python test.py 
Segmentation fault: 11
```
Even if I add `-X faulthandler` I don't seem to get any kind of stacktrace to help locate the issue. If someone can give me instructions for how to use gdb I can try to get a backtrace.

### Versions

```console
$ python collect_env.py 
Collecting environment information...
PyTorch version: 1.12.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.4 (arm64)
GCC version: Could not collect
Clang version: 13.1.6 (clang-1316.0.21.2.5)
CMake version: version 3.23.1
Libc version: N/A

Python version: 3.9.13 (main, Jun 18 2022, 21:43:00)  [Clang 13.1.6 (clang-1316.0.21.2.5)] (64-bit runtime)
Python platform: macOS-12.4-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] efficientnet-pytorch==0.6.3
[pip3] mypy==0.931
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.0
[pip3] pytorch-lightning==1.6.4
[pip3] pytorch-sphinx-theme==0.0.24
[pip3] segmentation-models-pytorch==0.2.0
[pip3] torch==1.12.0
[pip3] torchmetrics==0.7.2
[pip3] torchvision==0.12.0a0
[conda] Could not collect
```

cc @ezyang @gchanan @zou3519 @seemethere @malfet
`torch.renorm` gives wrong gradient for 0-valued input when `p` is even and `maxnorm=0`.
Labels: module: autograd, triaged, module: edge cases

### 🐛 Describe the bug

`torch.renorm` gives wrong gradient for 0-valued input when `p` is even and `maxnorm=0`.

```py
import torch

def fn(input):
    p = 2
    dim = -1
    maxnorm = 0
    fn_res = torch.renorm(input, p, dim, maxnorm, )
    return fn_res
input = torch.tensor([[0.1, 0.], [0., 0.]], dtype=torch.float64, requires_grad=True)

torch.autograd.gradcheck(fn, (input))
```

```
GradcheckError: Jacobian mismatch for output 0 with respect to input 0,
numerical:tensor([[0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.]], dtype=torch.float64)
analytical:tensor([[1., 0., 0., 0.],
        [0., 1., 0., 0.],
        [0., 0., 1., 0.],
        [0., 0., 0., 1.]], dtype=torch.float64)
```

Because `p=2` and `maxnorm=0`, this function should be `f(x) = 0` for every element. Therefore, it should return 0 as the gradient.

### Versions

pytorch: 1.11.0

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
`hardshrink` gives wrong gradient for 0 input when `lambd` is 0.
Labels: module: autograd, triaged, module: edge cases

### 🐛 Describe the bug

`hardshrink` gives wrong gradient for 0-valued input when `lambd` is 0.

```python
import torch

def fn(input):
    fn_res = input.hardshrink(lambd=0.0)
    return fn_res

input = torch.tensor([0.], dtype=torch.float64, requires_grad=True)
torch.autograd.gradcheck(fn, (input))
```

```
torch.autograd.gradcheck.GradcheckError: Jacobian mismatch for output 0 with respect to input 0,
numerical:tensor([[1.]], dtype=torch.float64)
analytical:tensor([[0.]], dtype=torch.float64)
```

Based on the definition of `hardshrink`, it should be `f(x) = x` if `lambd=0`. Thus, it's supposed to return 1 as the gradient when input is 0.

### Versions

pytorch: 1.11.0

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
`torch.inverse()` crash in cuda
Labels: triaged, module: linear algebra, module: correctness (silent)

### 🐛 Describe the bug


`tensor.inverse` produce wrong results. The code works fine before, until one day I move `import torch` to the end of imports
Unfortunately, I can't provide a snippet to reproduce the bug. It may be caused by the conficts with other libs.

The result that works correctly before:
![image](https://user-images.githubusercontent.com/9301117/176939973-16920989-b101-4607-9ba9-420f675fce32.png)


After I move `import torch` to the end of imports, I get:
![image](https://user-images.githubusercontent.com/9301117/176938848-47583fbb-1963-4ffc-a9f8-65a830ff0c2f.png)

It works correctly on CPU and `torch.pinverse`. 



### Versions

`torch.__version__`: 1.11.0+cu113

cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano
RPC: Make RRefProxy callable
Labels: oncall: distributed, enhancement, module: rpc

### 🚀 The feature, motivation and pitch

Executing remote callable objects (including `Module`) currently requires explicitly specifying `__call__` in the RPC command.

Consider:

```Python
import os
import torch
from torch.distributed import rpc

os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '29500'
rpc.init_rpc('worker0', world_size=1, rank=0)

class MyModule(torch.nn.Module):
    def forward(self, tensor):
        print(tensor)

mod = rpc.remote(0, MyModule)
t = torch.randn(10)

# Works:
mod.rpc_sync().__call__(t)

# TypeError: 'RRefProxy' object is not callable
mod.rpc_sync()(t)

rpc.shutdown()
```

It would be cleaner if users didn't have to explicitly call double-underscore methods.

Cheers.

### Alternatives

None

### Additional context

None

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501 @jjlilley @mrzzd
Anaconda is not a package manager
Labels: module: docs, triaged

### 📚 The doc issue

The documentation in ["Getting Started"](https://pytorch.org/get-started/locally/#start-locally) states:

> "*Anaconda is our recommended package manager since it installs all dependencies.*"

However, Anaconda is not a package manager, but a distribution of Python that includes the Conda package manager. This misusage confuses users and leads to people unnecessarily installing Anaconda when many users are better off with a Miniforge variant, such as Mambaforge.

### Suggest a potential alternative/fix

Please replace "Anaconda" with "Conda".

cc @svekars @holly1238
Let torch.utils.tensorboard support multiprocessing
Labels: module: multiprocessing, triaged, module: tensorboard

### 🚀 The feature, motivation and pitch

In TensorboardX, [GlobalSummaryWriter](https://github.com/lanpa/tensorboardX/blob/df1944916f3aecd22309217af040f2e705997d9c/tensorboardX/global_writer.py) is implemented. I want to use torch.utils.tensorboard in multiprocessing, but I failed after a simple modification of TensorboardX’s code. I believe this feature is very important and not difficult to implement. Thanks you very much!

### Alternatives

_No response_

### Additional context

_No response_

cc @VitalyFedyunin
`det` will return wrong gradient for `1x1` matrix with 0 value.
Labels: module: autograd, triaged, module: edge cases

### 🐛 Describe the bug

`det` will return wrong gradient for `1x1` matrix with 0 value.

```python
import torch

input = torch.tensor([[0.]], dtype=torch.float64, requires_grad=True)
torch.det(input).backward()
print(input.grad)
# tensor([[0.]], dtype=torch.float64)
```

The correct gradient should be 1. Instead, when the value isn't zero, it will return the correct gradient.

```python
import torch

input = torch.tensor([[0.1]], dtype=torch.float64, requires_grad=True)
torch.det(input).backward()
print(input.grad)
# tensor([[1.]], dtype=torch.float64)
```

### Versions

pytorch: 1.11.0

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
CapabilityBasedPartitioner requires is node supported to only return true for CALLABLE_NODE_OPS but no assertion for this invariant exists
Labels: triaged, module: fx, oncall: pt2

### 🐛 Describe the bug

Steps to reproduce: 
1. Write a OperatorSupport that always returns True
2. Try to run the partitioner on a graph

Expected: an error, or it fuses the entire graph
Actual: the resulting graph is malformed

@SherlockNoMad 

### Versions

master
Unable to use vmap atop torch.distribution functionality
Labels: high priority, triaged, module: functorch

Hello! I'm working on an application that requires computing a neural net's weight Jacobians through a torch.distribution log probability. Minimal example code show below:

```python
import torch
from torch.distributions import Independent, Normal
from functorch import make_functional_with_buffers, jacrev, vmap

def compute_fischer_stateless_model(fmodel, params, buffers, input, target):
    input = input.unsqueeze(0)
    target = target.unsqueeze(0)
    pred = fmodel(params, buffers, input)
    normal = Independent(Normal(loc=pred, scale=torch.ones_like(pred)), reinterpreted_batch_ndims=1)
    log_prob = normal.log_prob(target)
    return log_prob

# Instantiate model, inputs, targets, etc.
fmodel, params, buffers = make_functional_with_buffers(model)
ft_compute_jac = jacrev(compute_fischer_stateless_model, argnums=1)
ft_compute_sample_jac = vmap(ft_compute_jac, in_dims=(None, None, None, 0, 0))
jac = ft_compute_sample_jac(fmodel, params, buffers, inputs, targets)

```


Executing my script returns a `RuntimeError` error of the form: 

*RuntimeError: vmap: It looks like you're either (1) calling .item() on a Tensor or (2) attempting to use a Tensor in some data-dependent control flow or (3) encountering this error in PyTorch internals. For (1): we don't support vmap over calling .item() on a Tensor, please try to rewrite what you're doing with other operations. For (2): If you're doing some control flow instead, we don't support that yet, please shout over at https://github.com/pytorch/functorch/issues/257 . For (3): please file an issue.*

Any help would be appreciated -- thanks in advance for you time!

cc @ezyang @gchanan @zou3519 @Chillee @samdow @soumith @kshitij12345 @janeyx99
Add TorchDynamo as a submodule to Pytorch?
Labels: module: build, triaged

### 🚀 The feature, motivation and pitch

It has been hard to recommend TorchDynamo usage with Pytorch given that TorchDynamo does not have an official release and users often want to use specific versions of Pytorch (Nvidia releases containers on a monthly basis from TOT), however, TorchDynamo requires using Pytorch TOT, currently, and can break when not in sync.  

I wanted to propose adding TorchDynamo as a submodule to Pytorch.

### Alternatives

_No response_

### Additional context

_No response_

cc @malfet @seemethere
pad_sequence and pack_sequence should support length zero tensors
Labels: module: rnn, triaged, enhancement

### 🚀 The feature, motivation and pitch

This situation naturally occurs when training on irregularly sampled times series data with a fixed real-time sliding window, i.e. the model received all observations made within a time interval of say 2 hours.

Since the time series is irregular, it can happen that there are no observations in this time-slice, hence the resulting tensor has a 0-length dimension.

## MWE

```python
import torch
from torch.nn.utils.rnn import pack_sequence, pad_sequence

tensors = [torch.randn(abs(n - 3)) for n in range(6)]
pad_sequence(tensors, batch_first=True, padding_value=float("nan"))
pack_sequence(tensors, enforce_sorted=False)
```


### Alternatives

_No response_

### Additional context

_No response_

cc @zou3519
Overlapping Optimizer.step() with DDP backward
Labels: oncall: distributed, module: optimizer

### 🚀 The feature, motivation and pitch

DDP `all_reduce` parameters' gradients in `buckets`, which means that some parameters would get their finial gradients earlier than others. This exposes an opportunity for `Optimizer` to optimize part of parameters before `backward` finished. So that overlapping part of `Optimizer.step` and DDP `backward` is possible and may led to higher parallelism or even hide some `all_reduce` cost.

### Alternatives

This re-scheduling may be achieved by compilation optimizations like `LazyTensorCore` and `XLA`.

### Additional context

<img width="1160" alt="截屏2022-06-30 下午3 58 35" src="https://user-images.githubusercontent.com/73142299/176624701-e2ec53da-5240-42e9-8ca2-7c5e71f6b62a.png">

As the timeline above, if we can start `Optimizer.step` working on those parameters already has finial gradients, we may save almost all `Optimizer.step`'s cost.

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501 @vincentqb @jbschlosser @albanD
RuntimeError: DataLoader worker (pid 22822) is killed by signal: Aborted. 
Labels: module: dataloader, triaged

### 🐛 Describe the bug

I set num_workers=4 in dataloader,the training process can run normally, and an error is reported in the verification phase。

### Versions

torch                     1.11.0
torchvision               0.12.0

cc @SsnL @VitalyFedyunin @ejguan @NivekT
Semi-reproducible random torch.baddbmm NaNs
Labels: needs reproduction, triaged, module: NaNs and Infs

### 🐛 Describe the bug

The following code snippet appears to cause `torch.baddbmm` to randomly generate NaNs when run *on CPU*

```
for i in range(10000):
    out = torch.baddbmm(
        torch.zeros([1, 1, 1], dtype=torch.float32),
        torch.FloatTensor([[[1]]]),
        torch.FloatTensor([[[1]]]),
        beta=0,
    )
    assert not torch.isnan(out).any(), i
# AssertionError: 9886
# (or some other number)
```

Despite running the same calculation each time, it often fails not on the first try, but many tries in.

(Sometimes I need to run the loop several times before it actually encounters a NaN, which seems odd to me.)

I've tried this on two different hardware setups and encountered the same issue.

Hope I'm not just doing something silly!

### Versions

Collecting environment information...
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2
Libc version: glibc-2.27

Python version: 3.9.11 (main, Mar 29 2022, 19:08:29)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.18.0-305.28.1.el8_4.x86_64-x86_64-with-glibc2.27
Is CUDA available: True
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: Quadro RTX 8000
Nvidia driver version: 470.57.02
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.2
[pip3] torch==1.11.0
[pip3] torchaudio==0.11.0
[pip3] torchvision==0.12.0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.2.89              hfd86e86_1
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py39h7f8727e_0
[conda] mkl_fft                   1.3.1            py39hd3c417c_0
[conda] mkl_random                1.2.2            py39h51133e4_0
[conda] numpy                     1.21.2           py39h20f2e39_0
[conda] numpy-base                1.21.2           py39h79a1101_0
[conda] pytorch                   1.11.0          py3.9_cuda10.2_cudnn7.6.5_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.11.0               py39_cu102    pytorch
[conda] torchvision               0.12.0               py39_cu102    pytorch