gopikrsmscs/torch-issues huggingface dataset:
`torch.ops.aten.find` inconsistent with `str.find`
Labels: module: cpp, triaged, module: sorting and selection

### 🐛 Describe the bug

For empty target strings `aten::find` will always return the value of the starting position in the string, even when the given range is invalid.

```
>>> import torch
>>> "example".find("", 100, 0)
-1
>>> torch.ops.aten.find("example", "", 100, 0)
100
```

As far as I can tell the cause of the discrepancy is found in `register_prim_ops.cpp` on [line 1522](https://github.com/pytorch/pytorch/blob/b4e491798c0679ab2e61f36a511484d7b8ecf8d3/torch/csrc/jit/runtime/register_prim_ops.cpp#L1522) where if the target string is empty, it will always search for the target even if the search range is invalid in some way.


Additionally, this can lead to crashes for certain inputs as shown below.
```
>>> import torch
>>> torch.ops.aten.find("example", "a", 100, 200)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/quinn/torch-mlir/mlir_venv/lib/python3.10/site-packages/torch/_ops.py", line 148, in __call__
    return self._op(*args, **kwargs or {})
IndexError: basic_string::substr: __pos (which is 100) > this->size() (which is 7)

```

### Versions

Collecting environment information...
PyTorch version: 1.13.0.dev20220623+cpu
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Arch Linux (x86_64)
GCC version: (GCC) 12.1.0
Clang version: 13.0.1
CMake version: version 3.22.5
Libc version: glibc-2.35

Python version: 3.10.5 (main, Jun  6 2022, 18:49:26) [GCC 12.1.0] (64-bit runtime)
Python platform: Linux-5.18.5-arch1-1-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2070 SUPER
Nvidia driver version: 515.48.07
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.0
[pip3] torch==1.13.0.dev20220623+cpu
[pip3] torch-mlir==20220620.509
[pip3] torchvision==0.14.0.dev20220623+cpu
[conda] Could not collect


cc @jbschlosser
2-dimensional arange
Labels: triaged, enhancement, module: nestedtensor, module: tensor creation

### 🚀 The feature, motivation and pitch

When dealing with batches of variable-length sequences, we often track the start indices, length of sequences, and end indices.

```python
maximum_sequence_length = 3
number_of_sequences = 3
data = torch.random(maximum_sequence_length * number_of_sequences, feature)
sequence_length = torch.tensor([1, 3, 2])
start_idx = torch.tensor([0, 3, 6])
end_idx = torch.tensor([1, 6, 8])
```

When we want to index our `data` tensor, and select only the non-padded/valid data, we need to generate some indices that go from `start_idx` to `end_idx`. Currently, there is no vectorized way to do this, and so we do the following
```python
# Sad, non-vectorized loop :(
idx = torch.cat([torch.arange(start_idx[b], end_idx[b]) for b in range(batch)])
print(idx)
# Parenthesis added for readability
>>> [(0, ), (3, 4, 5), (6, 7)]
# With the indices, we can now index our data, free from padding
data[idx]
```

This has applications to filtering and message-passing GNNs. Over many batches, this for-loop concatenation overhead tends to add up. It would be nice to have something vectorized like

```python
torch.arange2d(start_idx: torch.Tensor, end_idx: torch.Tensor) -> torch.Tensor:
   """Given a vector of s_i \in starts and e_i \in ends, 
   returns [s_1, s_1 + 1, ... e_1, s_2, s_2 + 1, ... e_2, ... e_n]"""

```

### Alternatives

`repeat_interleave` can be used in some edge cases, but does not work in general. Slicing using a tensor would also solve this, i.e.
```python
data[start_idx: end_idx]
```
but this is not currently supported either.

### Additional context

cc @cpuhrsch @gchanan @mruberry
`bmm_sparse_cuda` kernel for `bfloat16`
Labels: module: sparse, module: cuda, triaged, module: bfloat16

### 🚀 The feature, motivation and pitch

The kernel would be useful for sparse attention with `bfloat16` weights

### Alternatives

```python
torch.bmm(
    sparse_matrix.float(),
    dense_matrix.float()
).to(torch.bfloat16)
```

### Additional context

_No response_

cc @nikitaved @pearu @cpuhrsch @amjames @ngimel
Nonliner conjugate gradient optimizer + Hager-Zhang line search
Labels: feature, module: optimizer, triaged, needs research

### 🚀 The feature, motivation and pitch

The nonlinear conjugate gradient (CG) method is a good alternative to the L-BFGS optimizer. Features of nonlinear CG:

1. It theoretically converges faster than gradient-descent methods (including most derived variants), and the memory footprint is just slightly higher than that of gradient-descent methods.
2. It is theoretically able to converge to machine precision, which may be important for regression-type problems when using single-precision floats.
3. Though it converges slower than L-BFGS does, it needs much less memory.
4. The specific CG method from Hager and Zhang automatically falls back to the vanilla gradient descent method (but with an auto-determined learning rate) if the conjugate direction does not perform better.
5. No need for configuring the learning rate. The step size (i.e., the learning rate) on the conjugate/gradient direction is determined by an inexact line search.
6. Hager and Zhang also proposed a companion inexact line-search algorithm for the proposed CG method.

I already have a working implementation of Hager and Zhang's nonlinear CG as a derived `torch.optim.Optimizer` class, including the inexact line search algorithm. I have been using this implementation for my own research. However, in terms of code tests, I only have two simple cases that can be used for code tests. But at least if this feature request is approved, we don't have to start from scratch.

This is directly related to #1359.

**Reference**
[1] W. W. Hager and H. Zhang, “Algorithm 851: CG_DESCENT, a conjugate gradient method with guaranteed descent,” ACM Trans. Math. Softw., vol. 32, no. 1, pp. 113–137, Mar. 2006, doi: 10.1145/1132973.1132979.

### Alternatives

_No response_

### Additional context

There are some issues mentioning implementing CG, like #53441 and #17902. However, I think they are more about linear CG, which is a special case of nonlinear CG. And I feel like linear and nonlinear CG solvers usually are implemented separately. So I don't think this feature request will resolve the issues requesting linear CG.

cc @vincentqb @jbschlosser @albanD
NVFuser should extend caching to remove necessity for PrimTorch's executor to Provide Tensor Contiguity Info
Labels: triaged, module: nvfuser, module: primTorch

### 🚀 The feature, motivation and pitch

Currently, PrimTorch's executor provides contiguity and tensor rank size information in the `FusionDefinition` for each fusion group sent to NVFuser.  This should be removed in favor of NVFuser receiving this information at runtime and caching the graph appropriately.

Therefore, the only information NVFuser should be provided in the Tensor Definition is the number of ranks of each Tensor. @mruberry does this match your understanding?

### Alternatives

_No response_

### Additional context

CC @csarofeen @jjsjann123 @ngimel @IvanYashchuk

cc @ezyang @mruberry @ngimel
Allow parameterization of Layouts
Labels: module: sparse, feature, triaged, module: python frontend

### 🚀 The feature, motivation and pitch

Some layouts can be arbitrarily parameterized. Block sparse layouts, for example require block sizes to completely specify the layout. 

If we were to implement a way of parameterizing such layouts, then passing the layout in any context would be fully specified without the need for additional arguments. 



### Alternatives

- All interfaces accepting a layout would strictly support a subset of layouts.
   - This would make the adoption of any "parameterized" layout more difficult.
- All interfaces accepting a `layout` would have to accept any additional parameters required to perform any possible conversions.
   - These interfaces would need to be updated when new layouts are introduced.
   - Specification of a parameter with a `layout` option that it does not pertain to would need to be detected and handled (Warning/Error), or ignored (potentially problematic)

### Additional context

A key example (see #80542). 

If the `Tensor.to` method were to support layout conversions a `blocksize` parameter would need to be accepted, but ignored for most layouts. If the `blocksize` was instead a parameter attached to the layout itself that interface (and all others accepting a `layout`) could safely be ignorant of the additional details for those layouts. The parameter need only be accessed at the point where the real format conversion function gets involved. 

```python
x = torch.Tensor(...)
y = x.to(layout=torch.sparse_bsr(blocksize=(...)))
z = x.to(layout=torch.sparse_coo)
```
vs

```python
x = torch.Tensor(...)
y = x.to(layout=torch.sparse_bsr, blocksize=(...))
z = x.to(layout=torch.sparse_coo, blocksize=(...))
```

The handling of the last case in above would be tedious, and get worse as more formats introduce additional specialty parameters. 

 

cc @nikitaved @pearu @cpuhrsch @amjames
[Prims+NVFuser] Prims with missing NVFuser ops
Labels: triaged, module: nvfuser, module: primTorch

These prims are missing NVFuser implementations. They were found by running https://github.com/pytorch/pytorch/compare/bahuang/nvfuser_e2e with additional decompositions.

- [ ] cat
- [ ] maximum
- [ ] transpose

cc @ezyang @mruberry @ngimel
DDP find_unused_parameters=True does not work for Sparse gradients
Labels: oncall: distributed

### 🐛 Describe the bug

If you have an unused parameter that is a sparse embedding table like below

```
class Example(nn.Module):
    def __init__(self): 
        self.a = nn.Embedding(10, 32, sparse=True)
        self.b = nn.Embedding(10, 32, sparse=True)

    def forward(self, x):
        return self.a(x)
```

and in main, you are using `DDP(model, find_unused_parameters=True)`
then the code above fails with 

```
  File "/home/ec2-user/anaconda3/envs/elanmark/lib/python3.7/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Expected sparse gradient to be defined.
```

Switching `self.b` to `sparse=False` fixes the issue. However, in my actual case, each embedding is required under different cases. 

It looks like `Reducer::mark_variable_ready_sparse` does not have the same checks as `Reducer::mark_variable_ready_dense`. Instead it just checks `grad.defined()`.



### Versions

Not available. Ask if specific relevant data needed.

torch==1.7.0+cu110

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
[bug] libtorch bug in nn::MultiheadAttention and nn::Transformer
Labels: module: cpp, module: nn, triaged, oncall: transformer/mha, module: correctness (silent)

### 🐛 Describe the bug

the attn_mask does not work in nn::MultiheadAttention
~~~
#include <torch/torch.h>
namespace nn = torch::nn;

// seq_length x batch_size x feature_size
torch::Tensor x = torch::randn({3,1,4});

torch::Tensor attn_mask = nn::TransformerImpl::generate_square_subsequent_mask(3);
torch::Tensor attn_mask_bool = attn_mask.to(torch::kBool);

nn::MultiheadAttention multihead_attention(4,1);
std::tuple<torch::Tensor, torch::Tensor> output_with_attn_mask = multihead_attention ->forward(x,x,x,{},true,attn_mask_bool);

torch::Tensor attn_output, attn_output_weights;
std::tie(attn_output, attn_output_weights) = output_with_attn_mask; //unpacking tuple into variables

std::cout << attn_output_weights << std::endl;
~~~

~~~
attn_mask:
/*
 0  1  1
 0  0  1
 0  0  0
[ CPUBoolType{3,3} ]
*/


attn_output_weights:

(1,.,.) = 
  0.1918  0.5302  0.2780
  0.2074  0.1919  0.6007
  0.1948  0.5092  0.2960
[ CPUFloatType{1,3,3} ]
~~~

* 1 the attn_mask does not affect the attn_output_weights.
* 2  the API in libtorch for MultiheadAttention and Transformer is not comparable with that of in pytorch. e.g. libtorch api have no batch_first param. 
* 3 more detail comparison see https://github.com/walkacross/pytorch-libtorch-API-translation/tree/main/translation/torch.nn/transformer_layers

### Versions

libtorch: 1.11.0+cu113

cc @jbschlosser @albanD @mruberry @walterddr @kshitij12345 @saketh-are @bhosmer @cpuhrsch @erichan1
Negative values still produced by torch.nn.functional.kl_div
Labels: high priority, module: nn, triaged

### 🐛 Describe the bug

## 🐛 Bug
Despite a fix in https://github.com/pytorch/pytorch/issues/32520, kl_div in torch.nn.functional still outputs negative values. 
## To Reproduce
Say we have outputs with the same values:
```python
a = torch.tensor([[1,2,3], [5.0, 5.0, 5.0]])
b = torch.tensor([[1,2,3], [5.0, 5.0, 5.0]])
torch.nn.functional.kl_div(torch.nn.functional.log_softmax(a, 1), torch.nn.functional.softmax(b, 1), reduction="none")
tensor([[ 0.0000e+00,  0.0000e+00, -1.9826e-08],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00]])
```
Or, say we have outputs with arbitrarily-chosen different values. https://github.com/pytorch/pytorch/issues/32520 attempts to resolve negative outputs by allowing for a target in log-space, however this has no effect here:
```python
a = torch.tensor([[1,2,3], [5.0, 5.0, 5.0]])
b = torch.tensor([[8,10,6], [5.0, 5.0, 5.0]])
torch.nn.functional.kl_div(torch.nn.functional.log_softmax(a, 1), torch.nn.functional.log_softmax(b, 1), reduction="none", log_target=True)
tensor([[ 0.0310,  1.0962, -0.0593],
        [ 0.0000,  0.0000,  0.0000]])
```
or with a log-space target:
```python
a = torch.tensor([[1,2,3], [5.0, 5.0, 5.0]])
b = torch.tensor([[8,10,6], [5.0, 5.0, 5.0]])
torch.nn.functional.kl_div(torch.nn.functional.log_softmax(a, 1), torch.nn.functional.softmax(b, 1), reduction="none")
tensor([[ 0.0310,  1.0962, -0.0593],
        [ 0.0000,  0.0000,  0.0000]])
```
## Expected Behavior
Outputs with values >= 0.

### Versions

pytorch==1.12.0 (same behavior in 1.8.2 LTS)
torchvision==0.13.0
cudatoolkit=11.3.1

cc @ezyang @gchanan @zou3519 @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
Revisit OpInfo samples for nn.functional.max_poolNd
Labels: module: nn, triaged, actionable, module: pooling, module: testing

As mentioned [here](https://github.com/pytorch/pytorch/issues/80314#issuecomment-1168426818), the `nn.functional.max_poolNd` OpInfo entries test input combinations extensively, resulting in long test run times. From the times reported [here](https://observablehq.com/d/b95f7b67261147ba), they appear to be within the top 15 longest running tests:

```
TestOps.test_dtypes_nn_functional_max_pool1d_cuda    45.902 sec
TestOps.test_dtypes_nn_functional_max_pool2d_cuda    61.708 sec
TestOps.test_dtypes_nn_functional_max_pool3d_cuda    27.613 sec
```

We should revisit these OpInfo entries and consider reducing the number of samples to bring down these test times.

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
CMake Error: File /opt/pytorch/build_variables.bzl does not exist.
Labels: triaged, module: regression, module: docker

### 🐛 Describe the bug

Following #80212, the docker build command failed again today on a freshly cloned repo:

```sh
$ git clone --recursive https://github.com/pytorch/pytorch
$ cd pytorch
$ DOCKER_BUILDKIT=1 docker build . > log.txt 2>&1
```

[log.txt](https://gist.github.com/iago-lito/61358872444050cfbbff05d18ccfea00)

Maybe related:  #78542 ?

### Versions

```sh
$ python collect_env.py
Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Arch Linux (x86_64)
GCC version: (GCC) 12.1.0
Clang version: 13.0.1
CMake version: version 3.23.2
Libc version: glibc-2.35

Python version: 3.10.5 (main, Jun  6 2022, 18:49:26) [GCC 12.1.0] (64-bit runtime)
Python platform: Linux-5.18.7-arch1-1-x86_64-with-glibc2.35
Is CUDA available: N/A
CUDA runtime version: 11.7.64
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 960M
Nvidia driver version: 515.48.07
cuDNN version: Probably one of the following:
/usr/lib/libcudnn.so.8.4.0
/usr/lib/libcudnn_adv_infer.so.8.4.0
/usr/lib/libcudnn_adv_train.so.8.4.0
/usr/lib/libcudnn_cnn_infer.so.8.4.0
/usr/lib/libcudnn_cnn_train.so.8.4.0
/usr/lib/libcudnn_ops_infer.so.8.4.0
/usr/lib/libcudnn_ops_train.so.8.4.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

Versions of relevant libraries:
[pip3] numpy==1.22.4
[pip3] torch==1.11.0
[pip3] torchvision==0.12.0a0+1665451
[conda] Could not collect
```
Torch fx print line number of each node
Labels: triaged, module: fx

### 🚀 The feature &motivation

It can be very helpful if each node in the graph, generated by symbolic_trace, would include the full path and line number of this operator, similarly to torch.onnx.export verbose output. For example:
%180 : Float(1, 512, 7, 7, strides=[25088, 49, 7, 1], requires_grad=1, device=cpu) = onnx::**Add**(%240, %243) # **C:\ProgramData\Miniconda3\lib\site-packages\torchvision\models\resnet.py:80:0**
One of possible use cases for such feature is to locate specific functions or attribute queries such as: x.size(), x.reshape 

Best regards


### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @SherlockNoMad
[DDP] output_device argument appears completely unused
Labels: oncall: distributed, triaged, better-engineering, module: ddp

### 🐛 Describe the bug

In current DDP implementation, the only reference to `self.output_device` is in DDP logging to log construction data: https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L760.

It does not actually appear used anywhere, possibly we should mark this argument as deprecated / unused.



### Versions

main

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
[c10d] Async object-based collectives
Labels: oncall: distributed, triaged, module: c10d

### 🚀 The feature, motivation and pitch

Feature request for supporting an async mode for object-based collectives was raised in https://discuss.pytorch.org/t/how-can-i-receive-the-outputs-from-dist-all-gather-object-asynchronously/155004. Creating this issue to track support for all object-based collectives to be async.

### Alternatives

_No response_

### Additional context

_No response_

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
Tracker: Slow gradcheck failures possibly indicating incorrect gradients
Labels: module: autograd, triaged, actionable

### 🐛 Describe the bug

For a very long time, most tests are being [skipped](https://ossci-raw-job-status.s3.amazonaws.com/log/7032407678) by slow gradcheck (see https://github.com/pytorch/pytorch/issues/80314#issuecomment-1168074007):
At least the following are failing in slow gradcheck (There are probably a lot more, but we can't see those right now because the test is timing out): 

- [ ] test_fn_fwgrad_bwgrad___rmod___cuda_float64 (non-differentiable points encountered when sample inputs are large)
- [ ] test_fn_fwgrad_bwgrad__masked_prod_cuda_complex128
- [ ] test_fn_fwgrad_bwgrad__masked_prod_cuda_float64
- [ ] test_fn_fwgrad_bwgrad_cat_cuda_complex128
- [ ] test_fn_fwgrad_bwgrad_cholesky_inverse_cuda_complex128
- [ ] test_fn_fwgrad_bwgrad_copysign_cuda_float64
- [ ] test_fn_fwgrad_bwgrad_fft_*
- [ ] test_fn_fwgrad_bwgrad_float_power_cuda_complex128
- [ ] test_fn_fwgrad_bwgrad_float_power_cuda_float64
- [ ] test_fn_fwgrad_bwgrad_fmod_cuda_float64 (probably same issue as `__fmod__`)
- [ ] test_fn_fwgrad_bwgrad_linalg_householder_product_cuda_complex128
- [ ] test_fn_fwgrad_bwgrad_linalg_lu_cuda_float64
- [ ] test_fn_fwgrad_bwgrad_linalg_lu_factor_ex_cuda_float64
- [ ] test_fn_fwgrad_bwgrad_linalg_matrix_power_cuda_complex128
- [ ] test_fn_fwgrad_bwgrad_remainder_cuda_float64
- [ ] test_fn_fwgrad_bwgrad_repeat_cuda_complex128
- [ ] test_fn_fwgrad_bwgrad_prod_cuda_complex128
- [ ] test_fn_fwgrad_bwgrad_slice_scatter_cuda_float64
- [ ] test_fn_fwgrad_bwgrad_tile_cuda_complex128
- [ ] test_fn_fwgrad_bwgrad_pow_cuda_float64
- [ ] test_fn_fwgrad_bwgrad_pow_cuda_complex128
- [ ] test_fn_fwgrad_bwgrad_zero__cuda_complex128
- [ ] test_fn_gradgrad_linalg_lu_factor_cuda_float64
- [ ] test_fn_grad_div_trunc_rounding_cuda_float64
- [ ] test_fn_grad_div_floor_rounding_cuda_float64

See https://ossci-raw-job-status.s3.amazonaws.com/log/7034710626

The failures don't necessarily mean the formulas are wrong, `__rmod__` (and possibly also `fmod`) for example only fails when perturbation of eps will crosses a point of non-differentiability, and when sample inputs are large enough that happens with high probability.


### Versions

main

cc @ezyang @gchanan @zou3519 @albanD @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
Support for learnable p Values in LPPOOL like Pool
Labels: module: nn, triaged, needs research

### 🚀 The feature, motivation and pitch

Make a LPPOOL layer that can learn p Values. The value for p = inf is max pooling and p = 1 is Average pooling. Different from LPPOOL where p = 1 is sum pooling.
The new LPPOOL has a p value for each feature map. So for each feature map a p Value can be learned.


### Alternatives

_No response_

### Additional context

I already tested this layer in my repo https://github.com/joerg-de/AdaptiveLearnableLPPool2d-test where I replaced the pooling layer of some well known Networks.
For shufflenet_v2_x0_5 Acc1 was around 1.9 % better and Acc5 was around 1.5 % better.
With 1024 additional Weights.

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
Modify _add_docstr to also set the correct module for the APIs
Labels: triaged, better-engineering, actionable, module: python frontend

Currently, many public APIs don't have their module set correctly to meet our requirements for public API (https://github.com/pytorch/pytorch/wiki/Public-API-definition-and-documentation). All the ops in `torch.linalg`, `torch.special`, `torch.sparse` etc currently use `_add_docstr` to correctly set the doc for all the functions in these modules. The proposal here is to modify `_add_docstr` to set the correct module. 

Once this is done, we can get rid of all the respective entries in this allowlist: https://github.com/pytorch/pytorch/blob/d67ce755ad1439b03d4abcc2d40496b20cfd570e/test/allowlist_for_publicAPI.json#L1786-L1827

cc. @albanD
[BE] Update ProcessGroupWrapper tests to test other collective message
Labels: oncall: distributed, triaged, better-engineering, module: c10d

### 🚀 The feature, motivation and pitch

Context of changes are in https://github.com/pytorch/pytorch/pull/79901.

Since the updated ProcessGroupWrapper message when `TORCH_DISTRIBUTED_DEBUG=DETAIL` is set will now print the op type, tensor shape, device type, etc. of the offending collective, we should also update the tests to check for these error messages. The tests in test_pg_wrapper.py currently only validate the error message for the current rank, but does not validate anything about the other rank's collective.

### Alternatives

_No response_

### Additional context

_No response_

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
Distributed Store `get` doesn't work well with `add`
Labels: high priority, triage review, oncall: distributed, module: docs, triaged, module: c10d

### 🐛 Describe the bug

While I was working on sharing distributed seed for DataLoader, internal users encountered a problem that the distributed process will hang when trying to `get` an integer counter from the distributed store (`PrefixStore`).

While I dived deeper on this problem, I found a comment here:
https://github.com/pytorch/pytorch/blob/590d3e5774110e4657dcaa6acdb387ef69e41b47/torch/distributed/distributed_c10d.py#L242-L246

Do we have a plan to remove the legacy store to make sure `get` is working with `add`? If not, should we add some notes to recommend users to use `add(key, 0)` to retrieve the value saved by `store.add`?
https://pytorch.org/docs/stable/distributed.html#torch.distributed.Store.add

### Versions

fblearner

cc @ezyang @gchanan @zou3519 @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501 @svekars @holly1238
DISABLED test_lobpcg (__main__.TestAutograd)
Labels: module: autograd, triaged, skipped

Platforms: linux

This test was disabled because it is failing on master ([recent examples](http://torch-ci.com/failure/test_lobpcg%2C%20TestAutograd)).

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
Illegal Memory Access from nonzero method when Tensor is Too Large
Labels: module: dependency bug, module: crash, triaged, module: edge cases

### 🐛 Describe the bug

There is a known issue that the nonzero method fails if the input tensor exceeds INT_MAX (#51871), raising a RuntimeError. However, the nonzero method also fails for tensors that are very large but slightly smaller than INT_MAX, causing an irrecoverable illegal memory access.

To Reproduce:

```
import torch
M = 2121269248
torch.ones(M, device=’cuda’).nonzero()
torch.cuda.synchronize()
```

Decreasing M by one eliminates the error.

Expected Behavior:

The method either returns the indices of non-zero elements, or raises a RuntimeError that can be caught and recovered from.

Additional Context:

Possibly related to #51872?

### Versions

PyTorch Version: 1.11.0+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Red Hat Enterprise Linux Server release 7.9 (Maipo) (x86_64)
GCC version: (GCC) 8.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.17

Python version: 3.7.3
Python platform: Linux-3.10.0-1160.66.1.el7.x86_64-x86_64-with-redhat-7.9-Maipo
Is CUDA available: True
CUDA runtime version: 11.3.58
GPU models and configuration:
GPU 0: Tesla V100-SXM2-32GB
GPU 1: Tesla V100-SXM2-32GB
GPU 2: Tesla V100-SXM2-32GB
GPU 3: Tesla V100-SXM2-32GB

Nvidia driver version: 470.52.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] functorch==0.1.1
[pip3] numpy==1.21.6
[pip3] torch==1.11.0+cu113
[pip3] torchtext==0.12.0
[pip3] torchvision==0.12.0+cu113
[conda] Could not collect
Add support for torch.nn.quantized.modules.FloatFunctional
Labels: module: nn, triaged, enhancement, oncall: pt2

cc @albanD @mruberry @jbschlosser @walterddr @saketh-are @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel
Need "valid" and "same" padding mode for convTranspose2d
Labels: feature, module: nn, module: convolution, triaged, module: padding

### 🚀 The feature, motivation and pitch

The "valid" and "same" padding mode have been added for conv2D and it was a heavily requested feature. I request to add the similar feature for convTranspose2D (this feature is already present in tf)

### Alternatives

_No response_

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
Sort tensors inplace
Labels: feature, triaged, module: sorting and selection

### 🚀 The feature, motivation and pitch

Hello Everyone,

I encounter Cuda-out-of-memory error during the sorting operation due to a huge tensor.
The tensor shape is [40000, 2048] thus I think If we can perform the sorting operation in an inplace manner it would be very nice feature to become more efficient.
Also, is there any workaround to make it inplace by myself without changing the torch version, i.e., 1.5?

Best regards.

### Alternatives

_No response_

### Additional context

_No response_
Cudnn batch norm kernel (batchnorm_bwtr_nhwc_semiPersist) gets blocked by overlapping NCCL all_reduce calls
Labels: module: dependency bug, module: cudnn, triaged, module: nccl, module: memory format

### 🐛 Describe the bug

When cudnn + channels_last format is enabled, under distributed DDP, the backward pass of cudnn batchnorm backward can be stalled by the overlapped nccl all_reduce call. 

Sorry I don't have a code snippet for repro, but here are some screenshots of torch profiler

Blocking batch_norm_backward
![image](https://user-images.githubusercontent.com/1190730/175750442-13d30ee6-5c8b-4f59-bbcd-95027448af5b.png)

Neither the aten native implementation nor the NCHW version of cudnn kernel `bn_bw_1C11_singleread_specialized` shows this behavior. 

Technically this seems like a cuDNN bug, but I don't know where/how to file a cudnn bug, so filing it here instead. This issue is also reported on https://github.com/NVIDIA/nccl/issues/338 https://github.com/NVIDIA/nccl/issues/661


### Versions

```
Collecting environment information...
PyTorch version: 1.11.0a0+gitbc2c6ed
Is debug build: False
CUDA used to build PyTorch: 11.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: 14.0.5 (https://github.com/conda-forge/clangdev-feedstock 0f793d7b2ad6a3c631eb76429f4e4def37c890dd)
CMake version: version 3.19.1
Libc version: glibc-2.31

Python version: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-5.4.0-64-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.4.120
GPU models and configuration: GPU 0: NVIDIA A100-SXM-80GB
Nvidia driver version: 470.57.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.19.5
[pip3] pytorch3d==0.6.1
[pip3] torch==1.11.0+bc2c6ed.cuda114.cudnn841.se02.ap
[pip3] torch-scatter==2.0.8
[pip3] torch-tb-profiler==0.4.0
[pip3] torchfile==0.1.0
[pip3] torchvision==0.9.0a0+8fb5838
[conda] magma-cuda111             2.5.2                         1    pytorch
[conda] mkl                       2020.4             h726a3e6_304    conda-forge
[conda] mkl-include               2022.1.0           h84fe81f_915    conda-forge
[conda] numpy                     1.19.5           py38h8246c76_3    conda-forge
[conda] pytorch3d                 0.6.1                    pypi_0    pypi
[conda] torch                     1.11.0+bc2c6ed.cuda114.cudnn841.se02.ap          pypi_0    pypi
[conda] torch-scatter             2.0.8                    pypi_0    pypi
[conda] torch-tb-profiler         0.4.0                    pypi_0    pypi
[conda] torchfile                 0.1.0                    pypi_0    pypi
[conda] torchvision               0.9.0a0+8fb5838          pypi_0    pypi
```

cc @csarofeen @ptrblck @xwang233 @VitalyFedyunin @jamesr66a
[complex] dropout and it's variants should support complex tensors
Labels: feature, module: nn, triaged, module: complex

### 🐛 Describe the bug

```python
import torch
torch.nn.functional.dropout(torch.randn(3, 3, dtype=torch.cfloat))
```

Output
```
RuntimeError: "bernoulli_scalar_cpu_" not implemented for 'ComplexFloat'
```

List of Dropout Modules:
* [ ] Dropout
* [ ] Dropout2d
* [ ] Dropout3d
* [ ] AlphaDropout
* [ ] FeatureAlphaDropout


### Versions

master

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are @ezyang @anjali411 @dylanbespalko @Lezcano @nikitaved
Write some torch.distributed.nn.* tests for the new dispatcher passable ops
Labels: oncall: distributed, triaged

Write some torch.distributed.nn.* tests for the new dispatcher passable ops. This is a follow up on #79669.

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
Change c10d APIs in ProcessGroup to accept const std::vector<at::Tensor>&
Labels: oncall: distributed, triaged

Change c10d APIs in ProcessGroup to accept `const std::vector<at::Tensor>&` instead of `std::vector<at::Tensor>&`.

This is a follow up on #79669.

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
test_conv_backend tests OOMing in 10.2 slow_gradcheck CI
Labels: module: nn, module: ci, module: convolution, triaged

I have NOT disabled this test yet because it may just be transient. To disable the test, add DISABLED at the front of the issue title.

Platforms: linux

This test is failing on periodic jobs due to running out of memory ([recent examples](http://torch-ci.com/failure/test_conv_backend_cudnn3d_transposed_has_bias_False_strided_False_contiguous_True_cuda%2C%20TestNNDeviceTypeCUDA)).

https://hud.pytorch.org/pytorch/pytorch/commit/bab1ea8592f7308cded5d146e2d921ed2a6702dc 
```
2022-06-24T07:26:25.5421038Z RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 7.44 GiB total capacity; 6.88 GiB already allocated; 20.06 MiB free; 6.88 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```



cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are @seemethere @malfet @pytorch/pytorch-dev-infra
[Prims+NVFuser] Supports 0-sized inputs
Labels: triaged, module: nvfuser, module: primTorch

A potential fix for this issues is to add a guard to ensure inputs to nvFuser kernels are non-empty tensor. Fallback to eager for empty tensor. 


Invoked with: <torch._C._nvfuser.FusionDefinition object at 0x7f144038b430>, <class 'RuntimeError'>, RuntimeError('sizes[i] > 0 INTERNAL ASSERT FAILED at "/fsx/users/bahuang/repos/pytorch_fsx/torch/csrc/jit/codegen/cuda/python_frontend/python_bindings.cpp":225, **please report a bug to PyTorch. Size of 0 is not supported in nvFuser. Expected size > 0.**'), <traceback object at 0x7f1440767500>


cc @ezyang @mruberry @ngimel
[Prims+NVFuser] Aten2Prim refs tracking items
Labels: triaged, module: nvfuser, module: primTorch

- [ ] aten.div https://github.com/pytorch/pytorch/pull/77936 
- [ ] aten.amax
- [ ] aten.view
- [ ] aten.new_zero
- [ ] aten.conj_phsycial https://github.com/pytorch/pytorch/pull/81014
- [ ] aten.type_as
- [ ] aten.rand_like
- [ ] aten.var


cc @ezyang @mruberry @ngimel
Support tensor subclasses as `UninitializedParameter`s
Labels: module: nn, triaged, enhancement, module: lazy, tensor subclass

Tensor subclass support as `Parameter`s within modules was added in #73459. A notable gap is support for lazy modules, which utilize `UninitializedParameter`s. We need a way to specify that the `UninitializedParameter` should become a particular subtype when materialized into a `Parameter`.

https://github.com/pytorch/pytorch/blob/35268bdc2a6f0be613b9fa0b5e3da6ae68ece67f/torch/nn/parameter.py#L92-L110

We can change the above, but this will notably require lazy modules (implementing `LazyModuleMixin`) to change slightly. Don't see any way around this change.

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
OpInfos for torch.ops.aten operations
Labels: feature, module: tests, triaged

### 🚀 The feature, motivation and pitch

We should have OpInfos for torch.ops.aten operations that are not covered by the forward pass of our existing torch.* based OpInfos. For example: at::convolution_backward is not directly invoked by the OpInfo for torch.nn.functional.conv2d, so it falls into this category.

## Motivation

Today, to test coverage for at::convolution_backward, someone must write a "forward + backward" test for their OpInfo.
- Performance: This is inefficient because the backward pass (convolution_backward) is generally 2x more expensive than the forward pass
- DRY: You start seeing subsystems (e.g. AOTAutograd, functorch vmap) having the same logic to run forward+backward tests for their OpInfos. They wouldn't all need to duplicate this logic if the backward operations were already a part of OpInfos.
- Coverage: OpInfo samples may not cover all of the aten operations that are involved in a particular operation. This is because when one writes an OpInfo, they look at the public python PyTorch API and try to come up with some test cases, not the ATen implementation of the operator.
- API mismatch: torch.ops.aten.index_put_ and Tensor.index_put_ actually accept different inputs (torch.ops.aten.index_put_ is more general and used in more situation, but not all of it is tested via OpInfo)

Having aten operators in torch.ops.aten will make it easier for transform and backend writers to gain confidence in their implementations. 

### Alternatives

n/a

### Additional context

_No response_

cc @mruberry
F.binary_cross_entropy_with_logits unexpected behaviour
Labels: module: nn, module: loss, triaged

I wanted to see if I put a non-binary input into `F. binary_cross_entropy_with_logits` if it will fail. Instead it gives me an answer without failing which is odd, but I can't figure out where the numbers are coming from. See example below (3rd row is confusing).

Two questions regarding below:
1. Why is this not chucking an error because of the 7 in `targets`
2. Why do I have to manually convert targets to `float` when `F.cross_entropy` expects a `LongTensor` and not `FloatTensor`.

```python
import torch
import torch.nn.functional as F

torch.manual_seed(42)

logits = torch.randn((5, 1))
targets = torch.LongTensor([0, 1, 7, 1, 0])[:, None]
log_prob = torch.cat([torch.log1p(-torch.sigmoid(logits)), F.logsigmoid(logits)], dim=-1)
print(F.binary_cross_entropy_with_logits(logits, targets_multilabel.float(), reduction="none"))
print(-log_prob)
"""
Output:
tensor([[ 0.8756],
        [ 0.6308],
        [-0.8240],
        [ 0.5846],
        [ 0.2817]])
tensor([[0.8756, 0.5389],
        [0.7596, 0.6308],
        [0.8172, 0.5828],
        [0.8149, 0.5846],
        [0.2817, 1.4045]])
"""
```



F.binary_cross_entropy_with_logits(logits, targets_multilabel.float(), reduction="none")


### Versions

Got this error, but my pytorch version is `1.11.0` and running on python 3.9
```
Collecting environment information...
Traceback (most recent call last):
  File "collect_env.py", line 492, in <module>
    main()
  File "collect_env.py", line 475, in main
    output = get_pretty_env_info()
  File "collect_env.py", line 470, in get_pretty_env_info
    return pretty_str(get_env_info())
  File "collect_env.py", line 319, in get_env_info
    pip_version, pip_list_output = get_pip_packages(run_lambda)
  File "collect_env.py", line 301, in get_pip_packages
    out = run_with_pip(sys.executable + ' -mpip')
  File "collect_env.py", line 289, in run_with_pip
    for line in out.splitlines()
AttributeError: 'NoneType' object has no attribute 'splitlines'
```

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
`soft_margin_loss` gives wrong gradient when `target` with dtype uint8
Labels: module: autograd, module: nn, module: loss, triaged

### 🐛 Describe the bug

`soft_margin_loss`gives wrong gradient when`target` with dtype uint8

```python
import torch

torch.random.manual_seed(9782)
def get_fn():
    target = torch.tensor([0, 0, 1, 1], dtype=torch.uint8, device='cpu')
    def fn(input):
        fn_res = torch.nn.functional.soft_margin_loss(input, target)
        return fn_res
    return fn
fn = get_fn()
input = torch.tensor([[0.9707, 0.2672, 0.2389, 0.6876]], dtype=torch.float64, device='cpu', requires_grad=True)
try:
    torch.autograd.gradcheck(fn, (input), check_sparse_nnz=False, atol=0.01, rtol=0.01, check_forward_ad=False, check_backward_ad=True, check_batched_grad=False)
except Exception as e:
    print(e)

# Jacobian mismatch for output 0 with respect to input 0,
# numerical:tensor([[ 0.0000],
#         [ 0.0000],
#         [-0.1101],
#         [-0.0836]], dtype=torch.float64)
# analytical:tensor([[-0.0000],
#         [-0.0000],
#         [-0.2500],
#         [-0.2500]], dtype=torch.float64)
```

If `target` has dtype `int16` or `int32`, it will return the correct gradient.

### Versions

pytorch: 1.11.0

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
`max_unpool` gives wrong gradient when `indices` has duplicate
Labels: module: autograd, module: nn, triaged, module: pooling, module: edge cases

### 🐛 Describe the bug

`max_unpool` doesn't consider that the `indices` may have duplicate elements in its gradient formula.

```python
import torch

def get_fn():
    indices = torch.tensor([[[4, 2, 3, 5]]], dtype=torch.int64, device='cuda')
    kernel_size = [1]
    stride = [2]
    def fn(input):
        fn_res = torch.nn.functional.max_unpool1d(input, indices, kernel_size, stride=stride,)
        return fn_res
    return fn
fn = get_fn()
input_tensor = torch.tensor([[[0.6418, 0.0359, 0.6088, 0.9408]]], dtype=torch.float64, device='cuda')

print(input_tensor)
# tensor([[[0.6418, 0.0359, 0.6088, 0.9408]]], dtype=torch.float64)
print(fn(input_tensor))
# tensor([[[0.6418, 0.9408, 0.6088, 0.0000, 0.0000, 0.0000, 0.0000]]],
#       dtype=torch.float64)

try:
    input = input_tensor.clone().detach().requires_grad_()
    torch.autograd.gradcheck(fn, (input), check_sparse_nnz=False, atol=0.01, rtol=0.01, check_forward_ad=False, check_backward_ad=True, check_batched_grad=False)
except Exception as e:
    print(e)

# Jacobian mismatch for output 0 with respect to input 0,
# numerical:tensor([[1.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
#         [0.0000, 1.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
#         [0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 0.0000, 0.0000],
#         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000]],
#        device='cuda:0', dtype=torch.float64)
# analytical:tensor([[1., 0., 0., 0., 0., 0., 0.],
#         [0., 1., 0., 0., 0., 0., 0.],
#         [0., 0., 1., 0., 0., 0., 0.],
#         [0., 1., 0., 0., 0., 0., 0.]], device='cuda:0', dtype=torch.float64)
```

In the above example, the output of `max_unpool` only has 3 elements but the gradient computed by reverse-mode has 4 elements.

I am not sure whether this is a big issue since `max_pool` could not return `indices` with duplicate elements.

### Versions

pytorch: 1.11.0

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
[NVFuser] Investigate models without any fusion groups found 
Labels: triaged, module: nvfuser

### 🚀 The feature, motivation and pitch

Many of the modules (29%) have no fusion groups found. It's possible that there's actually a bug in the partitioner, because this number seems high.

For those who have access: https://docs.google.com/document/d/1XU48ru9Cj4XFJJF6FTQ2X_QFuzyLroE1oomKGleYfP8/edit?usp=sharing

### Alternatives

_No response_

### Additional context

_No response_
[NVFuser] Choose partitioner op list based on supported prim decompositions
Labels: triaged, module: nvfuser

### 🚀 The feature, motivation and pitch

Currently the partitioner uses a list of supported ops that was copied from torchscript.

But for the fx -> prims -> nvfuser approach, we want the supported op list to instead be based on the list of ops that can be decomposed to prims. This could be generated by tracing through aten ops and trying to decompose them, and seeing whether or not they can be fully decomposed to prims.

### Alternatives

_No response_

### Additional context

_No response_
[NVFuser] Investigate modules with bad performance relative to eager
Labels: triaged, module: nvfuser, module: primTorch

### 🚀 The feature, motivation and pitch

In particular, `hf_DistilBert_forward_0` has ~30% regression compared to eager.

Status document (for those who have access): https://docs.google.com/document/d/1XU48ru9Cj4XFJJF6FTQ2X_QFuzyLroE1oomKGleYfP8/edit?usp=sharing

cc @ezyang @mruberry @ngimel @jjsjann123 @kevinstephano
Torch.fx: add reporting of the name of a module not found during tracing
Labels: triaged, module: fx

### 🚀 The feature, motivation and pitch

During fx tracing of modules, the function `path_of_module` tries to get the qualified name of submodules.  If it is unable to find the module (as a member of the root modules `named_modules()` method), it returns a `NameError` that does not report which module caused this issue.  This PR adds reporting of the name of that module, making debugging easier.

### Alternatives

N/A, this is just making error messages more useful

### Additional context

One example that would raise this issue is the following torch model:
```
class ExampleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3,64,(3,3))

    def forward(self, x):
        x = self.conv1(x)
        x = nn.ReLU()(x)
        return x
```
This model works training and inference, but fails fx tracing due to the initialization of the nn.ReLU() module in the forward pass.  In a toy example like above, tracing down this issue can be straightforward, but in a larger, production sized model, a lack of helpful error messages can make this a daunting task.

cc @ezyang @SherlockNoMad
GEGLU activation
Labels: module: nn, triaged, enhancement, needs research

### 🚀 The feature, motivation and pitch

Seems to be the latest and greatest activation function for transformers.

See https://arxiv.org/abs/2002.05202v1

### Alternatives

One could copy-paste the implementation from https://github.com/pfnet-research/deep-table/blob/237c8be8a405349ce6ab78075234c60d9bfe60b7/deep_table/nn/layers/activation.py

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
AMP step() enforce synchronization
Labels: triaged, module: amp (automated mixed precision)

### 🐛 Describe the bug

# torch/cuda/amp/grad_scalar.py
def _maybe_opt_step(self, optimizer, optimizer_state, *args, **kwargs):
        retval = None
        if not sum(v.item() for v in optimizer_state["found_inf_per_device"].values()): **# v is on gpu**
            retval = optimizer.step(*args, **kwargs)
        return retval

### Versions

Collecting environment information...
PyTorch version: 1.11.0+cu115
Is debug build: False
CUDA used to build PyTorch: 11.5
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.8.10 (default, Mar 15 2022, 12:22:08)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-105-generic-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: 11.5.119
GPU models and configuration: GPU 0: NVIDIA A100 80GB PCIe
Nvidia driver version: 510.47.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.11.0+cu115
[pip3] torch-geometric==2.0.4
[pip3] torch-scatter==2.0.9
[pip3] torch-sparse==0.6.13
[conda] Could not collect

cc @mcarilli @ptrblck