gopikrsmscs/torch-issues huggingface dataset:
Implement generic batch normalization layer.
Labels: oncall: distributed, feature, module: nn, triaged, needs research

### 🚀 The feature, motivation and pitch

Currently, the SyncBatchNorm module in PyTorch only has a `convert_sync_batchnorm` method to convert other Batch Normalization layers to SyncBatchNorm. However, it does not implement any method to revert the layers to their original classes.

However, implementing this would require either saving the original class type inside the SyncBatchNorm or implementing 1d, 2d, and 3d implementations of SyncBatchNorm. It would be preferable to implement instead a generic BatchNorm layer that can handle arbitrary dimensions.

Therefore, I would like to propose a generic `BatchNorm` layer that can handle arbitrary dimensions and can convert any related BatchNorm layer to itself, similar to `convert_sync_batchnorm`. This would allow removing the SyncBatchNorm layer for non-distributed inference and can convert any previous models to the new BatchNorm layer. Since the current functional `batch_norm` function is dimension neutral, the implementation will not be complicated.

### Alternatives

Possibly save the original dimensions in the SyncBatchNorm layer. However, this would not work if the original model is unknown.

### Additional context

SyncBatchNorm causes issues on single-process CPU inference and is challenging to convert to ONNX, etc. Having a generic BatchNorm would be the minimally invasive approach. Other issues have also proposed this solution before.

Also, I am willing to contribute.

https://github.com/pytorch/pytorch/issues/41081
https://github.com/pytorch/pytorch/issues/82464

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
Bad string in GLSL shader
Labels: triaged, module: vulkan, topic: build

### 🐛 Describe the bug

Hi,

I'm trying to compile the Vulkan inference backend using the instructions here: https://pytorch.org/tutorials/prototype/vulkan_workflow.html. I've checked out tag `v.1.13.0`, and I'm on Ubuntu 20.04.

I ran
```
USE_VULKAN=1 USE_VULKAN_SHADERC_RUNTIME=1 USE_VULKAN_WRAPPER=0 python setup.py install
```

The build stops on:
```
/home/user/pytorch/build/vulkan/ATen/native/vulkan/glsl.cpp:1166:1: error: unable to find string literal operator ‘operator""fully’ with ‘const char [1326]’, ‘long unsigned int’ arguments
 1166 | "  // and compute partial sums of texels that are "fully filled"\n"
```

This problem seems to originate on this [line](https://github.com/pytorch/pytorch/blob/60ffeb986648420810098cba6ac0ad1cee06bd95/aten/src/ATen/native/vulkan/glsl/layernorm.glsl#L38), with the quotes that aren't escaped when the shaders are gathered into `glsl.cpp`. `layernorm.glsl` is unchanged in master so this should still apply (unless the tool that puts `glsl.cpp` together corrects it).

### Versions

Results of collect env
```
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Jun 22 2022, 20:18:18)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.29
Is CUDA available: N/A
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

Versions of relevant libraries:
[pip3] numpy==1.23.4
[conda] Could not collect
```
pytreeify decorators
Labels: feature, triaged, module: pytree, module: functorch

Hello Functorch!

I am working on rewriting the PennyLane pytorch interface (quantum software) and I think your project can be really useful!

Currently I am trying to bypass the limitation of `autograd.Function` that only supports returning Tensor or a tuple of Tensor as return type. I have read that you want to be able to support `autograd.Function` https://github.com/pytorch/functorch/issues/1031 in Functorch. Do you think it will be possible to define functions  (`autograd.Function`) that return arbitrarily nested tuple structure? Like in Jax I would like to define `custom_vjp` (vjp needs to match Pytree structure of the function parameters) or `custom_jvp` (jvp needs to match Pytree structure of the function return).

To summarize: when you add support for custom gradient (with autograd.Function) can you also remove the structure limitation of the function return type `autograd.Function`? Or alternatively if you don't add this support but build your own custom_vjp, custom_jvp can try to avoid any limitation for the return type of functions?

Example with `autograd.function`:
```
import functorch

class ExecuteTapesNew(torch.autograd.Function):
    @staticmethod
    def forward(ctx, param_0, param_1, param_2, param_3):
        #Support Pytree structure
        return tuple([tuple([param_0, param_1]), tuple([param_2, param_3])])

    @staticmethod
    def backward(ctx, *dys):
        jac = compute_jac(params)
        vjp = dys * jacobian
        # VJP match the structure of the parameters
        return tuple(vjp)

def execute(param_0, param_1, param_2, param_3):
    return ExecuteTapesNew.apply(param_0, param_1, param_2, param_3)

jac = functorch.jacrev(execute, argnums=(0,1,2,3))(param_0, param_1, param_2, param_3)
```
Example Jax like:
```
@jax.custom_vjp
def f(param_0, param_1, param_2, param_3):
  return tuple([tuple([param_0, param_1]), tuple([param_2, param_3])])

def f_fwd(param_0, param_1, param_2, param_3):
  return f(param_0, param_1, param_2, param_3), ()

def f_bwd(res, g):
  return (param_0*g, param_1*g, param_2*g, param_3*g)

f.defvjp(f_fwd, f_bwd)
```

cc @zou3519 @soumith @Chillee @samdow @kshitij12345 @janeyx99
view_copy out= does not reshape zero element tensors
Labels: triaged, module: viewing and reshaping, module: functionalization

### 🐛 Describe the bug

from [how-does-out-work-in-pytorch](https://github.com/pytorch/pytorch/wiki/Developer-FAQ#how-does-out-work-in-pytorch) 
> if an out tensor has no elements it will be resized to the shape, stride, and memory format of the output of the computation.

however I get the following error when trying this:

```
In [4]: torch.view_copy(x, [3], out=torch.tensor([]))
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 torch.view_copy(x, [3], out=torch.tensor([]))

RuntimeError: The size of tensor a (0) must match the size of tensor b (3) at non-singleton dimension 0
``` 


### Versions

Collecting environment information...
PyTorch version: 1.14.0a0+gitcef13eb
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.24.1
Libc version: glibc-2.27

Python version: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-4.15.0-147-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: Quadro RTX 8000
GPU 1: Quadro RTX 8000

Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7
/usr/local/cuda-10.2.89/targets/x86_64-linux/lib/libcudnn.so.7
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] functorch==0.3.0a0+9166625
[pip3] mypy==0.812
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.2
[pip3] torch==1.14.0a0+git6dbde84
[conda] functorch                 0.3.0a0+9166625           dev_0    <develop>
[conda] magma-cuda117             2.6.1                         1    pytorch
[conda] mkl                       2022.1.0           h84fe81f_915    conda-forge
[conda] mkl-include               2022.1.0           h84fe81f_915    conda-forge
[conda] numpy                     1.23.2           py38h3a7f9d9_0    conda-forge
[conda] torch                     1.14.0a0+git6dbde84           dev_0    <develop>


cc @bdhirsh @ezyang @soumith
A more systematic API for resolving the "vmap-incompatible in-place operation" error
Labels: triaged, actionable, needs design, module: functorch

# Motivation

Users may run into "vmap-incompatible in-place operation" error when creating a Tensor via a factory function and in-placing into it. For example:

```py
def f(x):
  y = torch.empty(x.shape)
  y.copy_(x)
  return y

x = torch.randn(2, 3)
vmap(f)(x)
```
In this example, inside `f`: `y` is a Tensor of shape [3], but we're trying to copy a BatchedTensor (which has underlying shape [2, 3]) into `y`.

Our general advice for this situation is to use `Tensor.new_empty`. This ends up creating a `y` that is a BatchedTensor:

```py
def f(x):
  y = x.new_empty(x.shape)
  y.copy_(x)
  return y

x = torch.randn(2, 3)
vmap(f)(x)
```

Sometimes, it is not completely obvious to the user what tensor to call new_empty on. Take the following code as example:
```py
def f(x, y)
  z = torch.empty(x.shape)
  z.copy_(x)
  z.copy_(y)
  return z

# Case 1
vmap(f, (0, None))(torch.randn(2, 3), torch.randn(3))
# Case 2
vmap(f, (None, 0))(torch.randn(3), torch.randn(2, 3))
```
The way to rewrite f to support both of the above cases is a bit silly:
```py
def f(x, y)
  z = torch.cat([x, y]).new_empty(x.shape)
  z.copy_(x)
  z.copy_(y)
  return z
```
# Pitch

One possible API is that we provide an API to construct a BatchedTensor inside a function being vmapped. There are a couple of ideas on how it could look like:
```py

def f(x, y)
  # Option 1
  z = torch.empty(x.shape)
  z = torch.lift(z)
  
  # Option 2
  z = torch.lifted_empty(x.shape)
  
  # Option 3
  z = torch.empty(x.shape, lift=True)
  
  # Option 4
  z = torch.empty(x.shape)
  z = torch.lift_as(z, (x, y))  # Grants z the properties of x and y

  z.copy_(x)
  z.copy_(y)
  return z
```

Other ideas welcome.


cc @Chillee @samdow @soumith @kshitij12345 @janeyx99
Improve clarity of meaning of `torch.jit.trace`'s `example_inputs`
Labels: oncall: jit

### 📚 The doc issue

The document in question is the [torch.jit.trace documentation](https://pytorch.org/docs/stable/generated/torch.jit.trace.html#torch.jit.trace).

Specifically, it describes the `example_inputs` parameter as follows:

> **example_inputs** ([tuple](https://docs.python.org/3/library/stdtypes.html#tuple) or [torch.Tensor](https://pytorch.org/docs/stable/tensors.html#torch.Tensor)) – A tuple of example inputs that will be passed to the function while tracing. The resulting trace can be run with inputs of different types and shapes assuming the traced operations support those types and shapes. *example_inputs* may also be a single Tensor in which case it is automatically wrapped in a tuple.

This may be perfectly clear to most people, but for several days I thought that the plural "example_input**s**" meant that I could provide a tuple of a bunch of different inputs of various shapes as a tuple, and they'd all be traced. That doesn't seem to be the case; at least, in my experience, all of the inputs are passed to my `forward` function at once. What I believe this actually means, in reality, is that if your `forward` function requires multiple inputs, you can pass those inputs as a tuple.

### Suggest a potential alternative/fix

Something like this, in which I've tried to make it more clear that this is really just one example input, but it can be a tuple if the `forward` function uses multiple input values.

> **example_inputs** ([tuple](https://docs.python.org/3/library/stdtypes.html#tuple) or [torch.Tensor](https://pytorch.org/docs/stable/tensors.html#torch.Tensor)) – An example input that will be passed to the function while tracing. If the `forward` function requires multiple inputs, then this example input should be a tuple of those inputs. The resulting trace can be run with inputs of different types and shapes assuming the traced operations support those types and shapes. *example_inputs* may also be a single Tensor in which case it is automatically wrapped in a tuple.

If that is correct, then also it's a little confusing that this argument is called `example_inputs`, since it's more like "`example_input`, potentially with multiple parts", but for historical reasons I don't see that changing.

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
UserWarning: The TorchScript type system doesn't support instance-level annotations on empty non-base types in `__init__`.
Labels: oncall: jit

### 🐛 Describe the bug

This small script produces a warning, but the warning is not very actionable. Is there something we can do to make this not warn or tell me what to do to fix the warning?

```python
import torch
from torch.fx.experimental.proxy_tensor import make_fx
from functorch._src.compile_utils import strip_overloads

def f(x):
    return x

fx = make_fx(f)(torch.ones(2, 3))
strip_overloads(fx)
torch.jit.script(fx)
```


Output:
```
/usr/local/google/home/silvasean/.local/lib/python3.10/site-packages/torch/jit/_check.py:181: UserWarning: The TorchScript type system doesn't support instance-level annotations on empty non-base types in `__init__`. Instead, either 1) use a type annotation in the class body, or 2) wrap the type in `torch.jit.Attribute`.
  warnings.warn("The TorchScript type system doesn't support "
```

### Versions

Collecting environment information...
PyTorch version: 1.14.0.dev20221114+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux rodete (x86_64)
GCC version: (Debian 12.2.0-3) 12.2.0
Clang version: 14.0.6-2
CMake version: version 3.24.3
Libc version: glibc-2.35

Python version: 3.10.7 (main, Sep  8 2022, 14:34:29) [GCC 12.2.0] (64-bit runtime)
Python platform: Linux-5.18.16-1rodete4-amd64-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.4
[pip3] torch==1.14.0.dev20221114+cpu
[pip3] torchvision==0.15.0.dev20221114+cpu
[conda] Could not collect

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Extend test_proxy_tensor tests to support ops test non floating point types
Labels: module: tests, triaged, module: ProxyTensor

### 🚀 The feature, motivation and pitch

Currently, proxy tensor tests are only run for `torch.float` dtype. This means two things:
1. Ops that don't support floating point types (e.g., `bitwise_{...}` ops, `view_as_real` etc) are completely untested
2. Op behavior is only tested for floating point types and not for other dtypes. This means that we might completely miss on some dtype specific correctness for other dtypes.

### Alternatives

_No response_

### Additional context

_No response_

cc @mruberry
Add a `device` keyword argument to `torch.manual_seed`
Labels: triaged, module: random

### 🚀 The feature, motivation and pitch

I would argue that the default way to set seeds and to retrieve generators in PyTorch ML is to use `torch.manual_seed` which creates a generator underneath. 

So it's a common use case to do the following:

```python
import torch

generator = torch.manual_seed(0)
```

and then make use of the generator for downsteam pipelines. 

Now in many cases one would like the generator to be directly on GPU if all the computation happens on GPU and random tensors that are created along the way should then also directly be created on gpu:

```python
noise = torch.randn(shape, generator=generator, device="cuda")
```

In the above case, we would like `generator` to be on GPU, but it's currently a bit tedious to do this:

```python
generator = torch.Generator(device="cuda").manual_seed(0)
```

=> IMO a nicer API and easier to adapt API would be:

```python
generator = torch.manual_seed(0, device="cuda")
```

### Alternatives

```python
generator = torch.Generator(device="cuda").manual_seed(0)
```

### Additional context

With the rise of Diffusion Modes for generative tasks in ML making pipelines more deterministic becomes more and more important . E.g. we rely on the above functionality **a lot** in `difffusers`: https://github.com/huggingface/diffusers - a nicer API here would be great here.

cc @pbelevich
torch.distributed can't establish connection.
Labels: oncall: distributed

### 🐛 Describe the bug

I used three machines in the LAN to esbalish connection, it worked, however, when i can't esbalish connection with three machines in the WAN. I have closed the firewall and established connection by socket.

### Versions

```
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~18.04) 9.4.0
Clang version: Could not collect
CMake version: version 3.13.4
Libc version: glibc-2.9

Python version: 3.7.0 (default, Jun 28 2018, 13:15:42)  [GCC 7.2.0] (64-bit runtime)
Python platform: Linux-4.15.0-169-generic-x86_64-with-debian-buster-sid
Is CUDA available: N/A
CUDA runtime version: Could not collect
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

Versions of relevant libraries:
[pip3] numpy==1.15.1
[pip3] numpydoc==0.8.0
[conda] blas                      1.0                         mkl    defaults
[conda] mkl                       2019.0                      118    defaults
[conda] mkl-service               1.1.2            py37h90e4bf4_5    defaults
[conda] mkl_fft                   1.0.4            py37h4414c95_1    defaults
[conda] mkl_random                1.0.1            py37h4414c95_1    defaults
[conda] numpy                     1.15.1           py37h1d66e8a_0    defaults
[conda] numpy-base                1.15.1           py37h81de0dd_0    defaults
[conda] numpydoc                  0.8.0                    py37_0    defaults
```

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
[PT][1.13] torch .numpy() fn broke for some scenario
Labels: module: cpu, triaged, module: numpy, module: regression, actionable, ZeroTensor

### 🐛 Describe the bug

when backward tensor of sng operator is converted from torch tensor to numpy then output is not correct. 
it was expected that output should be 0's as torch tensor has filled with 0's but we get random output after performing .numpy() 
this happens only in 1.13 release, 1.12 works fine. Also if tensor is cloned then it works fine
```
import torch
torch.__version__ # 1.13.0a0 
x = torch.randn(20, dtype=torch.float32, requires_grad=True)
res = torch.sgn(x)
res_bwd = res.grad_fn(torch.ones(res.shape))
res_bwd
# tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
np_res = res_bwd.detach().numpy()
np_res
```

```
output = array([5.3426861e-35, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
       1.4012985e-45, 0.0000000e+00, 7.6396113e-35, 0.0000000e+00,
       1.7767014e+09, 4.5693540e-41, 7.6396113e-35, 0.0000000e+00,
       9.1476764e-41, 0.0000000e+00, 1.1210388e-43, 0.0000000e+00,
       3.5873241e-43, 0.0000000e+00, 1.8342932e+09, 4.5693540e-41],
      dtype=float32
```

### Versions

torch.__version__ # 1.13.0a0 

cc @VitalyFedyunin @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @mruberry @rgommers
Add smoke-tests for CPP extensions compilations
Labels: high priority, module: cpp-extensions, triaged, module: regression

### 🐛 Describe the bug

Followup for https://github.com/pytorch/pytorch/issues/88290

We should add CI/smoke tests for that by using something like the following:
https://github.com/pytorch/pytorch/pull/87615#issuecomment-1289539357

### Versions

CI

cc @ezyang @gchanan @zou3519
Fix fake tensor propagation for nvprims
Labels: triaged, open source, module: nvfuser, ciflow/inductor, no-stale

Fake tensors were not working with `torch.ops.nvprims` functions because the fake tensors code expects to see the "prim_meta_impl" attribute that was missing.

Also fixes https://github.com/pytorch/pytorch/issues/87236

cc @kevinstephano @jjsjann123
Incorrect version in the instructions on official website
Labels: module: docs, triaged

### 🐛 Describe the bug

![image](https://user-images.githubusercontent.com/30036289/201680260-31e7ee10-38d6-4f0f-a485-38f1c77fcead.png)

here https://pytorch.org/get-started/previous-versions/ in version 9.1 installation listed torchaudio of incomparable version

### Versions

9.1

cc @svekars @carljparker
nvprims.native_batch_norm doesn't support fake tensor inputs
Labels: triaged, module: nvfuser, module: primTorch

### 🐛 Describe the bug

Add `tracing_mode="fake"` here to see the error: https://github.com/pytorch/pytorch/blob/9943d46aab4465b887039aa1a9b5d9ebc0a01a35/test/test_prims.py#L614

In order to support fake tensors `prim` should have "prim_meta_impl" attribute that is a callable used for propagating fake tensors. See the usage here: https://github.com/pytorch/pytorch/blob/9943d46aab4465b887039aa1a9b5d9ebc0a01a35/torch/_subclasses/fake_tensor.py#L864-L870

### Versions

latest master

cc @kevinstephano @jjsjann123 @ezyang @mruberry @ngimel @Lezcano @fdrocha @peterbell10
Glog macro redefinition problem when including headers from both libtorch and glog
Labels: module: build, triaged

### 🐛 Describe the bug

Hi, I am trying to build a C++ shared library that uses Pytorch C++ custom class (torchscript class). I linked the library aganist Libtorch and glog but got macro redefinition warnings. Those macros are loggers like `LOG` and `CHECK`, which are defined in both Libtorch (`torch/include/c10/util/logging_is_google_glog.h` or `torch/include/c10/util/logging_is_not_google_glog.h`) and glog. I can see Pytorch have similar issues before (#31822, #14724) and a PR (#41504) to solve the redefinition issue when libtorch is not built with glog. However, I still have the redefinition problem when including the headers of glog and `torch/custom_class.h` (which will include `torch/include/c10/util/logging_is_not_google_glog.h` in the end, resulting in macro redefinition). I wonder whether there is a workaround to get rid of this problem?


### Versions

[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.1
[pip3] torch==1.12.1+cu113
[pip3] torchaudio==0.12.1+cu113
[pip3] torchmetrics==0.10.0
[pip3] torchvision==0.13.1+cu113

cc @malfet @seemethere
M1 runner i-090e1df32b6f48a20 run out of disk space
Labels: module: ci, triaged, module: m1

### 🐛 Describe the bug

See
https://github.com/pytorch/pytorch/actions/runs/3453509296/jobs/5764272366
https://github.com/pytorch/pytorch/actions/runs/3453862853/jobs/5764898047
https://github.com/pytorch/pytorch/actions/runs/3454468047/jobs/5765868837
https://github.com/pytorch/pytorch/actions/runs/3454765799/jobs/5766454600
https://github.com/pytorch/pytorch/actions/runs/3455173679/jobs/5767115747

### Versions

CI

cc @ezyang @gchanan @zou3519 @seemethere @pytorch/pytorch-dev-infra
[Inductor] Constant folding support
Labels: triaged, enhancement, oncall: pt2, module: inductor

## Motivating Example
Below is a case in MobileBertForMaskedLM which has a concatenation on two model parameters (`hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0))`). This concat takes >20% of the single-threaded inference time on CPU but this cost can be saved with constant folding.
```python
class MobileBertLMPredictionHead(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.transform = MobileBertPredictionHeadTransform(config)
        # The output weights are the same as the input embeddings, but there is
        # an output-only bias for each token.
        self.dense = nn.Linear(config.vocab_size, config.hidden_size - config.embedding_size, bias=False)
        self.decoder = nn.Linear(config.embedding_size, config.vocab_size, bias=False)
        self.bias = nn.Parameter(torch.zeros(config.vocab_size))
        # Need a link between the two variables so that the bias is correctly resized with `resize_token_embeddings`
        self.decoder.bias = self.bias

    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
        hidden_states = self.transform(hidden_states)
        hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0))
        hidden_states += self.decoder.bias
        return hidden_states
```


cc @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire @ezyang @msaroufim @wconstab @ngimel @bdhirsh
`torch.nn.functional.embedding_bag` Trigger RuntimeError under UndefinedBehaviorSanitizer
Labels: triaged, module: sanitizers

### 🐛 Describe the bug

The implementation of `torch.nn.functional.embedding_bag` contains undefined behavior caught by UB sanitizer. Without sanitizers, the test terminates normally.
Test:
```python
import torch

def test():
    arg_1 = torch.randint(-1024,1024,[6], dtype=torch.int32).clone()
    arg_2 = torch.rand([5, 2], dtype=torch.float32).clone()
    arg_3 = torch.randint(-8192,0,[0], dtype=torch.int64).clone()
    arg_9 = torch.rand([6], dtype=torch.float32).clone()
    _ = torch.nn.functional.embedding_bag(arg_1, arg_2, arg_3, None, 2.0, False, "sum", False, arg_9, False, None)

test()
```
Error log:
```
pytorch/aten/src/ATen/native/EmbeddingBag.cpp:708:11: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:44:28: note: nonnull attribute specified here
```

### Versions

```
PyTorch version: 1.14.0a0+git6e5f736
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: 11.1.0-6
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.9.12 (main, Apr  5 2022, 06:56:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] numpydoc==1.2
[pip3] torch==1.14.0a0+git6e5f736
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.21.5           py39he7a7128_1  
[conda] numpy-base                1.21.5           py39hf524024_1  
[conda] numpydoc                  1.2                pyhd3eb1b0_0  
[conda] torch                     1.14.0a0+git6e5f736          pypi_0    pypi
```
`torch.set_rng_state` Trigger RuntimeError under UndefinedBehaviorSanitizer
Labels: triaged, module: sanitizers

### 🐛 Describe the bug

The implementation of `torch.set_rng_state` contains undefined behavior caught by UB sanitizer. 
Test:
```python
import torch

def test():
    arg_1 = torch.randint(0,64,[5056], dtype=torch.uint8).clone()
    _ = torch.set_rng_state(arg_1,)

test()
```
Error log:
```
pytorch/aten/src/ATen/CPUGeneratorImpl.cpp:167:20: runtime error: load of value 46, which is not a valid value for type 'bool'
```

### Versions

```
PyTorch version: 1.14.0a0+git6e5f736
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: 11.1.0-6
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.9.12 (main, Apr  5 2022, 06:56:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] numpydoc==1.2
[pip3] torch==1.14.0a0+git6e5f736
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.21.5           py39he7a7128_1  
[conda] numpy-base                1.21.5           py39hf524024_1  
[conda] numpydoc                  1.2                pyhd3eb1b0_0  
[conda] torch                     1.14.0a0+git6e5f736          pypi_0    pypi
```
torch.linalg.matrix_rank memory leak
Labels: needs reproduction, module: memory usage, triaged, module: linear algebra

### 🐛 Describe the bug

Below is a minimal script for reproducing.

```python
    import torch
    import os, psutil
    
    a = torch.randn(100, 100, 512)
    for _ in range(10):
        torch.linalg.matrix_rank(a)
        print(psutil.Process(os.getpid()).memory_info().rss)
```

The above script gives the following output:
```
4214136832
8331022336
10537857024
12173451264
13269585920
14028787712
14656843776
15086813184
15620276224
16046718976
```
I did some test, and it seems that it only happens when the tensor is high dimensional. For example, if the dimension of `a` is (100, 100, 100) then everything is fine.

### Versions

PyTorch version: 1.13.0a0
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: SUSE Linux Enterprise Server 15 SP3 (x86_64)
GCC version: (SUSE Linux) 7.5.0
Clang version: Could not collect
CMake version: version 3.17.0
Libc version: glibc-2.31

Python version: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.3.18-150300.59.87-default-x86_64-with-glibc2.31
Is CUDA available: False
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.982
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.20.3
[pip3] numpydoc==1.1.0
[pip3] torch==1.13.0a0+gitunknown
[pip3] torchaudio==0.13.0+bc8640b
[pip3] torchtext==0.14.0a0+e2b27f9
[pip3] torchvision==0.14.0a0
[conda] blas                      1.0                         mkl
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-include               2021.4.0           h06a4308_640    <unknown>
[conda] mkl-service               2.4.0            py39h7f8727e_0
[conda] mkl_fft                   1.3.1            py39hd3c417c_0
[conda] mkl_random                1.2.2            py39h51133e4_0
[conda] numpy                     1.20.3           py39hf144106_0
[conda] numpy-base                1.20.3           py39h74d4b33_0
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1


cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano
`torch.Tensor.msort` Trigger RuntimeError under UndefinedBehaviorSanitizer
Labels: triaged, module: sanitizers

### 🐛 Describe the bug

The implementation of `torch.Tensor.msort` contains undefined behavior caught by UB sanitizer. Without sanitizers, the test terminates normally.
Test:
```python
import torch

def test():
    arg_1 = torch.rand([0, 16, 1024, 1024], dtype=torch.float16).clone()
    res = torch.Tensor.msort(arg_1,)

test()
```
Error log:
```
pytorch/aten/src/ATen/TensorIteratorInternal.h:34:17: runtime error: applying non-zero offset 1048576 to null pointer
```

### Versions

```
PyTorch version: 1.14.0a0+git6e5f736
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: 11.1.0-6
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.9.12 (main, Apr  5 2022, 06:56:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] numpydoc==1.2
[pip3] torch==1.14.0a0+git6e5f736
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.21.5           py39he7a7128_1  
[conda] numpy-base                1.21.5           py39hf524024_1  
[conda] numpydoc                  1.2                pyhd3eb1b0_0  
[conda] torch                     1.14.0a0+git6e5f736          pypi_0    pypi
```
`torch.linalg.eigvals` Trigger RuntimeError under UndefinedBehaviorSanitizer
Labels: triaged, module: sanitizers

### 🐛 Describe the bug

The implementation of `torch.linalg.eigvals` contains undefined behavior caught by UB sanitizer. Without sanitizers, the test terminates normally.
Test:
```
import torch

def test():
    arg_1 = torch.rand([0, 3, 3], dtype=torch.float64).clone()
    res = torch.linalg.eigvals(arg_1,)

test()
```
Error log:
```
pytorch/aten/src/ATen/native/BatchLinearAlgebra.cpp:954:18: runtime error: applying non-zero offset 24 to null pointer
```

### Versions

```
PyTorch version: 1.14.0a0+git6e5f736
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: 11.1.0-6
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.9.12 (main, Apr  5 2022, 06:56:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] numpydoc==1.2
[pip3] torch==1.14.0a0+git6e5f736
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.21.5           py39he7a7128_1  
[conda] numpy-base                1.21.5           py39hf524024_1  
[conda] numpydoc                  1.2                pyhd3eb1b0_0  
[conda] torch                     1.14.0a0+git6e5f736          pypi_0    pypi
```
`torch.topk` Trigger RuntimError under UndefinedBehaviorSanitizer
Labels: triaged, module: sanitizers

### 🐛 Describe the bug

The implementation of `torch.topk` contains undefined behavior caught by UB sanitizer. Without sanitizers, the test terminates normally.
Test:
```python
import torch

def test():
    arg_1 = torch.rand([3, 5], dtype=torch.float64).clone()
    res = torch.topk(arg_1,0,0,)

test()
```
Error log:
```
pytorch/aten/src/ATen/native/TopKImpl.h:30:45: runtime error: applying non-zero offset 8 to null pointer
```

### Versions

```
PyTorch version: 1.14.0a0+git6e5f736
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: 11.1.0-6
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.9.12 (main, Apr  5 2022, 06:56:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] numpydoc==1.2
[pip3] torch==1.14.0a0+git6e5f736
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.21.5           py39he7a7128_1  
[conda] numpy-base                1.21.5           py39hf524024_1  
[conda] numpydoc                  1.2                pyhd3eb1b0_0  
[conda] torch                     1.14.0a0+git6e5f736          pypi_0    pypi
```
`torch.vander` Trigger RuntimeError with UndefinedBehaviorSanitizer
Labels: triaged, module: linear algebra, module: sanitizers, module: edge cases

### 🐛 Describe the bug

The implementation of `torch.vander` contains undefined behavior caught by UB sanitizer. Without sanitizers, the test terminates normally.
Test:
```python
import torch

def test():
    arg_1 = torch.randint(-1,512,[4], dtype=torch.int64).clone()
    res = torch.vander(arg_1,12,)

test()
```
Error log:
```
pytorch/aten/src/ATen/native/cpu/ReduceOpsKernel.cpp:100:3: runtime error: signed integer overflow: 7179408129339016064 * 494 cannot be represented in type 'long'
```

### Versions

```
PyTorch version: 1.14.0a0+git6e5f736
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: 11.1.0-6
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.9.12 (main, Apr  5 2022, 06:56:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] numpydoc==1.2
[pip3] torch==1.14.0a0+git6e5f736
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.21.5           py39he7a7128_1  
[conda] numpy-base                1.21.5           py39hf524024_1  
[conda] numpydoc                  1.2                pyhd3eb1b0_0  
[conda] torch                     1.14.0a0+git6e5f736          pypi_0    pypi
```

cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano
`torch.svd_lowrank` Trigger RuntimeError under UndefinedBehaviorSanitizer
Labels: module: sparse, triaged, module: linear algebra, actionable, module: sanitizers

### 🐛 Describe the bug

The implementation of `torch.svd_lowrank` contains undefined behavior caught by UB sanitizer. Without sanitizers, the test terminates normally.
Test:
```python
import torch

def test():
    arg_1 = torch.rand([5, 5], dtype=torch.float64).to_sparse()
    res = torch.svd_lowrank(arg_1,False,)

test()
```
Error log:
```
pytorch/aten/src/ATen/native/sparse/SparseTensorMath.cpp:1270:19: runtime error: applying non-zero offset 8 to null pointer
```

### Versions

```
PyTorch version: 1.14.0a0+git6e5f736
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: 11.1.0-6
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.9.12 (main, Apr  5 2022, 06:56:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] numpydoc==1.2
[pip3] torch==1.14.0a0+git6e5f736
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.21.5           py39he7a7128_1  
[conda] numpy-base                1.21.5           py39hf524024_1  
[conda] numpydoc                  1.2                pyhd3eb1b0_0  
[conda] torch                     1.14.0a0+git6e5f736          pypi_0    pypi
```

cc @ezyang @gchanan @zou3519 @nikitaved @pearu @cpuhrsch @amjames @bhosmer @jianyuh @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano