gopikrsmscs/torch-issues huggingface dataset:
End-to-End AMP training with GradScaler
Labels: triaged, oncall: pt2

https://github.com/pytorch/torchdynamo/blob/main/benchmarks/training_loss.py performs end-to-end training with TorchDynamo and TorchInductor with AMP. However, we are not testing AMP training in a standard manner. 

AMP training has [GradScaler](https://pytorch.org/docs/stable/amp.html#gradient-scaling) which performs scaling, monitors gradients, discards batch sizes if gradients are inf/nan. More importantly, GradScaler is stateful. All this warrants an end to end training testing with TorchDynamo and TorchInductor stack.

There would be many great articles on AMP. This is the first one I stumbled upon - https://spell.ml/blog/mixed-precision-training-with-pytorch-Xuk7YBEAACAASJam

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @yanboliang 
cc @xuzhao9 if torchbench already has AMP setup for end to end training

cc @msaroufim if you are interested in this
torch.cuda.empty_cache() is not working
Labels: module: cuda, triaged

### 🐛 Describe the bug

Hi, I want to see how much does the pytorch model use GPU, but I went I try to test more than 2 models, `torch.cuda.empty_cache()` seems not working.

The code is as below, the first `logger` shows 0, but the second shows more than 0

```
model = transformers.T5ForConditionalGeneration.from_pretrained('t5-base')
torch.cuda.reset_peak_memory_stats()
torch.cuda.empty_cache()
torch.cuda.synchronize()
now_mem = torch.cuda.max_memory_allocated()
logger.info(f'Load original model, mem:{now_mem}')

train(model,dataloader,...)


model = transformers.T5ForConditionalGeneration.from_pretrained('t5-base')
torch.cuda.reset_peak_memory_stats()
torch.cuda.empty_cache()
torch.cuda.synchronize()
now_mem = torch.cuda.max_memory_allocated()
logger.info(f'Load original model, mem:{now_mem}')
train(model,dataloader,...)

```

### Versions

```
Collecting environment information...
PyTorch version: 1.12.1+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.17

Python version: 3.7.13 (default, Mar 29 2022, 02:18:16)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.13.0-48-generic-x86_64-with-debian-bullseye-sid
Is CUDA available: True
CUDA runtime version: 11.7.99
GPU models and configuration: GPU 0: Tesla V100-SXM2-32GB
Nvidia driver version: 515.48.07
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.12.1
[conda] numpy                     1.21.6                   pypi_0    pypi
[conda] torch                     1.12.1                   pypi_0    pypi

```

cc @ngimel
Improve FX naming for getitem calls
Labels: triaged, module: fx, module: aotdispatch

### 🐛 Describe the bug

When I have a tuple destructure, I end up with

```
getitem_0 = operator.getitem(add_0, 0)
getitem_1 = operator.getitem(add_0, 1)
getitem_2 = operator.getitem(add_0, 2)
getitem_3 = operator.getitem(add_0, 3)
```

names in FX IR. The getitem variable names are useless, they tell me nothing about where the name came from. A better naming scheme would see past the operator.getitem and derive a name based on the input variable name to the getitem call, e.g.,

```
add_0_0 = operator.getitem(add_0, 0)
add_0_1 = operator.getitem(add_0, 1)
add_0_2 = operator.getitem(add_0, 2)
add_0_3 = operator.getitem(add_0, 3)
```

### Versions

master

cc @SherlockNoMad @soumith
Dedicated function for shallow_copy_and_detach
Labels: module: autograd, triaged, tensor subclass

### 🐛 Describe the bug

Today when a tensor is internally shallow copy and detached, it is traced as a detach. But a detach has a particular semantic meaning: it means that we are intending to detach ALL levels of autograd. But that's not really the intention here; we are just breaking the autograd history at the current level; you should still be able to differentiably run the code later. By tracing it as detach, we lose this semantic info. So emit these as something else; maybe even just alias.

Related https://github.com/pytorch/pytorch/issues/71725

cc @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 we talked about doing it this way, but no one did it, writing it down so we don't forget

### Versions

master
Stack trace preservation should work on plain use of make_fx / AOTAutograd
Labels: triaged, module: aotdispatch

### 🐛 Describe the bug

The current stack preservation logic from #83558 only works for Dynamo. It would be nice if it worked for bare use of `make_fx` and `AOTAutograd`. I hacked this up to work:

1. I think the anomaly mode context manager should be pushed into `create_joint_forward_backward`, where we actually run the forwards. This guarantees it gets run by all the codepaths. Right now it is applied only on `aot_module_simplified`, which means you don't get it for the other AOTAutograd codepaths
2. make_fx has no way of turning on `record_stack_traces` in its public API. Actually, we should just do the @zdevito trick and make this cheap enough that we can turn it on unconditionally
3. You want this diff

```
diff --git a/torch/fx/traceback.py b/torch/fx/traceback.py
index a07b36b997..cee7626e5c 100644
--- a/torch/fx/traceback.py
+++ b/torch/fx/traceback.py
@@ -54,7 +54,7 @@ def format_stack() -> List[str]:
         return current_stack.copy()
     else:
         # fallback to traceback.format_stack()
-        return traceback.format_stack()
+        return traceback.format_list(traceback.extract_stack()[:-1])
 
 
 @compatibility(is_backward_compatible=False)
```

cc @SherlockNoMad @Chillee @albanD 

### Versions

master
DISABLED test_rmsprop (optim.test_optim.TestOptim)
Labels: triaged, module: flaky-tests, skipped, module: dynamo

Platforms: dynamo

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_rmsprop&suite=TestOptim) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/8754116789).

Over the past 3 hours, it has been determined flaky in 1 workflow(s) with 1 failures and 1 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT BE ALARMED IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_rmsprop`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


cc @vincentqb @jbschlosser @jansel @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305
[minifier] Accuracy minification
Labels: triaged, oncall: pt2

- [ ] For integer tensors, currently we have torch.zeros for rand_strided (https://github.com/pytorch/torchdynamo/blob/main/torchdynamo/testing.py#L254). This is ok for Exceptions, but not really useful for accuracy. We need to get the maximum value from the original tensor and then replace zeros with randint(max_value)

* `git revert 35590bb9ae5781d93e483540e9492e1409347800`
* `TORCHDYNAMO_REPRO_AFTER="dynamo" TORCHDYNAMO_REPRO_LEVEL=4 benchmarks/huggingface.py -d cuda --inductor --training --float32 --accuracy  --only=BertForQuestionAnswering` shows that accuracy divergence was observed
* But `python /tmp/minifier_anijain/minifier_launcher.py` does not minify
* Changing torch.zeros to torch.randint(10, ..) as mentioned above would fix it. So, the goal is to get the max_value plumbed through.

- [ ] We currently do fwd_output.sum().backward() - This could hide indexing issues, because the loss is same for all the tensors. We should provide torch.rand_like() input gradients. More context here - https://github.com/pytorch/torchdynamo/pull/1414#issuecomment-1262801213


cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @williamwen42 @yanboliang
[functorch] [aot_autograd] 
Labels: triaged, module: vmap, module: functionalization, module: aotdispatch

This is what I am trying to do:

1. Create an input and target tensor
2. Create a list of models for ensemble
3. Use vmap to create an ensemble
4. Obtain the vmap function and pass it to aot_function, to obtain the forward and backward graphs

Repro:
```
from typing import List
import torch
from torch._subclasses import FakeTensorMode, FakeTensor
from functorch.compile import aot_function, print_compile, config, aot_module
from functorch import make_functional_with_buffers, vmap, combine_state_for_ensemble
from functorch._src.named_members_polyfill import _named_parameters, _named_buffers
from torchvision.models import resnet18

g = {}
def fake_wrapper(gtype):
    def fake_compiler(fx_g, inps):
        print(fx_g.code)
        nonlocal gtype
        g[gtype] = fx_g
        return fx_g
    return fake_compiler


inp = torch.randn(32, 3, 224, 224, dtype=torch.float32).cuda()
targets = torch.zeros(32, dtype=int).cuda()

b_models:List[torch.nn.Module] = [resnet18().cuda() for _ in range(5)]

func_model, params, buffers = combine_state_for_ensemble(b_models)
for p in params:
    p.requires_grad = True

def compute_loss(weights, buffers, batch, targets):
    output = func_model(weights, buffers, batch)
    loss = torch.nn.functional.nll_loss(output,targets)
    return loss
parallel_func = vmap(compute_loss, in_dims=(0,0,None, None))
aot_func = aot_function(parallel_func, fake_wrapper("forward"), fake_wrapper("backward"))
out = aot_func(params, buffers, inp, targets)
out.mean().backward()
```

Error:
`RuntimeError: !self.requires_grad() || self.is_contiguous() INTERNAL ASSERT FAILED at “/scratch/anijain/work/pytorch/aten/src/ATen/native/TensorShape.cpp”:3609, please report a bug to PyTorch. as_strided_scatter is currently only supported for contiguous inputs
While executing %new_empty_strided_1 : [#users=1] = call_function[target=torch.ops.aten.new_empty_strided.default](args = (%copy__1, [5, 512, 32, 7, 7], [25088, 49, 125440, 7, 1]), kwargs = {})
Original traceback:
Gradient addition node due to multiple use of tensor around:`




cc @zou3519 @bdhirsh @ezyang @soumith
OpInfo Tests To Validate that All Operators Are Being Tested With Strided Tensors
Labels: module: tests, triaged

### 🚀 The feature, motivation and pitch

Many of the OpInfos such as [Unary OpInfos](https://github.com/pytorch/pytorch/pull/85976) do not currently test with strided inputs. We should add a test that validates all OpInfos are being tested with strided inputs. 

cc @mruberry
`conv_transpose` is not similar to `nn.grad.conv_input` when `output_padding` is passed with non-default values.
Labels: module: nn, triaged, actionable

### 🐛 Describe the bug

```python
import torch
import torch.nn as nn

device = 'cuda'
t = torch.rand(2, 2, 4, 4, device=device)
w = torch.rand(2, 2, 4, 5, device=device)
b = None

stride = (3, 2)
padding = (1, 2)
dilation = (4, 4)
output_padding = (2, 3)  # Errors with `RuntimeError: grad_output[2:] shape[4, 4] must be equal to output size 4 5`

# output_padding = 0  # Works if we don't specify a non-default output_padding
groups = 2

actual = torch.conv_transpose2d(t, w, b, groups=groups, stride=stride, padding=padding, dilation=dilation, output_padding=output_padding)

expected = nn.grad.conv2d_input(actual.shape, w, t, stride, padding,dilation, groups)

torch.testing.assert_close(actual, expected)
```

We try to use `grad.conv2d_input` as reference for Complex support for conv_transpose, but it fails for samples when `output_padding` is passed with non-default value. `nn.grad.conv2d_input` doesn't take that argument (probably doesn't require it).

### Versions

master

cc @albanD @mruberry @jbschlosser @walterddr @saketh-are
Jetson JIT: Memory Leak on inference after optimize_for_inference
Labels: oncall: jit

### 🐛 Describe the bug

After investigating a memory leak that happens only on one hardware platform, I found that torch::jit::optimize_for_inference induces a change in the torchscript that causes a memory leak on Jetson devices.

Running on the same libtorch compiled from source on Windows, Mac and Linux with the same optimisation applied does not cause any memory leak.

To reproduce, you can run a small model such as Retinaface that is traced, with and without torch::jit::optimize_for_inference.

I have attached two torchscript code files below. bad causes the memory leak and good does not.
[good___torch_mangle_205.txt](https://github.com/pytorch/pytorch/files/9721370/good___torch_mangle_205.txt)
-> torch::jit::optimize_for_inference ->
[bad___torch_mangle_205.txt](https://github.com/pytorch/pytorch/files/9721373/bad___torch_mangle_205.txt)

Only obvious change I can see is conv + relu fusion.
The only significant difference to the other working platforms I can find is: Jetson uses CUDA 10.2 / CUDNN 8.0, Jetson is ARM64, Jetson has shared GPU memory.

### Versions

1.12.1 and 1.13.0-dev2022.08.13
Add complex support for SparseAdam and LBFGS optimizers
Labels: module: optimizer, triaged, module: complex, actionable

As per title
These optimizers are less high priority than the other in https://github.com/pytorch/pytorch/issues/65711 and so are not being worked on by the core team right now.
Feel free to pick this up if you're interested.

cc @vincentqb @jbschlosser @ezyang @anjali411 @dylanbespalko @mruberry @Lezcano @nikitaved
`torch.special.round` doesn't support the same dtypes as `torch.round`
Labels: triaged, module: special

### 🐛 Describe the bug

`torch.special.round` is declared as an alias of `torch.round` in the `OpInfo`. But they behave differently (due to a bug?):

```py
In [2]: torch.special.round(torch.tensor([2], dtype=torch.short))
RuntimeError: "round_vml_cpu" not implemented for 'Short'

In [3]: torch.round(torch.tensor([2], dtype=torch.short))
Out[3]: tensor([2], dtype=torch.int16)

In [4]: torch.tensor([2], dtype=torch.short).round()
Out[4]: tensor([2], dtype=torch.int16)
```

The error message is from `AT_DISPATCH_SWITCH`. I haven't looked into why it happens.

Aliases are tested in `test_variant_consistency_eager`, but that test only checks against `torch.float` and `torch.cfloat` (see `_variant_ops`). So we didn't notice. If I allow testing for all dtypes there, I get the same error:

```py
# old:
# _variant_ops = partial(
#     ops, dtypes=OpDTypes.supported, allowed_dtypes=(torch.float, torch.cfloat)
# )
# new:
_variant_ops = partial(
    ops, dtypes=OpDTypes.supported
)
```

```bash
# run after allowing all types in _variant_ops
python -m pytest test/test_ops.py -k test_variant_consistency_eager_round -vv  --capture=no 
```

output:
```
FAILED test/test_ops.py::TestCommonCPU::test_variant_consistency_eager_round_cpu_int16 - RuntimeError: "round_vml_cpu" not implemented for 'Short'
FAILED test/test_ops.py::TestCommonCPU::test_variant_consistency_eager_round_cpu_int32 - RuntimeError: "round_vml_cpu" not implemented for 'Int'
FAILED test/test_ops.py::TestCommonCPU::test_variant_consistency_eager_round_cpu_int64 - RuntimeError: "round_vml_cpu" not implemented for 'Long'
FAILED test/test_ops.py::TestCommonCPU::test_variant_consistency_eager_round_cpu_int8 - RuntimeError: "round_vml_cpu" not implemented for 'Char'
FAILED test/test_ops.py::TestCommonCPU::test_variant_consistency_eager_round_cpu_uint8 - RuntimeError: "round_vml_cpu" not implemented for 'Byte'

FAILED test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_round_cuda_int16 - RuntimeError: "round_cuda" not implemented for 'Short'
FAILED test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_round_cuda_int32 - RuntimeError: "round_cuda" not implemented for 'Int'
FAILED test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_round_cuda_int64 - RuntimeError: "round_cuda" not implemented for 'Long'
FAILED test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_round_cuda_int8 - RuntimeError: "round_cuda" not implemented for 'Char'
FAILED test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_round_cuda_uint8 - RuntimeError: "round_cuda" not implemented for 'Byte'
```

### Versions

master (82d9592f1ba)

cc @mruberry @kshitij12345
Feature request: Tests for `int` should be tests for `numbers.Integral`
Labels: feature, triaged, module: numpy, module: python frontend

### 🐛 Describe the bug

Some code, such as that in `BatchSampler`, tests whether a value, e.g. `batch_size` is an integer with `isinstance(batch_size, int)`.

This fails in a case like this:
```py
for batch_size in np.logspace(1, 4, 7, dtype=int):
    # train a model with this batch size
```
because even though `batch_size` behaves just like an int, it's not an _instance_ of int.

The specific error in this case highlights that some logic is not quite right:
```
ValueError: batch_size should be a positive integer value, but got batch_size=10
```

Note that the issue isn't related to `np.logspace` specifically, just any int taken from any NumPy array:
```py
assert isinstance(np.array([1])[0], int) == False
assert isinstance(np.array([1])[0], numbers.Integral) == True
```

It would be better to replace `isinstance(batch_size, int)` with `isinstance(batch_size, numbers.Integral)`.

Unless there's something I don't understand, this is probably a good general rule for the whole codebase. Typically these numbers are then passed to `range()` or `slice()` which have the requirement of `Integral`, _not_ the narrower requirement of `int`.



### Versions

PyTorch version: 1.12.1+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.10.7 (main, Sep  7 2022, 15:22:19) [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 516.94
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.971
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.3
[pip3] torch==1.12.1+cu116
[pip3] torchaudio==0.12.1+cu116
[pip3] torchvision==0.13.1+cu116
[conda] Could not collect

cc @mruberry @rgommers @albanD
AOT Autograd Device Partitioning
Labels: triaged, module: functorch

### 🚀 The feature, motivation and pitch

Pytorch Eager will parallelize the backward based on devices. We should do something similar in AOT Autograd, perhaps with subprocesses. 

### Alternatives

_No response_

### Additional context

_No response_

cc @zou3519 @Chillee @samdow @soumith
`torch.multinomial` on MPS crashes with `Error: total bytes of NDArray > 2**32'`
Labels: triaged, module: regression, module: mps

### 🐛 Describe the bug

After https://github.com/pytorch/pytorch/pull/80760 added MPS version of multinomial op, operations with replacement fails for arrays of more than 32K elements with non-recoverable error:
```
 % python -c "import torch;print(torch.multinomial(torch.ones(1, 32768, device='mps'), 2, replacement=True))"
/AppleInternal/Library/BuildRoots/4883e71d-37bd-11ed-b0ef-b25c5e9b9057/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSCore/Types/MPSNDArray.mm:724: failed assertion `[MPSNDArray initWithDevice:descriptor:] Error: total bytes of NDArray > 2**32'
zsh: abort      python -c 
```

### Versions

Nightly

cc @kulinseth @albanD @DenisVieriu97 @razarmehr @abhudev
JIT miss the argument `as_tuple` for API `nonzero`
Labels: oncall: jit

### 🐛 Describe the bug

JIT miss the argument `as_tuple` for API `nonzero`

```py
import torch


def fn(input):
    as_tuple = False
    return torch.nonzero(input, as_tuple=as_tuple, )

inp = torch.tensor(1., dtype=torch.float32)

print(fn(inp))

jit_fn = torch.jit.script(fn)
print(jit_fn(inp))
```

```
tensor([], size=(1, 0), dtype=torch.int64)

RuntimeError:
Arguments for call are not valid.
The following variants are available:

  aten::nonzero(Tensor self) -> Tensor:
  Keyword argument as_tuple unknown.

  aten::nonzero.out(Tensor self, *, Tensor(a!) out) -> Tensor(a!):
  Argument out not provided.

The original call is:
def fn(input):
    as_tuple = False
    return torch.nonzero(input, as_tuple=as_tuple, )
           ~~~~~~~~~~~~~ <--- HERE
```

### Versions

```
PyTorch version: 1.13.0.dev20221001
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.2.0-19ubuntu1) 11.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.13 (main, Aug 25 2022, 23:26:10)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.19.5-051905-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.7.99
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060
Nvidia driver version: 515.65.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.3
[pip3] torch==1.13.0.dev20221001
[pip3] torchaudio==0.13.0.dev20221001
[pip3] torchvision==0.14.0.dev20221001
[conda] blas                      1.0                         mkl  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.23.3                   pypi_0    pypi
[conda] numpy-base                1.23.1           py39ha15fc14_0  
[conda] pytorch                   1.13.0.dev20221001 py3.9_cuda11.7_cudnn8.5.0_0    pytorch-nightly
[conda] pytorch-cuda              11.7                 h67b0de4_0    pytorch-nightly
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] torch                     1.13.0.dev20220923+cu116          pypi_0    pypi
[conda] torchaudio                0.13.0.dev20220923+cu116          pypi_0    pypi
[conda] torchvision               0.14.0.dev20220923+cu116          pypi_0    pypi
```
TransformerEncoder/TransformerDecoder has same initial parameters for all layers
Labels: triaged, oncall: transformer/mha

### 🐛 Describe the bug

Example code (based on [documentation](https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html#torch.nn.TransformerEncoder)):
```python
encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8)
transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=6)
```
Then you have:
```python
(transformer_encoder.layers[0].linear1.weight == transformer_encoder.layers[1].linear1.weight).all() == True
```
Etc.

This is because:
```python
self.layers = _get_clones(encoder_layer, num_layers)
```
Where `_get_clones` uses `deepcopy`.

The same problem is for `TransformerDecoder`.

This is unexpected behavior to me but maybe it is expected and just not well documented?

Note that `Transformer` itself does not have this problem because it calls `_reset_parameters` after all the layers are assigned.


### Versions

(I have an outdated version but I checked the source code of the current version.)

cc @jbschlosser @bhosmer @cpuhrsch @erichan1
TORCH_WARN is executed just once per set of parameters
Labels: module: logging, triaged

### 🐛 Describe the bug

Set up the fix in https://github.com/pytorch/pytorch/pull/86255 to repro this one, as otherwise you won't see the warnings even.

It seems that `TORCH_WARN` acts as `TORCH_WARN_ONCE` given a fixed set of arguments. To repro this, add the lines `TORCH_WARN("HI"); TORCH_CHECK(false);` in, say, `FuncionsManual.cpp` in the function `svd_backwards` after the second `if` statement. Then run `pytest test/test_ops_gradients.py -vk test_fn_grad_linalg_svd_cpu`. This will run two tests that will fail. Now, the warning will just be triggered in the first test, but not in the second.

More generally, it seems that if `TORCH_WARN` already printed a given parameter in a previous run, it will not print it again. More concretely, if you do `TORCH_WARN(a);` and, say, `a = 0`, then `a` will be printed, but if later on this warning is hit again with `a = 0`, it will not be printed. To see this, change in the code above `TORCH_WARN("HI")` by `TORCH_WARN(gS)`

cc @kurtamohler @peterbell10 

### Versions

After patching in  https://github.com/pytorch/pytorch/pull/86255
Support guard on thread number
Labels: triaged, enhancement, oncall: pt2, module: dynamo

As mentioned in https://github.com/pytorch/torchdynamo/pull/1452 and discussed in https://github.com/pytorch/torchdynamo/pull/1452#discussion_r986774789, thread number is similar to the shapes that can be specialized to get better perf. Supporting guard on thread number can make sure the specialized kernels are recompiled with the changes of thread number.

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
[Inductor] Support float32 accumulation type for float16 sum on CPU
Labels: triaged, oncall: pt2, module: cpu inductor

Currently float16 sum on CPU uses float16 accumulation type (https://github.com/pytorch/torchdynamo/pull/1452). We should use float32 accumulation type instead to reduce numerical errors.

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
How to perform unstructured interpolation 
Labels: triaged, module: interpolation

### 🚀 The feature, motivation and pitch

My feature request is very simple, I'm not sure if there already exists some approach or implementation to achieve this functionality.

In scipy, there is scipy.interpolate.NearestNDInterpolator class or scipy.interpolate.LinearNDInterpolator class to achieve unstructured interpolation, i.e., giving a set of sparse points that distribute non-uniformly in the spatial domain and interpolate values at any given points, however, currently torch seems only support simple grid structured interpolation like grid_sample

### Alternatives

I have found some implementation for this in 1d, but not sure if this is very efficient and support GPU, also how to extend it to 2D.

https://github.com/aliutkus/torchinterp1d

### Additional context

_No response_
Dynamo shouldn't name getitem variables getitem; instead it should derive the name from the variable that was getitem'ed
Labels: triaged, oncall: pt2

This would make return statements like

```
    return (self_encoder_dropout_1, getitem_5, getitem_10, getitem_13, getitem_16, getitem_19, getitem_22)
```

a lot more informative

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh
When you call tensor.size(), dynamo returns a tuple, instead of a torch.Size
Labels: triaged, oncall: pt2

I have this fragment of code, based off of the bug report https://github.com/pytorch/torchdynamo/issues/1462

```
         elif input_ids is not None:
              input_shape = input_ids.size()
              input_ids = input_ids.view(-1, input_shape[-1])
          elif inputs_embeds is not None:
              input_shape = inputs_embeds.size()[:-1]
          else:
              err_msg_prefix = "decoder_" if self.is_decoder else ""
              raise ValueError(f"You have to specify either {err_msg_prefix}input_ids or {err_msg_prefix}inputs_embeds  ")      
          print("early input shape: ", input_shape)
```

what I have done is inserted the print. When I run model normally, this print says

```
early input shape:  torch.Size([2, 1024])
```

however, when it's run under dynamo, I don't get a torch.Size

```
early input shape:  (2, 1024)
```

This is unexpected and can lead to behavior divergence.

I can try to minimize the repro if you like, but I'm guessing it is very obvious that we are not doing this correctly.

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh
torch.nn.functional.one_hot only works for int64
Labels: module: nn, triaged

### 🐛 Describe the bug

Tried applying torch.nn.functional.one_hot to an int32 and int16 tensor, got "[RuntimeError: one_hot is only applicable to index tensor.](https://github.com/pytorch/pytorch/blob/8d1ff9fc5dc70bdc65a83748c01cddf187728452/aten/src/ATen/native/Onehot.cpp#L6)"

It seems that it only works with int64 tensors. Is this behaviour expected? If so, why?

Thanks!

### Versions

PyTorch version: 1.12.1+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.22.6
Libc version: glibc-2.26

Python version: 3.7.14 (default, Sep  8 2022, 00:06:44)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.12.1+cu113
[pip3] torchaudio==0.12.1+cu113
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.13.1
[pip3] torchvision==0.13.1+cu113
[conda] Could not collect

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
MPSNDArray.mm:782: failed assertion; bufer is not large enough Mac M1 MPS
Labels: module: memory usage, triaged, module: regression, module: mps

### 🐛 Describe the bug

This appears a duplicate or a regression of #77886. I'm trying to use MPS with OpenAI's whisper. This had been failing but I saw that ` aten::index_put_impl ` was now supported so I tried again:

```{python}
import whisper
import torch
device = torch.device('mps')
model.transcribe("test dictation.m4a",language='english',verbose=True,fp16=False)


```

Which leads to a crash:
```
/usr/local/opt/miniforge3/lib/python3.9/site-packages/whisper/decoding.py:628: UserWarning: The operator 'aten::repeat_interleave.self_int' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
  audio_features = audio_features.repeat_interleave(self.n_group, dim=0)
/AppleInternal/Library/BuildRoots/a0876c02-1788-11ed-b9c4-96898e02b808/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSCore/Types/MPSNDArray.mm:782: failed assertion `[MPSNDArray, initWithBuffer:descriptor:] Error: buffer is not large enough. Must be 201452 bytes
```

I'm running `torch-1.13.0.dev20221003`



### Versions

```
Collecting environment information...
PyTorch version: 1.13.0.dev20221003
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.6 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.102)
CMake version: version 3.23.3
Libc version: N/A

Python version: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 17:00:33)  [Clang 13.0.1 ] (64-bit runtime)
Python platform: macOS-12.6-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.0
[pip3] torch==1.13.0.dev20221003
[pip3] torchaudio==0.13.0.dev20221003
[pip3] torchvision==0.14.0.dev20221003
[conda] numpy                     1.22.0           py39h61a45d2_0    conda-forge
[conda] torch                     1.13.0.dev20221003          pypi_0    pypi
[conda] torchaudio                0.13.0.dev20221003          pypi_0    pypi
[conda] torchvision               0.14.0.dev20221003          pypi_0    pypi
```

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
Debuggability++: Share instructions for building exotic CI configurations
Labels: module: ci, triaged

### 🚀 The feature, motivation and pitch

We know how to build PyTorch on various exotic CI configurations (@ezyang has them [documented internally](https://fb.workplace.com/groups/1274980366011614/posts/2317382101771430/?comment_id=2317519805090993)).

It would be great to make these more visible to people. In particular, if an exotic CI job fails, it would be good to link to instructions for how to build in that configuration directly. This could be done in HUD, or directly in the logfiles (although logfiles are at risk of not being seen by users.) Is anyone in Dev Infra interested in helping make this happen? I think it would be a great benefit for anyone working inside core PyTorch.

Practically, this would be two stages:
1. **Make the info available** externally. Perhaps copy/paste it to CONTRIBUTING.md or the wiki
2. **Surface the info** by sharing it in HUD under failing workflows 

### Why do this?
This is a usability improvement that will help people debug their failing builds, making them less likely to decide to `merge -f` a broken change

cc @seemethere @malfet @pytorch/pytorch-dev-infra
[TorchDispatch] Scalar Only Inputs Gets Matched To Tensor Schema
Labels: triaged, module: __torch_dispatch__

### 🐛 Describe the bug

When you call `torch.div` with scalar-only inputs, the schema that gets matched to is `aten::div.Tensor(Tensor self, Tensor other) -> Tensor`.  

```
import torch
from torch.utils._python_dispatch import TorchDispatchMode

class DebugMode(TorchDispatchMode):
    def __torch_dispatch__(self, func, types, args=(), kwargs=None):
        print(func, args, kwargs)
        return func(*args, **kwargs)

with DebugMode():
    print(torch.div(240, 8))
```
> aten.div.Tensor (240, 8) {}
tensor(30.)

I would expect the scalar inputs to get promoted to Tensors (lift_fresh) before the `aten.div` is invoked, or for a scalar schema to get matched to (probably the former). 

Also, I couldn't find the tests for scalar only inputs to Binary torch functions. 

Related: https://github.com/pytorch/torchdynamo/issues/1445

### Versions

master

cc @Chillee @ezyang @zou3519 @albanD @samdow
torch.jit.trace throwing Invalid name for qualified name eror 
Labels: oncall: jit

### 🐛 Describe the bug

While running `torch.jit.trace`, the function throws an error and explicitly asks to open a bug. Based on the name in the last line of the trace, it looks like something is wrong with the format, but I am not sure if this is a bug, as stated in the error message, or I can tweak something to change the name. 

Code: 
```
module = torch.jit.trace(model, test_input)
```

Trace 
```
  File "model_generator.py", line 89, in pytorch_to_fp32onnx
    module = torch.jit.trace(model, test_input)
  File "/home/hcsuser/envs/latest/lib/python3.7/site-packages/torch/jit/_trace.py", line 750, in trace
    _module_class,
  File "/home/hcsuser/envs/latest/lib/python3.7/site-packages/torch/jit/_trace.py", line 942, in trace_module
    module = make_module(mod, _module_class, _compilation_unit)
  File "/home/hcsuser/envs/latest/lib/python3.7/site-packages/torch/jit/_trace.py", line 568, in make_module
    return _module_class(mod, _compilation_unit=_compilation_unit)
  File "/home/hcsuser/envs/latest/lib/python3.7/site-packages/torch/jit/_trace.py", line 1072, in __init__
    tmp_module, lambda module: (), share_types=False, is_tracing=True
  File "/home/hcsuser/envs/latest/lib/python3.7/site-packages/torch/jit/_recursive.py", line 451, in create_script_module
    concrete_type = get_module_concrete_type(nn_module, share_types)
  File "/home/hcsuser/envs/latest/lib/python3.7/site-packages/torch/jit/_recursive.py", line 408, in get_module_concrete_type
    concrete_type = concrete_type_builder.build()
RuntimeError: atom.size() > 0INTERNAL ASSERT FAILED at "../aten/src/ATen/core/qualified_name.h":25, please report a bug to PyTorch. Invalid name for qualified name: '__torch__..model_wrapper.TigerViT'
```


### Versions

PyTorch version: 1.11.0+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2
Libc version: glibc-2.26

Python version: 3.7.5 (default, Dec  9 2021, 17:04:37)  [GCC 8.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-1083-azure-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: True
CUDA runtime version: 9.1.85
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.11.0
[pip3] torchvision==0.13.1
[conda] Could not collect
TransformerEncoder src_key_padding_mask does not work in eval()
Labels: high priority, triaged, oncall: transformer/mha

### 🐛 Describe the bug

Described here: https://discuss.pytorch.org/t/model-eval-predicts-all-the-same-or-nearly-same-outputs/155025/12

Starting with version 1.12, TransformerEncoder uses two different code paths, which perform differently between training and evaluation mode. In training, the `why_not_sparsity_fast_path` will be set, and the sparsity fast path will not be used. During eval(), if using a `src_key_padding_mask`, output will be modified:

https://github.com/pytorch/pytorch/blob/b04b2fa9aa52cacbdc9aaaf477d55b0af845ce81/torch/nn/modules/transformer.py#L271
```
            if (not why_not_sparsity_fast_path) and (src_key_padding_mask is not None):
                convert_to_nested = True
                output = torch._nested_tensor_from_mask(output, src_key_padding_mask.logical_not(), mask_check=False)
                src_key_padding_mask_for_layers = None
```

The output is usually some constant value. It seems like their might be some misalignment with the expected `src_key_padding_mask` and the `logical_not()` operation. Perhaps the mask inversion should not be performed.

### Versions

Python version: 3.8.13 (default, Mar 28 2022, 06:59:08) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22621-SP0
Is CUDA available: True
CUDA runtime version: 11.6.124
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 516.94
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] efficientnet-pytorch==0.7.1
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.3
[pip3] segmentation-models-pytorch==0.3.0
[pip3] torch==1.12.1+cu116
[pip3] torchaudio==0.12.1+cu116
[pip3] torchmetrics==0.9.3
[pip3] torchvision==0.13.1+cu116
[conda] efficientnet-pytorch      0.7.1                    pypi_0    pypi
[conda] numpy                     1.23.3                   pypi_0    pypi
[conda] segmentation-models-pytorch 0.3.0                    pypi_0    pypi
[conda] torch                     1.12.1+cu116             pypi_0    pypi
[conda] torchaudio                0.12.1+cu116             pypi_0    pypi
[conda] torchmetrics              0.9.3                    pypi_0    pypi
[conda] torchvision               0.13.1+cu116             pypi_0    pypi

cc @ezyang @gchanan @zou3519 @jbschlosser @bhosmer @cpuhrsch @erichan1
JIT fails to trace binary cross entropy with a strange error msg
Labels: oncall: jit

### 🐛 Describe the bug

JIT fails to trace binary cross entropy with a strange error msg

```py
import torch

def fn(input, target):
    weight = torch.tensor(64.0)
    reduction = "mean"
    return torch.nn.functional.binary_cross_entropy(input, target, weight=weight, reduction=reduction)

input = torch.tensor([0.1, 0.1, 0.1], dtype=torch.float32)
target = torch.tensor([0.15, 0.15, 0.15], dtype=torch.float32)

inputs = (input, target)

print(fn(*inputs))

new_fn = torch.jit.trace(fn, inputs)
print(new_fn(*inputs))
```

```
tensor(27.8364)

RuntimeError: expected int at position 0, but got: Tensor
```

I think the input here is valid and JIT should succeed to trace it

### Versions

```
PyTorch version: 1.13.0.dev20221001
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.2.0-19ubuntu1) 11.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.13 (main, Aug 25 2022, 23:26:10)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.19.5-051905-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.7.99
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060
Nvidia driver version: 515.65.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.3
[pip3] torch==1.13.0.dev20221001
[pip3] torchaudio==0.13.0.dev20221001
[pip3] torchvision==0.14.0.dev20221001
[conda] blas                      1.0                         mkl  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.23.3                   pypi_0    pypi
[conda] numpy-base                1.23.1           py39ha15fc14_0  
[conda] pytorch                   1.13.0.dev20221001 py3.9_cuda11.7_cudnn8.5.0_0    pytorch-nightly
[conda] pytorch-cuda              11.7                 h67b0de4_0    pytorch-nightly
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] torch                     1.13.0.dev20220923+cu116          pypi_0    pypi
[conda] torchaudio                0.13.0.dev20220923+cu116          pypi_0    pypi
[conda] torchvision               0.14.0.dev20220923+cu116          pypi_0    pypi
```
`cdist` should succeed when `p` is integer in JIT
Labels: oncall: jit

### 🐛 Describe the bug

`cdist` should succeed when `p` is integer in JIT. Actually, the [doc](https://pytorch.org/docs/stable/generated/torch.cdist.html) of `cdist` doesn't mention that `p` must be float and the non-JIT  function will succeed with integer `p`. However, the JIT version will fail with "Expected a value of type 'float' for argument 'p' but instead found type 'int'."

```py
import torch

def fn(x1, x2):
    return torch.cdist(x1, x2, p=2)

x1 = torch.rand([2, 2], dtype=torch.float64)
x2 = torch.rand([2, 2], dtype=torch.float64)

print(fn(x1, x2))

jit_fn = torch.jit.script(fn)
print(jit_fn(x1, x2))
```

```
tensor([[0.4998, 0.8013],
        [0.8789, 0.8492]], dtype=torch.float64)

RuntimeError:
cdist(Tensor x1, Tensor x2, float p=2., str compute_mode="use_mm_for_euclid_dist_if_necessary") -> Tensor:
Expected a value of type 'float' for argument 'p' but instead found type 'int'.
```

### Versions

```
PyTorch version: 1.13.0.dev20221001
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.2.0-19ubuntu1) 11.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.13 (main, Aug 25 2022, 23:26:10)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.19.5-051905-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.7.99
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060
Nvidia driver version: 515.65.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.3
[pip3] torch==1.13.0.dev20221001
[pip3] torchaudio==0.13.0.dev20221001
[pip3] torchvision==0.14.0.dev20221001
[conda] blas                      1.0                         mkl  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.23.3                   pypi_0    pypi
[conda] numpy-base                1.23.1           py39ha15fc14_0  
[conda] pytorch                   1.13.0.dev20221001 py3.9_cuda11.7_cudnn8.5.0_0    pytorch-nightly
[conda] pytorch-cuda              11.7                 h67b0de4_0    pytorch-nightly
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] torch                     1.13.0.dev20220923+cu116          pypi_0    pypi
[conda] torchaudio                0.13.0.dev20220923+cu116          pypi_0    pypi
[conda] torchvision               0.14.0.dev20220923+cu116          pypi_0    pypi
```
When will the torch.sparse module be usable?
Labels: module: sparse, triaged

### 🚀 The feature, motivation and pitch

Hello, 

I tried to use the ```torch.sparse``` package for creating neural networks but it utterly failed due to so many 
errors, incompatinilities and generally the error message.
```python
NotImplementedError: Could not run 'aten::is_coalesced' with arguments from the 'CPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::is_coalesced' is only available for these backends: [SparseCPU, SparseCUDA, BackendSelect, Python, Named, Conjugate, Negative, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradLazy, AutogradXPU, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, UNKNOWN_TENSOR_TYPE_ID, Autocast, Batched, VmapMode].
```
Even when using the functions from [this source](https://pytorch.org/docs/stable/sparse.html)

This includes: 
- batch addition and multiplication
- activation functions such as ReLU or Sigmoid on sparse tensors
- even creating simple loss functions gives an error with ```torch.sparse.sum``` gives an error

When can we be expecting to have a working version of ```torch.sparse``` that can be used for actually modelling or do we have to stay with tensorflow?

### Alternatives

Use tensorflow

### Additional context

_No response_

cc @nikitaved @pearu @cpuhrsch @amjames @bhosmer