gopikrsmscs/torch-issues huggingface dataset:
[feature request] make the input k in rot90 a list of int to rotate tensors individually in a batch
Labels: feature, triaged, module: python frontend

### 🚀 The feature, motivation and pitch

I'm using torch.rot90(input, k=1, dims=[0, 1]) to rotate a batch sample. Since the k here should be an int, which means all the samples in a batch will be rotated with the same angle (because they share the same k). What I want to do is to rotate each sample of the batch separately because this will improve the accuracy of my task.

Is it possible to extend rot90 to make the input k a list of int, to make it possible to rotate tensors individually in a batch? 

### Alternatives

A simple alternative to achieve the function described above is to use the for loop: 

```python
import numpy as np
import torch

batch_sample = torch.randn(10, 1, 32, 32, 129)
for i in range(len(batch_sample)):
    k = np.random.choice(4, 1, replace=False)[0]
    batch_sample[i, :] = torch.rot90(batch_sample[i, :], k=k, dims=[1,2])
```

But it is slow using the for loop. So, I'm wondering if it is possible to extend rot90 to make the input k a list of int, making it faster to rotate tensors individually in a batch.


### Additional context

_No response_

cc @albanD
Support Delay Loading of c10.dll in when using libtorch as a thirdparty library.
Labels: module: windows, module: abi, triaged

### 🚀 The feature, motivation and pitch

Hi PyTorch team,

I'm currently working on a project where I have added libtorch as a third-party dependency. Due to the size of libtorch, I wanted to make it an optional dependency using delay load hooks. However, I've encountered an issue with c10.dll, which is a dependency of libtorch. It seems that c10.dll exports a global variable, causing it to be unable to be delay loaded.

Here's the error message I encountered
![image](https://github.com/pytorch/pytorch/assets/7751507/0e15b927-c16f-411d-a399-72787387771e)

I'm wondering if it's possible to add support for delay loading c10.dll in libtorch. Having this feature would be really helpful for my project, as it will allow me to better manage the dependency loading and reduce the initial load time.

### Alternatives

_No response_

### Additional context

_No response_

cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite
Multiple dimensions support for `torch.max`
Labels: feature, triaged, module: numpy, needs design, module: python frontend

### 🚀 The feature, motivation and pitch

Numpy's  `np.max` has support for multiple dimensions, but `torch.max` does not.
To keep both synced and avoid implementing custom function like #105045, it would be really helpful to have this.

### Alternatives

#105045

### Additional context
@davidradl

cc @mruberry @rgommers @albanD
`assert has_same_metadata(inpt_new, inpt_old)` fails when capturing forwards + backwards in train_step with resnet18
Labels: triaged, oncall: pt2, module: functorch

### 🐛 Describe the bug

```
import torch
from torchvision.models import resnet18

torch.set_default_device('cuda')

model = resnet18()
optim = torch.optim.Adam(model.parameters(), lr=0.01)
inp = torch.randn(1, 3, 224, 224)

@torch.compile
def train_step():
    model(inp).sum().backward()
    optim.step()

train_step()
train_step()
```

```
 File "/scratch/chilli/fresh/pytorch/torch/fx/_symbolic_trace.py", line 817, in trace
    (self.create_arg(fn(*args)),),
  File "/scratch/chilli/fresh/pytorch/torch/fx/experimental/proxy_tensor.py", line 485, in wrapped
    out = f(*tensors)
  File "<string>", line 1, in <lambda>
  File "/scratch/chilli/fresh/pytorch/torch/_functorch/aot_autograd.py", line 1363, in fwd_helper
    return functionalized_f_helper(*args)
  File "/scratch/chilli/fresh/pytorch/torch/_functorch/aot_autograd.py", line 1352, in functionalized_f_helper
    assert has_same_metadata(inpt_new, inpt_old)
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
AssertionError: 
```

### Versions

N/A

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @zou3519 @samdow @kshitij12345 @janeyx99
DISABLED test_homogeneous_attributes (__main__.TestFSDPMiscMultiThread)
Labels: oncall: distributed, module: flaky-tests, skipped, module: fsdp

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_homogeneous_attributes&suite=TestFSDPMiscMultiThread) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/14960856221).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 6 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_homogeneous_attributes`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `distributed/fsdp/test_fsdp_misc.py`

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
DISABLED test_compile_vmap_hessian_cuda (__main__.TestCompileTransformsCUDA)
Labels: module: rocm, triaged, skipped

Platforms: rocm

Disabling this unit test since it showed up in our local CI testing for ROCm5.6 CI upgrade PR: https://github.com/pytorch/pytorch/pull/103092

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @BLOrange-AMD
Move tools/autograd to torchgen/autograd
Labels: fb-exported, Stale, release notes: releng, suppress-bc-linter

Summary:
Performed with:
```
hg move tools/autograd torchgen/autograd
sed -i 's$tools/autograd$torchgen/autograd$g' $(find . -type f)
sed -i 's$tools.autograd$torchgen.autograd$g' $(find . -type f)
# + some manual modification to tools/setup_helpers/generate_code.py and BUCK files
```

Test Plan: Existing tests

Differential Revision: D47375735
[export] tensor creation ops burn in device
Labels: triaged, oncall: pt2, module: export

```
import torch
import torch._export

def foo(x):
    return x + torch.ones(2, 2)

e = torch._export.export(foo, (torch.ones(2, 2),))
print(e.graph_module.graph)
```
produces:
```
graph():
    %arg0_1 : [num_users=3] = placeholder[target=arg0_1]
    %sym_size_int : [num_users=1] = call_function[target=torch.ops.aten.sym_size.int](args = (%arg0_1, 0), kwargs = {})
    %sym_size_int_1 : [num_users=1] = call_function[target=torch.ops.aten.sym_size.int](args = (%arg0_1, 1), kwargs = {})
    %eq : [num_users=1] = call_function[target=operator.eq](args = (%sym_size_int_1, 2), kwargs = {})
    %scalar_tensor_default : [num_users=1] = call_function[target=torch.ops.aten.scalar_tensor.default](args = (%eq,), kwargs = {})
    %_assert_async_msg : [num_users=0] = call_function[target=torch.ops.aten._assert_async.msg](args = (%scalar_tensor_default, Input arg0_1.shape[1] is specialized at 2), kwargs = {})
    %eq_1 : [num_users=1] = call_function[target=operator.eq](args = (%sym_size_int, 2), kwargs = {})
    %scalar_tensor_default_1 : [num_users=1] = call_function[target=torch.ops.aten.scalar_tensor.default](args = (%eq_1,), kwargs = {})
    %_assert_async_msg_1 : [num_users=0] = call_function[target=torch.ops.aten._assert_async.msg](args = (%scalar_tensor_default_1, Input arg0_1.shape[0] is specialized at 2), kwargs = {})
    %full_default : [num_users=1] = call_function[target=torch.ops.aten.full.default](args = ([2, 2], 1), kwargs = {dtype: torch.float32, layout: torch.strided, device: cpu, pin_memory: False})
    %add_tensor : [num_users=1] = call_function[target=torch.ops.aten.add.Tensor](args = (%arg0_1, %full_default), kwargs = {})
    return (add_tensor,)
```

Note the following line burns in `device=cpu`.
```
    %full_default : [num_users=1] = call_function[target=torch.ops.aten.full.default](args = ([2, 2], 1), kwargs = {dtype: torch.float32, layout: torch.strided, device: cpu, pin_memory: False})
```

These kwargs are optional; I'd like for them to not show up in the graph if the user didn't specify them.



cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
[not ready for review yet] torch.compile support for parseSemiStructuredTensor
Labels: release notes: sparse

cc @jcaip, this is an E2E test of compiling a small model with a `SparseSemiStructuredTensor` subclass tensor used as one of the parameters.

The generated inductor code looks like this (P788647425): you can see that the subclass desugars the matmul into a sparse_mm() + contiguous() call, and inductor is able to fuse the contiguous() call into the relu() that follows it.

A few things to note:

(1) the test actually... fails. I haven't figured out why, but the results don't match the non-sparse version. FWIW, the code that inductor outputs looks reasonable, so it's not immediately clear if it's a compile() bug, or something to do with sparsity giving less accurate results (or both).
(1) inference mode is a bit broken with torch.compile still: in the test, i needed to make sure that the model was instantiated, compiled and run all inside of inference_mode.


Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #104974
* #111011
Add a diagram showing the code structure to CONTRIBUTING.md 
Labels: module: docs, triaged

### 📚 The doc issue

Add a diagram showing the code structure to CONTRIBUTING.md 



### Suggest a potential alternative/fix

I will supply a diagram showing the simplified important parts of the code structure. I will put it in the top level folder - if it should be moved to a sub folder please let me know.  

cc @svekars @carljparker
ONNX export process  failed to keep consistence of input_names specified
Labels: module: onnx, triaged

### 🐛 Describe the bug

When I use the torch.onnx.export to make the ONNX format model, the result onnx model, when loaded with onnxruntime, is failed to keep the input_names identical with my settings.

eg.
```
import torch
import torch.nn as nn

class MyModel(nn.Model):
    def __init__(self):
        super(MyModel, self).__init__()
        self.layer = nn.Linear(10,  1)

    def forward(self, input_a, input_b=None):    # the input_b is not actually used in the forward computation 
         return self.layer(input_a)

#  Here we export the trained model!  the file "MyModel_checkpoint_file.pt" is the training result
my_model = torch.load("MyModel_checkpoint_file.pt")
my_model.eval()

input_names = ["input_a", "input_b"]
output_names = ["output"]
fake_input_a = torch.rand(10)
fake_input_b = torch.rand(10)
with torch.no_grad():
    torch.onnx.export(
         model=my_model,
         args = (fake_input_a, fake_input_b),
         f = "result.onnx",
         input_names = input_names,
         output_names = output_names
    )
```
Here I got a "result.onnx”  model file,  I specified that it should take two inputs, names "input_a", and "input_b";   although in fect the parameter "input_b" is not really used  in the graph.

But when I load the onnx model with onnxruntime

```
import onnxruntime
session = onnxruntime.InferenceSession("result.onnx", providers=["CPUExecutionProvider"])
for input in session.get_inputs():
    print(input.name)
```
the result  output string is only "input_a",  No "input_b" at All.

So there is an dis-consistence  between the exportation and inference.

I do not known  the detail of the onnx conversion, I guess that this process may have removed the input items that not really used in the final onnx graph.  But in my  view,  I would like to keep the input_names just as I had specified, even if it is not really used. So that in the software project using this model, the inference interface is fixed as I had designed. 

### Versions

python == 3.8.5
torch  == 1.10.2+cu111
onnxruntime == 1.14.1
torch version compare
Labels: triaged, module: python frontend

### 🐛 Describe the bug

When I install pytorch from source code, I got torch version '2.0.0a0+gitc263bd4',  but 'torch.torch_version.TorchVersion('2.0.0a0+gitc263bd4') >= (2, 0, 0)' return false.  Torch installed from whl will show version '2.0.0+cpu' and
'torch.torch_version.TorchVersion('2.0.0+cpu') >= (2, 0, 0)' is correct.

```
import torch
from torch.torch_version import TorchVersion
torch.torch_version.TorchVersion('2.0.0a0+gitc263bd4') >= (2, 0, 0)
```

### Versions

2.0.0a0+gitc263bd4

cc @albanD
torch.jit.frontend.NotSupportedError: keyword-arg expansion is not supported: for dgl.nn.HeteroGraphConv()
Labels: oncall: jit

### 🚀 The feature, motivation and pitch

Hello,

I get this error once I call the following line after my model is trained.

```
model_scripted = torch.jit.script(model) # Export to TorchScript
model_scripted.save('model_scripted.pt') # Save

```

my model has a HeteroGraphConv layer that uses dictionary-based arguments (two node types). However, TorchScript has limitations compared to dynamic Python execution, and keyword argument expansion is one of the features not supported in TorchScript. 

Not sure if it's a bug or a feature to be developed, but NotSupportedError made me think it's a feature request.

Thanks!

### Alternatives

_No response_

### Additional context

```
torch.jit.frontend.NotSupportedError: keyword-arg expansion is not supported:
  File "../opt/anaconda3/lib/python3.9/site-packages/dgl/nn/pytorch/hetero.py", line 178
                    (src_inputs[stype], dst_inputs[dtype]),
                    *mod_args.get(etype, ()),
                    **mod_kwargs.get(etype, {}))
                      ~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
                outputs[dtype].append(dstdata)
        else:
```

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Refactor Adam and AdamW by abstracting out common code
Labels: module: optimizer, triaged, better-engineering, actionable

AdamW differs with Adam only in the weight_decay handling. Everything else is the same. We should reuse code instead of hosting the same exact logic in two places.

One way to do this is to have certain functions in adam.py that can be accessed in adamw.py. The main work that would need to be done would be consolidating the forloop and foreach implementations. The fused implementation already takes advantage of the commonality. 

cc @vincentqb @jbschlosser @albanD @crcrpar
[dynamo][higher_order_op] assert in check_kwargs leads to error instead of graph-break
Labels: good first issue, triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

https://github.com/pytorch/pytorch/blob/e695b397e16e46e17481ab26517c9713a62e8c4a/torch/_dynamo/variables/higher_order_ops.py#L223-L227

cc: @zou3519 



### Versions

master

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @aakhundov
DISABLED test_custom_op_cuda_cuda_wrapper (__main__.TestCudaWrapper)
Labels: triaged, module: flaky-tests, skipped, module: inductor

Platforms: linux, slow

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_custom_op_cuda_cuda_wrapper&suite=TestCudaWrapper) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/14905256174).

Over the past 3 hours, it has been determined flaky in 3 workflow(s) with 9 failures and 3 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_custom_op_cuda_cuda_wrapper`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `inductor/test_cpp_wrapper.py`

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8
errors in CONTRIBUTING.md
Labels: module: docs, triaged

In [CONTRIBUTING.md](https://github.com/pytorch/pytorch/blob/main/CONTRIBUTING.md) it says 

```
Install clang-tidy driver script dependencies

pip3 install -r tools/linter/clang_tidy/requirements.txt
Run clang-tidy
# Run clang-tidy on the entire codebase
make clang-tidy
# Run clang-tidy only on your changes
make clang-tidy CHANGED_ONLY=--changed-only
```
requirements.txt does not exist in the source tree

When I ran `make clang-tidy` from the root folder - I get
```
pytorch % make clang-tidy
make: *** No rule to make target 'clang-tidy'.  Stop.
```
I says to install the LLVM 8 binaries. It would be useful to add in the docs why such a back level version is needed; as there have been 8 releases since this.  It might be this was just the level that was tested to have worked. 



cc @svekars @carljparker
Conversion of a CSR tensor with batches to COO tensor fails
Labels: module: sparse, triaged

## Issue description

As in the title.

## Code example

```python
>>> torch.tensor([[[0, 1], [2, 3]], [[4, 5], [6, 0]]]).to_sparse_csr().to_sparse()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: crow_indices is supposed to be a vector, but got 2 dimensional tensor.
```

The expected result is the same as in the following example:
```python
>>> torch.tensor([[[0, 1], [2, 3]], [[4, 5], [6, 0]]]).to_sparse_csr().to_dense().to_sparse()
tensor(indices=tensor([[0, 0, 0, 1, 1, 1],
                       [0, 1, 1, 0, 0, 1],
                       [1, 0, 1, 0, 1, 0]]),
       values=tensor([1, 2, 3, 4, 5, 6]),
       size=(2, 2, 2), nnz=6, layout=torch.sparse_coo)
```


## More information

The exception is raised from `convert_indices_from_csr_to_coo` which does not implement batched CSR tensor support. A number of tests (to_sparse tests, reduction tests, binary operation tests) disable the corresponding samples meaning that the issue has existed for a long time.

Now, while adding sparse samples to TestGradients methods in order to enable gradcheck tests for operations with sparse inputs, I realized that there is no straightforward way to disable batched sparse samples for these tests without disabling all sparse samples. So, the issue blocks my task. I suggest resolving it, especially because supporting batched crow_indices in `convert_indices_from_csr_to_coo` ought to be a straightforward task.

## System Info

- PyTorch version: main


cc @alexsamardzic @nikitaved @cpuhrsch @amjames @bhosmer
DISABLED test_custom_op_cuda (__main__.CudaTests)
Labels: triaged, module: flaky-tests, skipped, oncall: pt2, module: inductor

Platforms: inductor

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_custom_op_cuda&suite=CudaTests) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/14894386161).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 6 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_custom_op_cuda`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `inductor/test_torchinductor.py`

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8
DISABLED test_custom_op_cpu_dynamic_shapes_cpp_wrapper (__main__.DynamicShapesCppWrapperCpuTests)
Labels: triaged, module: flaky-tests, skipped, module: inductor

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_custom_op_cpu_dynamic_shapes_cpp_wrapper&suite=DynamicShapesCppWrapperCpuTests) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/14894136800).

Over the past 3 hours, it has been determined flaky in 4 workflow(s) with 12 failures and 4 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_custom_op_cpu_dynamic_shapes_cpp_wrapper`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `inductor/test_cpp_wrapper.py`

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8
torch.norm inconsistency?
Labels: module: numerical-stability, triaged, module: norms and normalization

### 🐛 Describe the bug

```
import torch as ch

a = ch.randn(2, int(1e6))
# The following should be equivalent
print(a[0].norm())
print(a.norm(dim=1)[0])

```
===

```
tensor(999.3896)
tensor(999.1658)
```

### Versions

Collecting environment information...
PyTorch version: 1.12.0
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: Ubuntu 16.04.6 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.9
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: NVIDIA Quadro K620
GPU 1: NVIDIA TITAN X (Pascal)
GPU 2: NVIDIA TITAN X (Pascal)
GPU 3: NVIDIA TITAN X (Pascal)
GPU 4: NVIDIA TITAN X (Pascal)

Nvidia driver version: 465.19.01
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] torch==1.12.0
[pip3] torchaudio==0.12.0
[pip3] torchvision==0.13.0
[conda] blas                      1.0                         mkl    anaconda
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] libblas                   3.9.0            12_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            12_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            12_linux64_mkl    conda-forge
[conda] mkl                       2021.4.0           h06a4308_640    anaconda
[conda] mkl-service               2.4.0            py39h7e14d7c_0    conda-forge
[conda] pytorch                   1.12.0          py3.9_cuda10.2_cudnn7.6.5_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.12.0               py39_cu102    pytorch
[conda] torchvision               0.13.0               py39_cu102    pytorch
[feature request] torch.mix function to generalize/symmetrize addcmul
Labels: feature, triaged, needs research, module: python frontend

### 🚀 The feature, motivation and pitch

I propose to introduce a convenience/ux function in core:
`mix = lambda a = 1, b = 1, c = 1, A = 1, B = 1, C = 1: a*b*c + A*B*C` in one go where all a,b,c,A,B,C can be scalars or tensors. The most frequent cases would be:
- a, A, b, B are scalars, c and C are tensors
- a, A, b are scalars, c, B, C are tensors

This would generalize/symmetrize torch.addcmul, and could be used in optimizers to make second moment calculations more idiomatic and clear. This function could be implemented using torch.compile in core. Also an inplace version is needed (kind of generalization of torch.lerp_)

Instead of `mix` (as `mix` in graphics context often means lerp: https://registry.khronos.org/OpenGL-Refpages/gl4/html/mix.xhtml) , the name could also be `linear_combination` (?) (although if B=C or other cases, it would not actually be a linear combination) Or maybe existing `addcmul` could be extended to support more arguments. The inplace mode of mix would be different from existent addcmul in that it should allow inplace multiplication of the accumulator by scalar/tensor prior to adding the rhs summand.

Some older context: https://github.com/pytorch/pytorch/issues/71683#issuecomment-1379058082 https://github.com/pytorch/pytorch/issues/79352#issuecomment-1152946076 https://github.com/pytorch/pytorch/pull/104781#issuecomment-1625709917

Here is that line from Adam https://github.com/pytorch/pytorch/blob/main/torch/optim/adam.py#L363:
```python
exp_avg_sq.mul_(beta2).addcmul_(grad, grad.conj(), value=1 - beta2)
# exp_avg.lerp_(grad, 1 - beta1)
```

Maybe another alternative for design is `lerp2` : `exp_avg_sq.lerp2_(grad, grad.conj(), 1 - beta2)`, but IMO generalized `addcmul`/`addcmul2`/`linear_combination` is still better


cc @albanD
Implement `diag` method for sparse COO tensors
Labels: module: sparse, triaged

### 🚀 The feature, motivation and pitch

## Feature
To implement `diag` method for sparse coo tensors. Currently it raises `NotImplementedError`.

## Motivation
Given a 1-d sparse tensor, I want to make a diagonal out of it to use it in matrix multiplications then.

### Alternatives

Given 1-d sparse tensor `t`, I do:

```python
torch.sparse_coo_tensor(t.indices().repeat(2, 1), t.values())
```

Thankfully, this alternative preserves grad operations, if `t.requires_grad=True`.


### Additional context

_No response_

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer
MPS matmul with sliced (strided) out argument produces wrong output, may corrupt memory
Labels: triaged, module: mps

### 🐛 Describe the bug

When using `torch.matmul` with an `out` argument that is a view that is not contiguous, the op has no change to the `out` argument instead of computing the matmul correctly. Here is a minimal example that replicates the behavior. It seems to happen consistently across many different sizes (i.e. the `256` here can be changed to something else and it still outputs zeros).

```
import torch

X = torch.randn(256,256).to("mps")
Y = torch.randn(256,256).to("mps")
Z = torch.zeros(256,2*256).to("mps")

torch.matmul(X,Y,out=Z[:,128:(128+256)])
```

Here, `torch.matmul` both outputs all zeros and does not change `Z`. Compare the behavior of the non-MPS code

```
import torch

X = torch.randn(256,256)
Y = torch.randn(256,256)
Z = torch.zeros(256,2*256)

torch.matmul(X,Y,out=Z[:,128:(128+256)])
```

which runs correctly and produces a non-zero output.

After doing this, I sometimes get segfaults in my program, which suggests that this is also corrupting memory, although I am unable to find any code that consistently replicates this segfault behavior.

### Versions

Collecting environment information...
PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.4 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: version 3.26.3
Libc version: N/A

Python version: 3.9.13 (main, Aug 25 2022, 18:24:45)  [Clang 12.0.0 ] (64-bit runtime)
Python platform: macOS-13.4-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M2 Max

Versions of relevant libraries:
[pip3] numpy==1.24.0
[pip3] pytorch-pretrained-bert==0.6.2
[pip3] torch==2.0.1
[pip3] torchaudio==0.13.1
[pip3] torchdata==0.6.1
[pip3] torchtext==0.15.2
[pip3] torchvision==0.13.1
[conda] htorch                    0.1                      pypi_0    pypi
[conda] numpy                     1.24.0                   pypi_0    pypi
[conda] pytorch-pretrained-bert   0.6.2                    pypi_0    pypi
[conda] torch                     2.0.1                    pypi_0    pypi
[conda] torchaudio                0.13.1                 py39_cpu    pytorch
[conda] torchdata                 0.6.1                    pypi_0    pypi
[conda] torchtext                 0.15.2                   pypi_0    pypi
[conda] torchvision               0.13.1                   pypi_0    pypi

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
[RFC] Let in-place foreach functions return a list of Tensors
Labels: triaged, module: mta

Currently in-place foreach functions are a void function as you can see in the symbols in native_functions.yaml such as https://github.com/pytorch/pytorch/blob/3d071849300048a50555391874ab0d663a074d23/aten/src/ATen/native/native_functions.yaml#L9919.

This is mainly because of the limitation of torchgen/model here: https://github.com/pytorch/pytorch/blob/d8cb80e3827226eeaafd6a007018112301f89d41/torchgen/model.py#L1411-L1428

Even after enabling in-place foreach to return a TensorList, method chaining is infeasible. I however expect it to make the behavior a little bit more intuitive.

rel:
- https://github.com/pytorch/pytorch/pull/104780#discussion_r1256289461
- https://github.com/pytorch/pytorch/issues/58833

cc @mcarilli @albanD @soulitzer @janeyx99
[compile][dynamic] dsplit is seeing a list of mixed ints and symints
Labels: triaged, oncall: pt2, module: dynamic shapes

### 🐛 Describe the bug

```
import torch

@torch.compile(dynamic=True)
def fn(x, sections):
    return torch.dsplit(x, sections)


fn(torch.randn(4, 4, 4), [1,2,3])
```

```
  File "/scratch/anijain/work/pytorch/torch/_dynamo/utils.py", line 890, in wrap_fake_exception
    return fn()
  File "/scratch/anijain/work/pytorch/torch/_dynamo/utils.py", line 1301, in <lambda>
    lambda: run_node(tx.output, node, args, kwargs, nnmodule)
  File "/scratch/anijain/work/pytorch/torch/_dynamo/utils.py", line 1366, in run_node
    raise RuntimeError(fn_str + str(e)).with_traceback(e.__traceback__) from e
  File "/scratch/anijain/work/pytorch/torch/_dynamo/utils.py", line 1353, in run_node
    return node.target(*args, **kwargs)
torch._dynamo.exc.TorchRuntimeError: Failed running call_function <built-in method dsplit of type object at 0x7fc96edb3fa0>(*(FakeTensor(..., size=(s0, s0, s0)), [1, s1, s2]), **{}):
dsplit() received an invalid combination of arguments - got (FakeTensor, immutable_list), but expected one of:
 * (Tensor input, int sections)
      didn't match because some of the arguments have invalid types: (FakeTensor, immutable_list of [int, SymInt, SymInt])
 * (Tensor input, tuple of ints indices)
      didn't match because some of the arguments have invalid types: (FakeTensor, immutable_list of [int, SymInt, SymInt])


from user code:
   File "/scratch/anijain/work/pytorch/examples/dsplit.py", line 5, in fn
    return torch.dsplit(x, sections)

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information


You can suppress this exception and fall back to eager by setting:
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True
```

cc @ezyang @msaroufim @wconstab @bdhirsh 

### Versions

N/A
Regression in Dalle2 due to dynamic shapes
Labels: triaged, oncall: pt2, module: dynamic shapes

### 🐛 Describe the bug

See regression in Dalle2_Pytorch [here](https://hud.pytorch.org/benchmark/torchbench/inductor_with_cudagraphs?startTime=Fri,%2030%20Jun%202023%2020:02:40%20GMT&stopTime=Fri,%2007%20Jul%202023%2020:02:40%20GMT&granularity=hour&mode=inference&dtype=bfloat16&lBranch=main&lCommit=54e320d4d1654d45b65c3a5a1ba8fe35faf21460&rBranch=main&rCommit=7ae100628ec530e1da7bd5e5f86024afa8843a32).

And corresponding Error from [log](https://ossci-raw-job-status.s3.amazonaws.com/log/14785878501):

```
2023-07-05T09:35:43.8510365Z ERROR:common:Failed running call_function <function rearrange at 0x7f524b7b6ef0>(*(FakeTensor(..., device='cuda:0', size=(4, 128, s0, s1)), 'b c ... -> b ... c'), **{}):
2023-07-05T09:35:43.8511611Z unhashable type: 'SymInt'
2023-07-05T09:35:43.8515016Z 
2023-07-05T09:35:43.8515940Z from user code:
2023-07-05T09:35:43.8517079Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 1670, in forward
2023-07-05T09:35:43.8517834Z     h = rearrange(h, 'b c ... -> b ... c')
2023-07-05T09:35:43.8518130Z 
```

### Error logs

_No response_

### Minified repro

_No response_

### Versions

master

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
have inductor fallback for fp16.view(dtype=torch.int16)
Labels: Stale, module: inductor, ciflow/inductor

This is a potential fix for https://github.com/pytorch/pytorch/issues/104791. Totally open to suggestions (should we not land this, and instead figure out how to get `tmp0.to(tl.int16, bitcast=True)` to work in triton?) - but this PR detects the case mentioned in the issue and falls back to eager necessary.

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #104793
* #106381



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @ngimel
inductor/triton fails on `view(..., dtype=torch.int16)`
Labels: triaged, oncall: pt2, module: inductor

It looks like triton can't handle bitcasting `float16` to `int16`. This example works on cpu though. Repro:
```
import torch

@torch.compile
def f(x):
    x_view = x.view(dtype=torch.int16)
    return x_view.mul(2)

x = torch.ones(4, dtype=torch.float16, device='cuda')
out = f(x)
```

output:
```
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
CompilationError: at 8:19:def triton_(in_ptr0, out_ptr0, xnumel, XBLOCK : tl.constexpr):
    xnumel = 4
    xoffset = tl.program_id(0) * XBLOCK
    xindex = xoffset + tl.arange(0, XBLOCK)[:]
    xmask = xindex < xnumel
    x0 = xindex
    tmp0 = tl.load(in_ptr0 + (x0), xmask).to(tl.float32)
    tmp1 = tmp0.to(tl.int16, bitcast=True)
                   ^
ValueError('Cannot bitcast data-type of size 32to data-type of size 16')
```

One question: should triton be able to handle this? Otherwise, maybe we can change the lowering for `view.dtype` to fall back to eager when it sees an fp16->int16 conversion.


I ran into this when trying to use @jcaip 's `SparseSemiStructuredTensor` subclass prototype with torch.compile.

cc @ezyang @msaroufim @wconstab @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8
[BE] Evaluate and improve eager for-loop optimizer memory perf
Labels: module: optimizer, triaged, better-engineering, actionable

We are not the most careful with using the least amount of memory in our for-loop implementations of our optimizers. For example, we have `max_exp_avg_sqs[i].copy_(torch.maximum(max_exp_avg_sqs_i, exp_avg_sq))` in adam.py which could be `torch.maximum(max_exp_avg_sqs_i, exp_avg_sq, out = max_exp_avg_sqs[i])` to save an allocation. Motivated by this comment: https://github.com/pytorch/pytorch/pull/104781#issuecomment-1625621945

This issue is to track the task of going through each of our for-loop (single_tensor) optimizers and ensuring that we are clean and minimal with our memory footprint.
- [ ] Adam
- [ ] AdamW
- [ ] Adagrad
- [ ] Adadelta
- [ ] Adamax
- [ ] NAdam
- [ ] RAdam
- [ ] ASGD
- [ ] SGD
- [ ] Rprop
- [ ] RMSprop
- [ ] LBFGS
- [ ] SparseAdam

cc @vincentqb @jbschlosser @albanD
Use `isinstance` instead of `type` when checking for `torch.nn.Parameter`
Labels: module: nn, triaged

### 🚀 The feature, motivation and pitch

From comment https://github.com/pytorch/pytorch/pull/104069?notification_referrer_id=NT_kwDOAhpHxbM2ODUxNzMzMDg5OjM1Mjc2NzQx#discussion_r1256262813

```
p = torch.nn.Parameter(t)
type(p) is torch.nn.Parameter
```

only works if `t` (the data passed to nn.Parameter) is a regular `torch.Tensor` and not a tensor subclass. However `isinstance(p, torch.nn.Parameter)` will work for both regular tensors and tensor subclasses. We should update all instances of type(t) is Parameter to be `isinstance(t, Parameter)` within PyTorch core.



### Alternatives

_No response_

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr
Subgraph matcher returned a false match
Labels: triaged, module: fx, oncall: pt2

I ran into a "bug" where the subgraph matcher returned a match, but the matched pattern doesn't actually exist in the original model. This resulted in the following error in subgraph rewriter:

```
  File "/home/andrewor/local/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/fx/subgraph_rewriter.py", line 218, in replace_pattern_with_filters
    return _replace_pattern(gm, pattern, replacement, match_filters, ignore_literals)
  File "/home/andrewor/local/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/fx/subgraph_rewriter.py", line 326, in _replace_pattern
    gn = match.nodes_map[node]
KeyError: div__scalar
```

Matched pattern looks like this (doesn't exist in the original model):
```
def forward(self, x):
    arg0, = fx_pytree.tree_flatten_spec(([x], {}), self._in_spec)
    empty_like_default = torch.ops.aten.empty_like.default(arg0, memory_format = torch.contiguous_format)
    bernoulli__float = torch.ops.aten.bernoulli_.float(empty_like_default);  empty_like_default = None
    div__scalar = torch.ops.aten.div_.Scalar(bernoulli__float, 0.5);  bernoulli__float = None
    mul_tensor = torch.ops.aten.mul.Tensor(arg0, div__scalar);  arg0 = div__scalar = None
    return pytree.tree_unflatten([mul_tensor], self._out_spec)
```

Returned nodes map looks like this (doesn't look the same as the match pattern above):

```
NODES MAP: 
   mul_tensor :  mul_tensor
   arg0 :  arg0
```

For more detail, see P784006912.

cc @ezyang @SherlockNoMad @EikanWang @jgong5 @wenzhe-nrv @msaroufim @wconstab @bdhirsh @anijain2305
Support for `eval` in functional_call
Labels: module: nn, triaged, module: functional UX

### 🚀 The feature, motivation and pitch

`functional_call` on a model currently runs in `.train()` mode by default, which will fail on models containing for eg `BatchNorm2d` layers which explicitly need the `training` variable to be `False` at inference.
`functional_call` currently has no way of calling layers as such, hence fails on BatchNorm2d layers.

### Alternatives

_No response_

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @zou3519
fsdp load model causing insufficient CPU memory
Labels: triaged, module: fsdp

### 🚀 The feature, motivation and pitch

when use fsdp, it need load model on cpu, but every process load which means it need 8 times cpu memory on a 8 GPU machine, causing insufficient CPU memory, is there any solution to this now? If not, please optimize as soon as possible. Thank you  

### Alternatives

_No response_

### Additional context

_No response_

cc @zhaojuanmao @mrshenli @rohan-varma @awgu
Error reporting uses formal parameter names of downstream C++ function
Labels: triaged, module: multi-headed-attention

### 🐛 Describe the bug

When calling sdpa, I get the error message `RuntimeError: The size of tensor a (256) must match the size of tensor b (2) at non-singleton dimension 2` -- there are no parameters a and b in sdpa.  Likely it's the formal parameter names of a function called from inside sdpa, which does not relate to the visible Python call stack.  (Partial stack backtrace below.) 

Can we / Should we check parameters inside sdpa and use proper sdpa formal parameter names?  I think this behavior has shown for other functions, although getting the order of tensor dimensions right when updating arbitrary transformer code feels an order of magnitude harder.  No concern about having to do that per se, but it would be nice to have errors expressed in terms of sdpa to help debug.

```
  File "/data/users/mikekg/fbsource/buck-out/v2/gen/fbcode/46d412658cf1d160/careml/nlp/toolkit/__tests__/tests#link-tree/pytorch/text/fb/nn/modules/transformer.py", line 69, in forward
    attention = self.attention(input, key_padding_mask)
  File "/data/users/mikekg/fbsource/buck-out/v2/gen/fbcode/46d412658cf1d160/careml/nlp/toolkit/__tests__/tests#link-tree/torch/nn/modules/module.py", line 1505, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/users/mikekg/fbsource/buck-out/v2/gen/fbcode/46d412658cf1d160/careml/nlp/toolkit/__tests__/tests#link-tree/torch/nn/modules/module.py", line 1514, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/users/mikekg/fbsource/buck-out/v2/gen/fbcode/46d412658cf1d160/careml/nlp/toolkit/__tests__/tests#link-tree/pytorch/text/fb/nn/modules/multihead_attention.py", line 320, in forward
    attn = F.scaled_dot_product_attention(q, k, v, key_padding_mask, self.dropout_p, False);
RuntimeError: The size of tensor a (256) must match the size of tensor b (2) at non-singleton dimension 2
```

### Versions

fbcode trunk
Add support for NEON ISA in the Inductor C++ backend
Labels: triaged, module: inductor

### 🚀 The feature, motivation and pitch

Context: The TorchInductor C++ backend currently supports vectorization in C++ Codegen through two Intel ISAs: AVX2 and AVX512, as mentioned in the [Update 5 Blog](https://dev-discuss.pytorch.org/t/torchinductor-update-5-cpu-backend-backend-performance-update-and-deep-dive-on-key-optimizations/1117#vectorization-in-c-codegen-4). While the Aten library does support Arm as well, we are yet to leverage its NEON/SVE ISAs to generate optimized kernels. The blog also [mentions](https://dev-discuss.pytorch.org/t/torchinductor-update-5-cpu-backend-backend-performance-update-and-deep-dive-on-key-optimizations/1117#vectorization-in-c-codegen-4:~:text=It%20can%20be,sub%2Dclasses.) that the VecISA class can be subclassed in order to support other ISAs.

Proposal: I am working on providing NEON ISA support for the TorchInductor's C++ backend. Particularly, I intend to provide a NEON implementation of the `vec_reduce_all()` function, which currently has optimized [AVX2 and AVX512 intrinsics implementations](https://github.com/pytorch/pytorch/blob/ced5c89b6fbe827a538b7ada96b2f9a5989871c7/aten/src/ATen/cpu/vec/functional_base.h#L37-L79) for x86 processors introduced by @mingfeima in #73953, as well as a [slow path](https://github.com/pytorch/pytorch/blob/ced5c89b6fbe827a538b7ada96b2f9a5989871c7/aten/src/ATen/cpu/vec/functional_base.h#L12-L28) implementation for other processors including Arm. I have implemented a NEON version for the function, wired up the Inductor's generated C++ to invoke this NEON path on Arm CPUs & I've seen performance improvements, particularly in the Softmax operation.

Posting this here for any discussion before raising a PR.

### Alternatives

_No response_

### Additional context

_No response_

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng
Nondeterministic segfault in test_content_store.py under Dynamo config
Labels: module: tests, triaged, module: dynamo

### 🐛 Describe the bug

Running `PYTORCH_TEST_WITH_DYNAMO=1 python test/test_content_store.py -v` enough (~5 times) leads to a segfault. Weirdly, some unrelated PRs (e.g. https://github.com/pytorch/pytorch/pull/104481) seem to cause the segfault to be triggered more consistently. 

Stack trace looks like the following and involves Python GC and weakrefs.: https://gist.github.com/zou3519/799c8c069d220beca9fa56a908044212 

### Versions

main; Python 3.10.4

cc @mruberry @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @chenyang78
New Loss Function Add In Pytorch
Labels: feature, module: loss, triaged

### 🚀 The feature, motivation and pitch

I am trying to add a new loss function for using object detection time. 
One Short Story: Suppose you create one project like face detection. And that time find a loss function to train your model. At that point your hand two ways have one is to call Pytorch inbuild loss function for using this type of task. And another way to create a custom loss function similar to the PyTorch loss function. My question is which way do you choose? 
I know most people answer the first way because it's a fast way.
![Screenshot from 2023-07-10 19-49-09](https://github.com/pytorch/pytorch/assets/93595990/78490e48-fe22-4c02-84c3-41a36bf97753)


And a similar type of loss function for used the YoloV8 model, see this link: https://arxiv.org/pdf/2305.09972.pdf 

### Alternatives

_No response_

### Additional context

_No response_
generate_vmap_rule=True sometimes gives batched grad_output
Labels: triaged, module: functorch

### 🐛 Describe the bug

```python
import torch
from torch.func import jacrev, vmap
class Foo(torch.autograd.Function):
    generate_vmap_rule = True
    @staticmethod
    def forward(x):
        a = 0.5 * x**2
        b = 0.5 * x**2
        return a, b
    @staticmethod
    def setup_context(ctx, inputs, output):
        ctx.save_for_backward(inputs[0])
    @staticmethod
    def backward(ctx, *grad_outputs):
        x = ctx.saved_tensors[0]
        # not vmapped when grad_outputs[-1] = 0
        non_vmapped = grad_outputs[-1].new_zeros(x.shape)
        non_vmapped[:] = x
        return x
foo = Foo.apply
def bar(a, b):
    return a * 10
def f(z):
    return bar(*foo(z))
ttt = torch.randn(10, 2)
grad_vmap = vmap(jacrev(f))(ttt)
```
raises a "vmap incompatible in-place" error. Changing `bar` to `a * 10 + b` fixes the problem. It's unclear to me if this is a bug or not.

### Versions

main

cc @Chillee @samdow @kshitij12345 @janeyx99