gopikrsmscs/torch-issues huggingface dataset:
AOT autograd: Avoid dependency on strides for manual regeneration of outputs that are aliased to inputs
Labels: triaged, oncall: pt2, module: aotdispatch

### 🐛 Describe the bug

AOT autograd's manual regeneration of outputs assumes strided output tensor  support. But backends like Habana (https://docs.habana.ai/en/latest/index.html) produces contiguous tensor outputs. As a result, examples like the below one results in accuracy issues

 ```
def fn(a):
  b = a.t()
  b.mul_(1.0)
  return b   

x = torch.arange(6).reshape([2, 3]).to('hpu')

print("x ", x.cpu())

compiled_fn = torch.compile(fn, backend="aot_hpu_training_backend")        
y = compiled_fn(x)

print("y ", y.cpu())

```

**Console prints:**

x  tensor([[0, 1, 2],
        [3, 4, 5]])
		
y  tensor([[0, 1],
        [2, 3],
        [4, 5]])
		
Tensor y has incorrect values. Expected output is 
y  tensor([[0, 3],
        [1, 4],
        [2, 5]])
		
Our analysis suggests that the issue occurs in step 4 of runtime_wrapper(), inside gen_alias_from_base(). This function assumes strided tensor for mul's output. Related bug for TPU backend: https://github.com/pytorch/xla/issues/5179

**Request:** 
Please avoid dependency on the tensor strides for output regeneration. Instead the information regarding the original alias operation (transpose in the above example) should be used. 



### Versions

Collecting environment information...
PyTorch version: 2.0.1a0+git7f346ef
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 14.0.5 (ssh://gerrit:29418/tpc_llvm10 38f3bfdbf5b8581125aa7ac5a2605c9674624566)
CMake version: version 3.26.3
Libc version: glibc-2.31

Python version: 3.8.11 (default, Aug  3 2021, 15:09:35)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-137-generic-x86_64-with-glibc2.17
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
Insert nvtx markers into generated triton kernels
Labels: triaged, open source, module: inductor, ciflow/inductor, topic: devs

Fixes #103552 Adds nvtx markers around the generated triton kernel call functions.  Provides descriptive information about the original operators that were fused by the kernel.  Useful for performance analysis and general understanding of how the fusion algorithm works.

Information about the originating operators is appended to the RecordFunctionFast profiling message in the CachingAutotuner.  The OriginOp message is a list consisting of op_type, module_name, torch_op, seq_nr.  Each operator in the list is one of the ops that was included in the fused triton kernel.  The OriginOp message is enabled by default in the same nvtx marker as the kernel_name string.  There is an additional logic path for the case when the kernel is called from the extern_kernel() function.  In this case, an nvtx_range push/pop pair is inserted around the kernel launch in the generated inductor code.  The nvtx ranges are only inserted when **torch._inductor.config.profiler_mark_wrapper_call** is enabled.


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
"addmm_out_sparse_csr_impl_mkl" not implemented for 'Byte'
Labels: module: sparse, triaged

### 🐛 Describe the bug

```
import torch
a =torch.zeros(128, 12288*4, dtype=torch.uint8)                                                                                                                                                                                                                                                                                                                              
                                                                                                                                                                                                           
a = a.to_sparse_csr()                                                                                                                                                                                      
# b = torch.rand(12288*4, 4096, dtype=torch.int8)                                                                                                                                                          
b = torch.randint(high=255, size=(12288*4,12288), dtype=torch.uint8)                                                                                                                                       
torch.sparse.mm(a,b)
```

### Versions

PyTorch version: 2.0.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 9.3.0
Clang version: Could not collect
CMake version: version 3.26.3
Libc version: glibc-2.17

Versions of relevant libraries:
[pip3] intel-extension-for-pytorch==2.0.100
[pip3] numpy==1.24.1
[pip3] torch==2.0.1
[pip3] torch-scatter==2.1.1
[pip3] torch-sparse==0.6.17
[pip3] torchaudio==2.0.2
[pip3] torchvision==0.15.2
[pip3] triton==2.0.0
[conda] intel-extension-for-pytorch 2.0.100                  pypi_0    pypi
[conda] numpy                     1.24.1                   pypi_0    pypi
[conda] torch                     2.0.1                    pypi_0    pypi
[conda] torch-scatter             2.1.1                    pypi_0    pypi
[conda] torch-sparse              0.6.17                   pypi_0    pypi
[conda] torchaudio                2.0.2                    pypi_0    pypi
[conda] torchvision               0.15.2                   pypi_0    pypi
[conda] triton                    2.0.0                    pypi_0    pypi

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer
Disclose C++ ATen ops type promotion rules under OpOverload in Python
Labels: module: docs, triaged

### 🚀 The feature, motivation and pitch

As title, is it possible or good idea?

Motivation is to enable doing fx aten -> fx aten transformation with casts explicitly inserted. 

### Alternatives

_No response_

### Additional context

_No response_

cc @svekars @carljparker
DISABLED test_backward_ddp_outside (__main__.TensorPipeDdpUnderDistAutogradTest)
Labels: oncall: distributed, module: flaky-tests, skipped

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_backward_ddp_outside&suite=TensorPipeDdpUnderDistAutogradTest) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/14264987221).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 3 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_backward_ddp_outside`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `distributed/rpc/test_tensorpipe_agent.py`

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
ARM based GPU support for Distributed Data Parallelism Module
Labels: oncall: distributed

### 🚀 The feature, motivation and pitch

I'm working on building a self driving model for electric race cars and would love to be able to use the full power of my ARM based GPUs for training. I know that this is being worked on but I can't find anything related to allowing Distributed training using multiple ARM based GPUs.  I see this being huge for a lot of developers as ARM based chips are only becoming more and more popular as they are cost effective and efficient. I also recognize that there is already support for ARM based GPUs but not in a distributed fashion. If I am incorrect please let me know and provide me with an example if you can. 

### Alternatives

An alternative would be to provide more documentation on the PyTorch DistributedDataParallel module regarding ARM based chips. Making it abundantly clear that this module is developed solely for use with NVIDIA GPUs. This would prevent futile efforts to making it work. 

### Additional context

_No response_
```[tasklist]
### Tasks
```


cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Can't call allow_in_graph inside of a function being torch.compile'd 
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

We are unable to call allow_in_graph inside of a function being torch.compile'd with fullgraph=True and the error message is confusing. We should either allow this or improve the error message.
```py
import torch
import torch._dynamo

@torch.compile(backend='aot_eager', fullgraph=True)
def f(x):
    @torch._dynamo.allow_in_graph
    def apply_func(x):
        return x.clone()
    
    return apply_func(x)

x = torch.randn([], requires_grad=True)

f(x)
```
produces: 
> Unsupported: call_function allow_in_graph in skip_files /raid/rzou/pt/debug-cpu4/torch/_dynamo/__init__.py

Discovered with @drisspg 

### Error logs

_No response_

### Minified repro

_No response_

### Versions

main

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
Installing Torch on AMD Platform Leads to Huge Docker Image
Labels: module: rocm, triaged

### 🐛 Describe the bug

I am not sure this is the best place to raise this issue and I am not sure "bug" is the best characterization, but I found some unexpected behavior when installing pytorch in docker images built for different platforms. In particular, when I install torch in an image built for the `linux/arm64` platform I get an image that is 628MB, but when I install torch in an image built for the `linux/amd64` platform I get an image that is 6.87GB. Ideally the image would be a similar size for both platforms because the large images are slow to build/push/pull.

Here is my `Dockerfile`:
```
FROM python:3.11.2-slim-bullseye
ENV DEBIAN_FRONTEND=noninteractive
RUN pip3 install torch==2.0.0
```

Here are the docker build commands I ran:
```
docker build --platform linux/amd64 -t amd_build -f Dockerfile .
docker build --platform linux/arm64 -t arm_build -f Dockerfile .
```

And here you can see the resulting image sizes:
```
REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
amd_build    latest    6a5b409f32bf   27 minutes ago   6.87GB
arm_build    latest    1fc9604a61f3   35 minutes ago   628MB
```

### Versions

Note: Some of this information may not be relevant because the issue I am raising happens in docker.

Collecting environment information...
PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.4 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: Could not collect
Libc version: N/A

Python version: 3.11.3 (main, Apr  7 2023, 20:13:31) [Clang 14.0.0 (clang-1400.0.29.202)] (64-bit runtime)
Python platform: macOS-13.4-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==1.24.2
[pip3] torch==2.0.0
[conda] Could not collect

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo
test_fstrings2 fails with dynamic
Labels: good first issue, triaged, module: dynamic shapes

### 🐛 Describe the bug

```
    @make_test
    def test_fstrings2(x):
        tmp = f"{x.shape[0]} bar"
        if tmp.startswith("10"):
            return x + 1
```

fails with

```
stats [('calls_captured', 1), ('unique_graphs', 1)]
.[2023-06-14 10:10:13,223] torch._dynamo.utils: [ERROR] Accuracy failed (<class 'NoneType'>): None != tensor([[ 0.8883,  0.5034,  1.1631,  0.1183,  1.0539,  1.6684,  0.9403,  0.5325,
          0.7847,  1.8840],
        [ 0.2416,  0.6311,  0.6576, -0.4020,  1.3206, -0.0219,  1.7988,  0.9077,
          0.2951, -0.6024],
        [ 1.2891,  1.4899,  0.6147,  0.2880,  0.8294, -0.4594,  1.2207,  1.2463,
         -0.3248,  1.6970],
        [ 0.3369,  2.2158, -0.4949,  1.8810, -0.1786,  0.0660,  0.4325,  0.7228,
         -1.1834,  1.3668],
        [ 1.9380,  1.0078,  0.6861, -0.1567,  2.8409, -0.0174,  2.2192,  1.1601,
          2.5985,  0.9531],
        [-0.5270, -1.0143, -0.5173,  1.3877, -0.1849,  1.6897,  2.3232,  2.8169,
          1.6808,  1.7244],
        [ 1.0323, -0.6593, -0.8773,  1.7372,  1.9257,  1.9247,  1.1825,  0.9263,
          1.3147, -0.0369],
        [ 1.2100,  1.6144,  1.0628,  0.6703, -0.7970,  1.8728,  1.7670,  0.8862,
          0.0572,  1.7540],
        [ 1.1407,  0.3063,  0.3841,  0.2705,  2.3204,  2.5997, -0.0792,  0.6604,
         -0.4538, -1.6740],
        [ 2.5984,  1.8021,  1.5722,  1.0653,  0.9765,  1.8876,  2.4689,  2.2647,
          0.7247,  0.8675]])
stats [('calls_captured', 1), ('unique_graphs', 1)]
F
======================================================================
FAIL: test_fstrings2_dynamic_shapes (torch._dynamo.testing.DynamicShapesFunctionTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/testing.py", line 305, in _fn
    return fn(*args, **kwargs)
  File "/data/users/ezyang/b/pytorch/test/dynamo/test_functions.py", line 42, in test_fn
    return torch._dynamo.testing.standard_test(self, fn=fn, nargs=nargs)
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/testing.py", line 225, in standard_test
    self.assertTrue(same(val1a, correct1))
AssertionError: False is not true

----------------------------------------------------------------------
Ran 2 tests in 2.427s

```

My guess is that SymNodeVariable isn't implementing printing correctly.

The obvious thing to do is specialize and guard in this situation. However, it's a little subtle: if someone creates a string and then ends up not using it, we would prefer to NOT have specialized. This can matter for poorly written error checking code.

### Versions

main
torch.cuda.mem_get_info to return 0 if CUDA context isn't initialized
Labels: module: cuda, triaged, actionable

### 🐛 Describe the bug

It appears that mem_get_info itself triggers CUDA context initialization which can be a bit surprising.

But overall this issue is minor, so maybe just being explicit about this in the docs would solve the surprise

### Versions

N/A

cc @ptrblck
Support ByteTensor and ShortTensor for nn.Embedding and nn.EmbeddingBag
Labels: module: nn, triaged, enhancement, actionable, topic: improvements

### 🚀 The feature, motivation and pitch

Torch's embedding layers only accept int32 and int64 as input. However, for sequences with a small number of distinct possible tokens (e.g., ASCII character embeddings or DNA sequences) int8 or int16 are sufficient to index all of the tokens. Currently, modeling long sequences that consist of only a few possible tokens means wasting a lot of GPU memory and being forced to use smaller batch sizes than might be desirable.

### Alternatives

_No response_

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
add default argument device type api
Labels: triaged, open source, Stale, topic: not user facing

Fixes #103828
1、For many operators(such as pin_memory), the device argument is cuda if not given; but for other device, we must have to give extra argument device_type comparing to cuda, so we add an API to set the default argument device just once at the begining to keep usage consistent with cuda.
2、And there are some API defined in Python, we add a argument named device_type and the default value is cuda, so that we could support more device (privateuse1 device). So we use the this api to get the default device to keep usage consistent with cuda if not gived device_type.
File Missing When i build with C++
Labels: module: cpp, triaged

I installed pytorch For C++distribution and added in CMakeList.txt.

1) Initially it given torch/torch.h No Such file or directory and I used CMAKE_PREFIX_PATH to libtorch.
2) I try to build it will showing ATen/Tensor.h No such file or directory. I checked for .h file(No file neme Tensor.h).

3) I tried different versions of NVIDIA toolkit CUDA versions for Build the Cmake.

4) Now it showing ATen/Tensor.h No such file or directory. I checked for .h file(No file Name Tensor.h).

which Version best suitable for Running the smallest Applications.
#include <iostream>
#include <torch/torch.h>

int main() {
    torch::Tensor tensor = torch::rand({ 2, 3 },torch::kCUDA);
    std::cout << tensor << std::endl;
}

cc @jbschlosser
Request: flag to know model is compiled after torch.compile()
Labels: triaged, enhancement, oncall: pt2

### 🚀 The feature, motivation and pitch

As a user, it will be great to have a flag that reports that the model has been compiled successfully. Like:

model = torch.compile(model)

print(model.is_compile)
True

Even if the flag does not exist before compilation and it is created later, that would be great.

Thanks!

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
Inject detailed NVTX markers into the Inductor Triton generated kernels
Labels: triaged, oncall: pt2, module: inductor

### 🚀 The feature, motivation and pitch

The inductor/triton generated kernels have limited NVTX marker support and the kernel names don't provide much information related to which operators they implement.  Since many of the generated kernels represent fused ops, it would be useful to be able to see from the profiler which high level ops contributed to the fused kernels.  There is an existing attribute in the inductor graph ir called called origin_node.  Origin node indicates which high level op the node in the IR is associated with.  For fused kernels there is a list of origin_nodes which describe which high level ops contributed to the fused kernel.  This is really useful information for understanding how the fusion algorithm works.  I am proposing to capture this information for each triton kernel and inject a marker into the generated code so it appears in the profiler at runtime.  The markers will use the record_function python api.

The screen shot below shows an example of the markers I implemented.  They are prefixed with **triton_info:** and show details about each of the origin ops, including the module name, op type and sequence id.

![inductor_fwd_pass](https://github.com/pytorch/pytorch/assets/37422826/a11e6f7f-a524-416e-96c6-9ffa9dc95524)


### Alternatives

Not really, some of this information is available in the debug logs but it is difficult to map the kernel instance directly to the ops in the original fx graphs.

### Additional context

This is related to #102375 which adds fwd and bwd sequence id tracking to aot autograd.  The sequence id of each op is included in the triton_info nvtx markers.

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @davidberard98
Deduplicate the operands passed into torch.cond after dynamo tracing.
Labels: triaged

### 🚀 The feature, motivation and pitch

Currently, we lift the free variables inside torch.cond branches as extra inputs to the branch graph. As a result, for simplicitly, we naively extend the torch.cond operands list with free lifted variables from each branch. For example, let's consider `cond(pred, true_fn, false_fn, [x])` where `true_fn` has `a, b, c` as free variables and `false_fn` has `a, b, d` as free variables. Then, dynamo will rewrite it as `cond(pred, true_fn, false_fn, [x, a, b, c, a, b, d])`. Ideally, we should detect this and deduplicate the operands list. 


### Alternatives

_No response_

### Additional context

_No response_
`gradcheck` produces false positives with sparse inputs when `masked=False`.
Labels: module: sparse, module: autograd, triaged

### 🐛 Describe the bug

As per title. As an example, let's consider the `sampled_addmm` method which is semantically equivalent to
`sampled_addmm(s, m1, m2, alpha, beta) := alpha * (m1 @ m2).sparse_mask(s) + beta * s`.

If we inspect the subgradient of `sampled_addmm` wrt `s` in `derivatives.yaml`, we find the following:
```
- name: sparse_sampled_addmm(Tensor self, Tensor mat1, Tensor mat2, *, Scalar beta=1, Scalar alpha=1) -> Tensor
  self: maybe_multiply(grad, beta.conj())
```
Note, that under the assumption of masked semantics this formula is correct, even though it does not account for the `(mat1 @ mat2).sparse_mask(self)` part. This follows from the sparse semantics that implies `self.indices == (self + perturbation_of_self).indices`. Hence we can expect `gradcheck` to work with `masked=True`:

```python
In [1]: import torch

In [2]: x = torch.eye(3, dtype=torch.double).to_sparse_csr().requires_grad_(True)

In [3]: y = torch.rand(3, 3, dtype=torch.double)

In [4]: z = torch.rand(3, 3, dtype=torch.double)

In [5]: torch.autograd.gradcheck(lambda x: torch.sparse.sampled_addmm(x, y, z).to_dense(masked_grad=True), (x,), masked=True)
Out[5]: True

```

However, the situation is reversed for `masked=False`. In this case the backward formula for `self` should take `alpha * (m1 @ m2).sparse_mask(self)` into consideration, so it is expected for `gradcheck` with `masked=False` to fail.
This, however, does not happen:
```python
In [6]: torch.autograd.gradcheck(lambda x: torch.sparse.sampled_addmm(x, y, z).to_dense(masked_grad=False), (x,), masked=False)
Out[6]: True
```

As per @pearu's insight, this happens during the densification process in gradcheck. Namely, it sometimes expands `self.indices` to full dimensions while producing a new sparse input `self_densified`. Unfortunately, `sampled_addmm(self)` and `sampled_addmm(self_densified)` are not equivalent in backward, because `sampled_addmm(self_densified)` should pass gradcheck with either `masked=True` or `masked=False` since it's mask is the whole space.

### Versions

Current master.

cc @alexsamardzic @pearu @cpuhrsch @amjames @bhosmer @ezyang @albanD @zou3519 @gqchen @soulitzer @Lezcano @Varal7
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 6047) of binary: /home/win10-ubuntu/anaconda3/envs/vicuna-7b/bin/python
Labels: oncall: distributed, triaged

### 🐛 Describe the bug

![image](https://github.com/pytorch/pytorch/assets/39661319/0b4f213d-6022-4873-a40a-b5cf4825ed18)


### Versions

Fine-tune vicuna-7b error
Fine-tuning commands：
![image](https://github.com/pytorch/pytorch/assets/39661319/8151b496-0bc6-4995-b39f-3c32857cce6c)
But I got an error：
![image](https://github.com/pytorch/pytorch/assets/39661319/4b0ec6e9-06a2-42cf-b0b3-c49770d2dbab)


cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
DISABLED test_mem_get_info (__main__.TestCuda)
Labels: module: cuda, triaged, module: flaky-tests, skipped

Platforms: rocm

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_mem_get_info&suite=TestCuda) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/14208052987).

Over the past 3 hours, it has been determined flaky in 4 workflow(s) with 4 failures and 4 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_mem_get_info`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_cuda.py`

ConnectionTimeoutError: Connect timeout for 5000ms, GET https://raw.githubusercontent.com/pytorch/pytorch/main/test/test_cuda.py -2 (connected: false, keepalive socket: false, socketHandledRequests: 1, socketHandledResponses: 0)
headers: {}

cc @ptrblck
No backward implementation for `torch._native_multi_head_attention`
Labels: triaged, module: multi-headed-attention

### 🚀 The feature, motivation and pitch

There is a forward implementation for `torch._native_multi_head_attention` but no coressponded backward implementation. So use torch to realize training MHA, we need to use small ops to compose it or use `torch.nn.MultiHeadAttention`. 

### Alternatives

* Small ops like linear/bmm to compose MHA.
* `torch.nn.MultiHeadAttention`

### Additional context

_No response_
Document CI retry rules
Labels: triaged, module: devx

With all the recent changes w.r.t retrying to harden PyTorch CI, we need to create a wiki page to document all these mechanisms.  The tentative list includes:

* Individual test case retry (flaky bot)
* Retry test file
* Retry on workflow steps (using GHA)
* Retry the job itself (retry bot)

In addition, we also want to gather data points to answer the following questions

* How much resource do we spend on retrying these cases?
* And a rough estimation on how frequently people manually retry stuffs on their PR to get green signals or to debug flaky issue


cc @ZainRizvi @kit1980 @clee2000
[Inductor] Optimize More Cases of Int32 -> Int64 
Labels: triaged, enhancement, module: inductor

### 🚀 The feature, motivation and pitch

Inductor has [an existing](https://github.com/pytorch/pytorch/blob/main/torch/_inductor/optimize_indexing.py#L313-L317) optimization which will convert indirect indexing that is done in int64 to int32 for index expressions we can prove are expressible in int32. However this optimization is incomplete.

1. We do not propagate the bounds of tensors from one kernel to the other. 
2. We do not change the dtype of temporary tensors which could be converted to int32. 
3. We use ValueRange Analysis to strength reduce on indices whose bounds we can prove. However, in some cases, as in the kernel below, we are indexing from Tensor values whose bounds we don't know.

For the following repro:

```
import torch
import triton

inp = torch.rand([6, 3, 352, 352], device="cuda", requires_grad=False)
inp2 = torch.rand([6, 352, 352, 2], device="cuda", requires_grad=False)

def grid(inp, inp2):
    return torch.ops.aten.grid_sampler_2d.default(inp, inp2, 0, 0, False)

def invoke_grid():
    return grid(inp, inp2)

median_ms = triton.testing.do_bench(
            lambda: invoke_grid()
)

grid_opt = torch.compile()(invoke_grid)

median_ms2 = triton.testing.do_bench(
            lambda: grid_opt()
)

print(f"Eager Execution time: {median_ms} secs")
print(f"Compiled Execution time: {median_ms2} secs")
```

Inductor is significantly slower than eager (25%) which can be reduced to 10% with int64->int32 conversions. 
 
[Original code](https://gist.github.com/eellison/cb422e219f8be58a3d8787a0dea75401), and then [optimize code](https://gist.github.com/eellison/91b80dac72c088344a92da17411161b9), where the int64 has been replaced with int32.

This could be optimized, but it's a bit tricky. If we guard on the tensor used in the second kernel having numel < 2^32, we can set the bounds of this final expression to also be in the range[0, 2^32). 

```
tmp61 = tl.load(in_ptr6 + (x0 + (123904*x2)), None, eviction_policy='evict_last')
tmp64 = tl.load(in_ptr7 + (x0 + (123904*x2)), None, eviction_policy='evict_last')
```

From there, we would need to propagate the bounds backwards. There is initial work to do that by @ysiraichi in https://github.com/pytorch/pytorch/pull/97963, and we could extend it for the set of ops that appears in this kernel to start. Once the bounds are propagated we could reduce the dtype of the int64 to int32, as well as the ops that construct those tensors.

I originally wrote this up as starter task but it might be a bit complicated for that lol.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @aakhundov @chenyang78 @ezyang, @lezcano for other thoughts here



### Alternatives

_No response_

### Additional context

_No response_
Error encountered when tracing model with Dynamo/Functorch for export with trilinear interpolation
Labels: triaged, oncall: pt2, module: dynamic shapes, module: export

### 🐛 Describe the bug

When exporting a small network via any of the following functions, one of two errors is encountered. The model runs successfully as-is (via `model(sample_inputs)`) and also works with `torch.compile`, but fails during export. Additionally, the model has no explicit control flow and `torch._dynamo.explain` shows no graph breaks.

**Functions With Errors:**
- `torch.fx.experimental.proxy_tensor.make_fx`
- `torch._dynamo.export`
- `torch._export.export`
- `torch._functorch.aot_autograd`

**Sample Script with Network**
```python
import torch

class MyModule(torch.nn.Module):
    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)

    def forward(self, x):
        out = torch.nn.functional.interpolate(x, size=(10, 20, 30), mode="trilinear", align_corners=True)
        return out + 1

# Build model, sample inputs, and validate model succeeds on sample inputs
model = MyModule().eval().cuda()
sample_input = torch.rand((1, 2, 3, 4, 5)).cuda()

model(sample_input)

# Try various export/tracing methods
try:
    torch._dynamo.export(model, sample_input, aten_graph=True, tracing_mode="symbolic")
except Exception as e:
    print("Dynamo export:\n", e)

try:
    torch._export.export(model, sample_input)
except Exception as e:
    print("Torch export:\n", e)

try:
    torch._functorch.aot_autograd.aot_export_module(model, sample_input, trace_joint=False)
except Exception as e:
    print("AOT export:\n", e)

try:
    torch.fx.experimental.proxy_tensor.make_fx(model, tracing_mode="symbolic", _allow_non_fake_inputs=True, pre_autograd=True)(sample_input)
except Exception as e:
    print("Make FX:\n", e)

print(torch._dynamo.explain(model, sample_input))
```

**Errors**

**Error 1 [`torch._export.export` + `aot_export_module`]:**
```python
Failed running call_function <function interpolate at 0x7f1b7d1769d0>(*(FakeTensor(..., device='cuda:0', size=(2, 3, 4, 5)),), **{'size': (10, 20, 30), 'mode': 'trilinear', 'align_corners': True}):
Input and output must have the same number of spatial dimensions, but got input with spatial dimensions of [4, 5] and output size of (10, 20, 30). Please provide input tensor in (N, C, d1, d2, ...,dK) format and output size in (o1, o2, ...,oK) format.
```

**Error 2 [`torch._dynamo.export` + `make_fx`]:**
```python
Failed running call_function <function interpolate at 0x7f1b7d1769d0>(*(FakeTensor(..., device='cuda:0', size=(1, s0, s1, s2, s3)),), **{'size': (10, 20, 30), 'mode': 'trilinear', 'align_corners': True}):
Cannot call sizes() on tensor with symbolic sizes/strides
```

### Versions

```python
Versions of relevant libraries:
[pip3] torch==2.1.0.dev20230608+cu118
```

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
[inductor] multi-kernel support
Labels: module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #106012
* __->__ #103469

For a persistent reduction, we generate 2 flavor of 'equivalant' kernels at the same time
- persistent reduction
- regular reduction

A MultiKernel wraps these 2 kernels and pick the one with better performance at runtime.

Here I talk more about implementation details:
- Inductor maintains states for generating kernels. E.g. the wrapper code.  After we generate code for one kernel, we need restore the inductor state before we can generate the counterpart.

***There is one thing I need some comments from others***:
There is one tricky thing about kernel arguments. In general, inductor removes a buffer from the argument list if it's only used inside the kernel.  But somehow a buffer removed by persistent reduction kernel may still be kept by the regular (non-persistent) reduction kernel because of some CSE invalidation rule. My current implementation avoid removing buffers if multi_kernel is enabled. This makes sure both flavors of reduction has consistent argument list.  Another idea I have is, we generate the multi-kernel definition with the union of arguments from both sub-kernels. Let each sub-kernel pick the subset of arguments it wants. But this will make the code-gen or multi-kernel much complex.

I'm not sure if there is some easy and clean way to resolve this.


Testing command:
```

TORCHINDUCTOR_MULTI_KERNEL=1 TORCH_LOGS=+torch._inductor.graph TORCHINDUCTOR_UNIQUE_KERNEL_NAMES=1 python benchmarks/dynamo/huggingface.py --backend inductor --amp --performance --only BertForMaskedLM --training

```


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @aakhundov
[ao] making hist_obs handle torch.inf and closeby values
Labels: module: cpu, Stale, with-ssh, release notes: quantization, topic: bug fixes

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #103467
* #107623

Summary: This PR does 2 things:

1) Previously this would simply error, now it will ignore any
torch.inf values that it recieves. note: The code checks for torch.inf after
aminmax that way if there are no torch.inf values found, the perf is a
relatively unchanged

2) as mentioned in https://github.com/pytorch/pytorch/issues/100051,
values close to (but not quite at) the maximum/minimum float value could
overflow to infinity in the course of _adjust_min_max() (when this large
value would be multiplied by something in the middle of a calculation
that would otherwise result in a non inf value). This was fixed by
rearranging the order of operations for the lines in question without
altering the actual equations. Specifically, where operations in lines
1095, 1098 and 1100 have multiplication and division of large values,
its better to divide the two large values before multiplying, rather
than multiplying the two large values together (creating overflow) before dividing like it had been.

Test Plan: python test/test_quantization.py
TestObserver.test_histogram_observer_ignore_infinity

python test/test_quantization.py TestObserver.test_histogram_observer_handle_close_to_infinity
Reviewers:

Subscribers:

Tasks:

Tags:

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
Asynchronous CUDA AveragedModel
Labels: module: optimizer, triaged, needs research

### 🚀 The feature, motivation and pitch

This is a proposal to improve the efficiency of CUDA `AveragedModel` used in EMA and SWA. (Follow-up of https://github.com/pytorch/pytorch/pull/94820). I would provide the implementation but would like feedback/approval before opening a PR.

Currently the EMA/SWA weight update is done on the default stream, same as all the other GPU work.

Because EMA/SWA weights are typically updated at the end of a training iteration (after an optimizer step), and are not needed until the end of the next iteration, we can actually run the EMA/SWA update in parallel of the forward/backward/optimizer step, virtually eliminating the overhead of EMA/SWA in many cases.

This can be done by using a separate dedicated CUDA stream to perform the weight update.

This is how it is done in the NeMo framework:
- Stream creation: https://github.com/NVIDIA/NeMo/blob/a87702a522387da0aac62dc1f90a88a8e0bfc7cc/nemo/collections/common/callbacks/ema.py#L234
- Synchronization between the dedicated stream and the main stream: https://github.com/NVIDIA/NeMo/blob/a87702a522387da0aac62dc1f90a88a8e0bfc7cc/nemo/collections/common/callbacks/ema.py#L259
- Weight update in the dedicated stream: https://github.com/NVIDIA/NeMo/blob/a87702a522387da0aac62dc1f90a88a8e0bfc7cc/nemo/collections/common/callbacks/ema.py#L261
- API to manually synchronize the dedicated stream: https://github.com/NVIDIA/NeMo/blob/a87702a522387da0aac62dc1f90a88a8e0bfc7cc/nemo/collections/common/callbacks/ema.py#L310

If the team is interested in extending the current `AveragedModel` class with an optional asynchronous feature, let me know and I'll work on a PR.

### Alternatives

_No response_

### Additional context

_No response_

cc @vincentqb @jbschlosser @albanD @janeyx99 @crcrpar
Deprecation warning on lr_scheduler.step(num_steps)
Labels: module: optimizer, triaged, actionable

### 🐛 Describe the bug

`step(num_steps)` produces a deprecation warning currently. However, there is a legitimate use case for this API in learning rate schedulers — if reloading a trained model and continuing to train, it is necessary to advance the number of steps inside the scheduler to match the current model state.

The scheduler does not provide an alternative way to advance the number of steps. And it is not possible to use `state_dict()`  + `load_state_dict()` because they also prevent changing the learning rate and other hyperparameters during transitions, unless the user manually changes the state dict, which is a hacky.

```
UserWarning: The epoch parameter in `scheduler.step()` was not necessary and is being deprecated where possible. Please use `scheduler.step()` to step the scheduler. During the deprecation, if epoch is different from None, the closed form is used instead of the new chainable form, where available. Please open an issue if you are unable to replicate your use case: https://github.com/pytorch/pytorch/issues/new/choose.
```

### Versions

Versions of relevant libraries:
[pip3] torch==2.0.1


cc @vincentqb @jbschlosser @albanD @janeyx99
The document does not emphasize Illegal value in nn.Bilinear
Labels: module: nn, triaged, actionable, module: edge cases

### 🐛 Describe the bug

`Illegal value of in1_feature parameter in nn.Bilinear`

`ZeroDivisionError: float division by zero`
### Code

```py
import torch
from torch import nn


class lenet(nn.Module):
    def __init__(self):
        super(lenet, self).__init__()
        self.conv = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=5, stride=1)
        self.linear = nn.Bilinear(in1_features=0, in2_features=0, out_features=0)

    def forward(self, x):
        # 1st block
        x = self.conv(x)
        x = self.linear(x)

        return x


if __name__ == '__main__':
    net = lenet()
```

### Versions

### Version

```
PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.6 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.102)
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.10 (main, Mar 21 2023, 13:41:05) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-12.6-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==2.0.0
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.15.1
[conda] numpy                     1.23.5          py310hb93e574_0  
[conda] numpy-base                1.23.5          py310haf87e8b_0  
[conda] torch                     2.0.0                    pypi_0    pypi
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.15.1                   pypi_0    pypi
```


cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
The document does not emphasize hidden range in nn.Embedding
Labels: needs reproduction, module: docs, triaged

### 🐛 Describe the bug


`Hidden range of padding parameter in nn.Embedding`

### Code

```py
import torch
import torch.nn as nn


class BiLSTM(nn.Module):
    def __init__(self, batch_size, hidden_dim, vocab_size, sequence_len):
        super().__init__()

        self.batch_size = batch_size
        self.hidden_dim = hidden_dim
        self.input_size = vocab_size
        self.num_classes = vocab_size
        self.sequence_len = sequence_len

        # Dropout
        self.dropout = nn.Dropout(0.25)

        # Embedding layer
        self.embedding_0 = nn.Embedding(num_embeddings=2, embedding_dim=2, padding_idx=3)

    def forward(self, x):

        # Bi-LSTM
        # hs = [batch_size x hidden_size]
        # cs = [batch_size x hidden_size]
        hs_forward = torch.zeros(x.size(0), self.hidden_dim)
        cs_forward = torch.zeros(x.size(0), self.hidden_dim)
        hs_backward = torch.zeros(x.size(0), self.hidden_dim)
        cs_backward = torch.zeros(x.size(0), self.hidden_dim)

        # LSTM
        # hs = [batch_size x (hidden_size * 2)]
        # cs = [batch_size x (hidden_size * 2)]
        hs_lstm = torch.zeros(x.size(0), self.hidden_dim * 2)
        cs_lstm = torch.zeros(x.size(0), self.hidden_dim * 2)

        # Weights initialization
        torch.nn.init.kaiming_normal_(hs_forward)
        torch.nn.init.kaiming_normal_(cs_forward)
        torch.nn.init.kaiming_normal_(hs_backward)
        torch.nn.init.kaiming_normal_(cs_backward)
        torch.nn.init.kaiming_normal_(hs_lstm)
        torch.nn.init.kaiming_normal_(cs_lstm)

        # From idx to embedding
        x = self.embedding_0(x.long())
        return x


if __name__ == '__main__':
    net = BiLSTM(5, 2, 100, 100)
```


### Version

```
PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.6 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.102)
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.10 (main, Mar 21 2023, 13:41:05) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-12.6-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==2.0.0
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.15.1
[conda] numpy                     1.23.5          py310hb93e574_0  
[conda] numpy-base                1.23.5          py310haf87e8b_0  
[conda] torch                     2.0.0                    pypi_0    pypi
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.15.1                   pypi_0    pypi
```


cc @svekars @carljparker
The document does not emphasize hidden range in nn.MaxPool2d
Labels: needs reproduction, module: docs, triaged

### 🐛 Describe the bug


`Hidden range of padding parameter in nn.MaxPool2d`

`pad should be at most half of kernel size, but got pad=2 and kernel_size=2`

### Code

```py
import torch
from torch import nn


class lenet(nn.Module):
    def __init__(self):
        super(lenet, self).__init__()
        self.conv = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=2, padding=2, stride=1)
        self.pool = nn.MaxPool2d(padding=2, kernel_size=2)

    def forward(self, x):
        # 1st block
        x = self.conv(x)
        x = self.pool(x)

        return x


if __name__ == '__main__':
    net = lenet()
```

### Versions

### Version
PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.6 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.102)
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.10 (main, Mar 21 2023, 13:41:05) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-12.6-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==2.0.0
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.15.1
[conda] numpy                     1.23.5          py310hb93e574_0  
[conda] numpy-base                1.23.5          py310haf87e8b_0  
[conda] torch                     2.0.0                    pypi_0    pypi
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.15.1                   pypi_0    pypi



cc @svekars @carljparker
[Distributed] Limit world_size to 8 for FSDP Unit tests
Labels: module: rocm, triaged, open source, ciflow/trunk, topic: not user facing, ciflow/periodic, rocm, rocm priority, merging

There are few unit tests in FSDP that can support upto 8 GPUs.
In this case, for example test_fsdp_uneven has an input size of [8,3]. For each process/rank we pass the data as input[self.rank] as below. So when we use 16 GPUs for our tests, these tests throw an index/key error. So basically to avoid such corner cases, I would like to add this change to use 8GPUs if there are more than 8 GPUs. This is applicable to both ROCm and CUDA builds as well.

https://github.com/pytorch/pytorch/blob/main/test/distributed/fsdp/test_fsdp_uneven.py#L44
https://github.com/pytorch/pytorch/blob/main/test/distributed/fsdp/test_fsdp_uneven.py#L55

cc: @jithunnair-amd 

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
[decomp][bad accuracy] AlbertForQuestionAnswering
Labels: triaged, oncall: pt2, module: inductor, module: pt2 accuracy

### 🐛 Describe the bug

Repro - `python benchmarks/dynamo/huggingface.py  --backend=aot_eager_decomp_partition --amp --training --device cuda  --accuracy --only=AlbertForQuestionAnswering`

Setup - Get my branch - `https://github.com/pytorch/pytorch/tree/tb-pin` and run the above cmd.

Note that the accuracy fails with `aot_eager_decomp_partition`. It passes with `aot_eager`

My branch
* removes all the decomps except softmax from the decomp table.
* further fires off decomp only for the first softmax.

This limits the scope to just one decomp. But, I am out of ideas. I am unable to debug this further. The softmax decomp looks really simple. Would love to have someone look into this.

### Versions

N/A

cc @ezyang @msaroufim @wconstab @bdhirsh @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8