gopikrsmscs/torch-issues huggingface dataset:
when train with multi GPUS
Labels: oncall: distributed

### 🐛 Describe the bug

train command:
xport NGPUS=2
CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 --master_port=9531 ./train.py

but when traininf fow a while ,raise errors: what did it  means?


ARNING:torch.distributed.elastic.multiprocessing.api:Sending process 25236 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 1 (pid: 25237) of binary: /anaconda3/envs/paddle_env/bin/python
Traceback (most recent call last):
  File "/anaconda3/envs/paddle_env/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/anaconda3/envs/paddle_env/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/anaconda3/envs/paddle_env/lib/python3.8/site-packages/torch/distributed/launch.py", line 195, in <module>
    main()
  File "/anaconda3/envs/paddle_env/lib/python3.8/site-packages/torch/distributed/launch.py", line 191, in main
    launch(args)
  File "/anaconda3/envs/paddle_env/lib/python3.8/site-packages/torch/distributed/launch.py", line 176, in launch
    run(args)
  File "/anaconda3/envs/paddle_env/lib/python3.8/site-packages/torch/distributed/run.py", line 753, in run
    elastic_launch(
  File "/anaconda3/envs/paddle_env/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/anaconda3/envs/paddle_env/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 


### Versions

pytorch 13.1 cuda 116

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
torch.save() fails if path contains multibyte characters
Labels: module: serialization, triaged

### 🐛 Describe the bug

After upgrading PyTorch from 1.8.1 (LTS) to 2.0.1.
When saving a model using torch.save(), if the save destination path contains multi-byte characters, saving the model will fail.

Below is the sample code and error message.


Code to reproduce:
```python
import torch

model = torch.load("mnist_cnn.pt")
save_path = model_save.pt
torch.save(model, save_path) # OK

save_path_multibyte = "C:\\Users\\ユーザー名\\source\\model_save_multibyte.pt"        
torch.save(model, save_path_multibyte) # NG
```

The error message:
```
Traceback (most recent call last):
  File ".\sample.py", line 147, in <module>
    main()
  File ".\sample.py", line 142, in main
    torch.save(model, save_path_multibyte)
  File "C:\Users\ユーザー名\source\.venv\lib\site-packages\torch\serialization.py", line 440, in save
    with _open_zipfile_writer(f) as opened_zipfile:
  File "C:\Users\ユーザー名\source\.venv\lib\site-packages\torch\serialization.py", line 315, in _open_zipfile_writer
    return container(name_or_buffer)
  File "C:\Users\ユーザー名\source\.venv\lib\site-packages\torch\serialization.py", line 288, in __init__
    super().__init__(torch._C.PyTorchFileWriter(str(name)))
RuntimeError: Parent directory C:\Users\ユーザー名\source\ does not exist.
```

### Versions

PyTorch version: 2.0.1+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
Clang version: 11.0.1
CMake version: version 3.19.2
Libc version: N/A

Python version: 3.8.6 (tags/v3.8.6:db45529, Sep 23 2020, 15:52:53) [MSC v.1927 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19041-SP0
Is CUDA available: True
CUDA runtime version: 11.0.221
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2060 with Max-Q Design
Nvidia driver version: 512.78
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin\cudnn_ops_train64_8.dll
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=3000
DeviceID=CPU0
Family=107
L2CacheSize=4096
L2CacheSpeed=
Manufacturer=AuthenticAMD
MaxClockSpeed=3000
Name=AMD Ryzen 9 4900HS with Radeon Graphics
ProcessorType=3
Revision=24577

Versions of relevant libraries:
[pip3] numpy==1.19.3
[pip3] torch==2.0.1+cu118
[pip3] torchaudio==2.0.2+cu118
[pip3] torchvision==0.15.2+cu118
[conda] Could not collect

cc @mruberry @mikaylagawarecki
DISABLED test_gather_state_dict_dtensor (__main__.TestShardUtilsDistributedDTensor)
Labels: oncall: distributed, triaged, skipped

Platforms: linux

This test was disabled because it is failing on master ([recent examples](http://torch-ci.com/failure/distributed%2Ffsdp%2Ftest_shard_utils.py%3A%3ATestShardUtilsDistributedDTensor%3A%3Atest_gather_state_dict_dtensor)).

This looks related to https://github.com/pytorch/pytorch/issues/103863 and it fails on trunk multigpu test https://hud.pytorch.org/pytorch/pytorch/commit/4bd14d97f81f10373cf5ce2d272e7e9035cc8f83:

```
2023-06-20T23:41:02.6306643Z FAILED [2.9096s] distributed/fsdp/test_shard_utils.py::TestShardUtilsDistributedDTensor::test_gather_state_dict_dtensor - RuntimeError: Process 0 exited with error code 10 and exception:
2023-06-20T23:41:02.6307151Z Traceback (most recent call last):
2023-06-20T23:41:02.6307715Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 657, in run_test
2023-06-20T23:41:02.6308130Z     getattr(self, test_name)()
2023-06-20T23:41:02.6308678Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 543, in wrapper
2023-06-20T23:41:02.6309048Z     fn()
2023-06-20T23:41:02.6309592Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/_tensor/common_dtensor.py", line 174, in wrapper
2023-06-20T23:41:02.6310029Z     func(self)  # type: ignore[misc]
2023-06-20T23:41:02.6310623Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper
2023-06-20T23:41:02.6311054Z     return func(*args, **kwargs)
2023-06-20T23:41:02.6311814Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_shard_utils.py", line 81, in test_gather_state_dict_dtensor
2023-06-20T23:41:02.6312241Z     device_mesh = self.build_device_mesh()
2023-06-20T23:41:02.6312855Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/_tensor/common_dtensor.py", line 101, in build_device_mesh
2023-06-20T23:41:02.6313360Z     return DeviceMesh(DEVICE_TYPE, list(range(NUM_DEVICES)))
2023-06-20T23:41:02.6313940Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/_tensor/device_mesh.py", line 124, in __init__
2023-06-20T23:41:02.6314358Z     self._get_or_create_default_group()
2023-06-20T23:41:02.6314932Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/_tensor/device_mesh.py", line 135, in _get_or_create_default_group
2023-06-20T23:41:02.6315449Z     raise RuntimeError(
2023-06-20T23:41:02.6315822Z RuntimeError: Mesh should not be bigger than default world size, but found 4 ranks!
```

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
[dynamo] functools.wraps : graph-break when wrapping nested functions.
Labels: triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

Repro:
```python
import torch
from torch._dynamo.utils import counters
import json
from functools import wraps

counters.clear()

def foo(x, y):
    def add(x, y):
        return x + y

    @wraps(add)
    def wrapped_call(x, y):
        return add(x, y)

    return wrapped_call(x, y)

x = torch.ones(1,)
y = torch.ones(1,)

o = torch.compile(foo, fullgraph=False, backend='aot_eager')(x, y)

torch.testing.assert_close(o, 2 * x)
assert counters["graph_break"] == {
    "call_function wraps in skip_files /home/kshiteej/.conda/envs/pytorch-cuda-dev/lib/python3.9/functools.py": 1,
    "call_method UserDefinedObjectVariable(partial) __call__ [NestedUserFunctionVariable()] {}": 1
}
# print(json.dumps(counters, indent=2))

```
Note, that it doesn't graph-break for non-nested functions
```python
import torch
from torch._dynamo.utils import counters
import json
from functools import wraps

counters.clear()

def add(x, y):
        return x + y

def foo(x, y):
    @wraps(add)
    def wrapped_call(x, y):
        return add(x, y)

    return wrapped_call(x, y)

x = torch.ones(1,)
y = torch.ones(1,)

o = torch.compile(foo, fullgraph=False, backend='aot_eager')(x, y)

torch.testing.assert_close(o, 2 * x)
assert counters["graph_break"] == {}
# print(json.dumps(counters, indent=2))
```

Found in https://github.com/pytorch/pytorch/pull/102264 (this leads to graph-breaks when using nested functions with `torch.func.grad`)

### Versions

master

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @chenyang78
Docker images: faster linker for `torch.compile`
Labels: triaged, module: docker, oncall: pt2

### 🚀 The feature, motivation and pitch

As we are going to u se the linker can we switch by default to a faster linker in the official pytorch image (Gold, LLVM LD or [Mold](https://github.com/rui314/mold))

See also..
https://github.com/pytorch/pytorch/issues/103417


### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78
Can ``torch.vmap`` add ``grad_fn``= SelectBackward when maping over some dimension of the inputs?
Labels: triaged, actionable, module: functorch

### 🐛 Describe the bug

When applying ``torch.vmap`` to calculate gradients, I come up with the error ``element 0 of tensors does not require grad and does not have a grad_fn`` with the following code:
```python
import torch
from torch import vmap

# Setup
N = 5
def f(x):
    return torch.stack([x**2, x**3, x**4, x**5, x**6], dim=0)

x = torch.randn(N, requires_grad=True)
y = f(x)
basis_vectors = torch.eye(N)

def get_vjp(y, v):
    print(y)
    return torch.autograd.grad(y, x, v)[0]

jacobian_vmap = vmap(get_vjp, in_dims=(0,0))(y, basis_vectors)
```
I check ``y`` of ``get_vjp`` and the reason is that the mapping operation does not create a computational graph.

### Versions

PyTorch version: 2.0.1
CUDA used to build PyTorch: 11.8
OS: Ubuntu 22.04.2 LTS (x86_64)

cc @zou3519 @Chillee @samdow @kshitij12345 @janeyx99
DISABLED test_create_chunk_dtensor (__main__.TestShardUtilsDistributedDTensor)
Labels: oncall: distributed, triaged, skipped

Platforms: linux

This test was disabled because it is failing on master ([recent examples](http://torch-ci.com/failure/distributed%2Ffsdp%2Ftest_shard_utils.py%3A%3ATestShardUtilsDistributedDTensor%3A%3Atest_create_chunk_dtensor)).

This test is failing on trunk with the following error https://hud.pytorch.org/pytorch/pytorch/commit/3e42854caa6bfd4cd2694b7da0b13ddd9695ca58:

```
2023-06-19T19:45:30.2835537Z =========================== short test summary info ============================
2023-06-19T19:45:30.2836236Z FAILED [2.8097s] distributed/fsdp/test_shard_utils.py::TestShardUtilsDistributedDTensor::test_create_chunk_dtensor - RuntimeError: Process 0 exited with error code 10 and exception:
2023-06-19T19:45:30.2836752Z Traceback (most recent call last):
2023-06-19T19:45:30.2837391Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 657, in run_test
2023-06-19T19:45:30.2837814Z     getattr(self, test_name)()
2023-06-19T19:45:30.2838344Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 543, in wrapper
2023-06-19T19:45:30.2838742Z     fn()
2023-06-19T19:45:30.2839347Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/_tensor/common_dtensor.py", line 174, in wrapper
2023-06-19T19:45:30.2839777Z     func(self)  # type: ignore[misc]
2023-06-19T19:45:30.2840340Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 174, in wrapper
2023-06-19T19:45:30.2840755Z     return func(*args, **kwargs)
2023-06-19T19:45:30.2841180Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_shard_utils.py", line 64, in test_create_chunk_dtensor
2023-06-19T19:45:30.2841588Z     device_mesh = self.build_device_mesh()
2023-06-19T19:45:30.2842211Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/_tensor/common_dtensor.py", line 101, in build_device_mesh
2023-06-19T19:45:30.2842710Z     return DeviceMesh(DEVICE_TYPE, list(range(NUM_DEVICES)))
2023-06-19T19:45:30.2843264Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/_tensor/device_mesh.py", line 124, in __init__
2023-06-19T19:45:30.2843685Z     self._get_or_create_default_group()
2023-06-19T19:45:30.2844267Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/_tensor/device_mesh.py", line 135, in _get_or_create_default_group
2023-06-19T19:45:30.2844679Z     raise RuntimeError(
2023-06-19T19:45:30.2845023Z RuntimeError: Mesh should not be bigger than default world size, but found 4 ranks!
```

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Long PR description leads to "Argument list too long" error from docker
Labels: module: ci, triaged, module: devx

### 🐛 Describe the bug

Per title. 

Also, if your PR description contains a error message it can trick the hud into highlighting that line.

An example: https://github.com/pytorch/pytorch/pull/103735 (I've since updated to link to the error message in a gist, see edit history for the original PR description)

### Versions

main

cc @seemethere @malfet @pytorch/pytorch-dev-infra @ZainRizvi @kit1980 @huydhn @clee2000
[ONNX] FX exporter: replace `aten::copy_` with out-place version
Labels: module: onnx, triaged

In-place ops like `aten::copy_` should be replaced or removed before converting to ONNX.

Standard cases are handled by functionalization pass. This issue targets edge cases like [nn.InstanceNorm with track_running_stats=True](https://pytorch.org/docs/master/generated/torch.nn.InstanceNorm2d.html#torch.nn.InstanceNorm2d), which **updates model state** even in inference mode. 

Perhaps solution is to write an `unsafe_remove_inplace_pass` that emits diagnostic warnings to detect and handle these inplace ops. 

Repro: 

```python
import torch

class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.instance_norm = torch.nn.InstanceNorm1d(3, affine=True, track_running_stats=True)

    def forward(self, x, y):
        out = x + y
        out = self.instance_norm(out)
        out = out + x
        return out

model = Model()
model.eval()
x = torch.randn(1, 3, 3)
y = torch.randn(1, 3, 3)

torch.onnx.dynamo_export(model, x, y)
```

cc @BowenBao @thiagocrepaldi @wschin
The document style is inconsistent with other documents, and the parameter type is not clearly highlight
Labels: module: docs, triaged, actionable

### 📚 The doc issue

The document style is inconsistent with other documents, and the parameter type is not clearly highlight in torch.nn.functional.conv_transpose3d、torch.nn.functional.conv_transpose2d、torch.nn.functional.bilinear、torch.nn.functional.conv1d、torch.nn.functional.max_unpool3d、torch.nn.functional.max_pool3d、torch.nn.functional.conv3d、torch.nn.MultiheadAttention、torch.nn.functional.avg_pool1d、torch.nn.functional.avg_pool3d、torch.nn.functional.max_pool2d、torch.nn.functional.linear、torch.nn.functional.max_pool1d.
![image](https://github.com/pytorch/pytorch/assets/45327670/f61a1fc2-d70c-4b6a-ad38-9a8579305cd6)
![image](https://github.com/pytorch/pytorch/assets/45327670/d04ebb60-305f-41de-89ec-50dc7e2ba42e)


### Suggest a potential alternative/fix

_No response_

cc @svekars @carljparker
Missing examples in some API docs
Labels: module: docs, triaged, actionable

### 📚 The doc issue

torch.nn.functional.mse_loss、torch.are_deterministic_algorithms_enabled、torch.nn.functional.batch_norm、torch.nn.functional.triplet_margin_loss、torch.nn.RNNBase missing examples in API docs.

### Suggest a potential alternative/fix

_No response_

cc @svekars @carljparker
[question] [docs] Short/mid/long-term status of TorchScript / JIT / torch.jit.trace / FX / symbolic tracing and its replacement by Dynamo
Labels: oncall: jit, module: docs, triaged

### 📚 The doc issue

At OP https://github.com/pytorch/vision/pull/7624#discussion_r1233691474 I stumbled that there might be lack of official communication on short-term / mid-term / long-term status of TorchScript / JIT interpreter.

In that OP case, need to support TorchScript necessitated use of inelegant indexing instead of simpler `tensor[..., perm, :, :]` which seems not supported by TorchScript (btw is this construct supported by Dynamo?)

Any official information on need to support TorchScript in new code of domain libraries / in general? Does TorchScript continue to receive development? Any deprecation calendar? I would propose that this information / overview (even if no decisions made yet) made very clear directly on main page of https://pytorch.org/docs (preferrably at the top of this long page)

This question comes from the fact that PyTorch had offered many tracing/scripting/compilation technologies over the years, their recipes/tutorials are still all available, and the users need to have information at least about mid-term level plans and recommendations of the core team. My related question on deployment as well: https://discuss.pytorch.org/t/torch-compiles-deployment-story-to-non-python-host-processes/180943/4 - would be good to have this explained very clearly as well

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel @svekars @carljparker @ezyang @albanD
Gradient operations (zero_grad and gradient accumulations) as graphs
Labels: module: autograd, triaged, oncall: pt2

### 🚀 The feature, motivation and pitch

Hi,

We already can see in PT2.0 that operations which are not part of main FWD/BWD/OPT blocks - especially **zero_grad** and **gradient accumulation** - can become serious host bottleneck due to being scaled with topology size and executed eagerly.

Are there any plans to incorporate these operations into the graphs in PT2.1 or later? Thanks.



### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @msaroufim @wconstab @bdhirsh @anijain2305
Please consider the SCFA/dynamic flash attention for your implementation of scaled dot product attention
Labels: module: sparse, triaged, oncall: transformer/mha, topic: new features

### 🚀 The feature, motivation and pitch

Sparse Causal Flash Attention as implmented [here](https://github.com/epfml/dynamic-sparse-flash-attention) and described in [this paper](https://arxiv.org/abs/2306.01160) claims to 

> extend FlashAttention to accommodate a large class of attention sparsity patterns that, in particular, encompass key/query dropping and hashing-based attention. This leads to implementations with no computational complexity overhead and a multi-fold runtime speedup on top of FlashAttention. Even with relatively low degrees of sparsity, our method improves visibly upon FlashAttention as the sequence length increases. Without sacrificing perplexity, we increase the training speed of a transformer language model by 2.0× and 3.3× for sequences of respectively 8k and 16k tokens.

Adding these implementations to main Pytorch might lead to Transformers being able to use both flash attention, PEFT/LoRA and methods for handling long contexts

### Alternatives

1. Leaving everything as is and finalising the current implementation to allow handling attention masks (see issue #96099 )
2. Relying on flash_attn by HazyResearch, which works great on its own but apparently does not work with LoRA

### Additional context

I'm just not sure functions written in triton are readily convertible into C++ functions.

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer @jbschlosser @erichan1 @drisspg
Torch 1.13 for GPU breaks if libcublas is already present.
Labels: triaged, module: docker

### 🐛 Describe the bug

When I install pytorch into the docker image `gcr.io/deeplearning-platform-release/base-cu113` from https://cloud.google.com/deep-learning-containers/docs/choosing-container or `nvidia/cuda:11.3.1-cudnn8-runtime-ubuntu20.04` with gpu, it installs also installs `nvidia-cublas-cu11` which is a dependency, but it's already present in these images, so when importing torch, it errors out, see
```python
>>> import torch
Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/my-env-jEFYeG2G-py3.10/lib/python3.10/site-packages/torch/__init__.py", line 172, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/opt/conda/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /root/.cache/pypoetry/virtualenvs/my-env-jEFYeG2G-py3.10/lib/python3.10/site-packages/torch/lib/../../nvidia/cublas/lib/libcublas.so.11: undefined symbol: cublasLtGetStatusString, version libcublasLt.so.11
```
see https://stackoverflow.com/a/75095447/5224881 where people were also struggling with it.
The cuda toolkit is automatically installed on GCP docker images, so it makes it hard to use.
When I uninstall the `nvidia-cublas-cu11`, it starts working, but I would like to be able to use it purely using poetry
Of course I can manually uninstall the dependency, but option to install or uninstall it using extras would be very welcome.

### Versions

[pip3] flake8==3.9.2
[pip3] mypy==0.950
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.24.3
[pip3] torch==1.13.1
[conda] numpy                     1.23.5                   pypi_0    pypi
RPC Framework support custom backend
Labels: oncall: distributed, triaged

### 🚀 The feature, motivation and pitch

Custom backend(PrivateUse1) also has requirements for the RPC framework, and hopes to provide a registration method or implement a common logic. 

I'm currently reading the code of the RPC part, according to my analysis and research, the RPC module has a great dependence on the third-party project tensorpipe, and the important channel module is also implemented in it.

I understand that the current RPC module needs to support third-party devices, and there are several things to do :
1. Extend the corresponding device logic in the RPC module of pytorch project to provide a processing branch for third-party devices. 
2. Third-party devices need to inherit the `TensorpipeDeviceTypeConverter `class to implement their own converter class, and complete the preparation for sending and receiving of devcie tensor. And correctly registered in pytorch to ensure that the object can be obtained correctly
3. Third-party devices need to refer to cuda_basic channel, implement their own basic channel, and register

Our current demands:
1. At present, for the pytorch project, we can improve and expand the corresponding device logic.
2. For the tensorpip project, we hope to expose more header files, so that third-party devices can implement their own basic channel based on tensorpip

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @ezyang.

### Alternatives

_No response_

### Additional context
1. The current tensorpipe project cannot contribute or discuss etc.
_No response_
Upgrading SpGEMM algorithm to resolve Cusparse SpGEMM insufficient resources problem
Labels: module: sparse, module: cuda, triaged

### 🚀 The feature, motivation and pitch

The SpGEMM algorithm in cuda 11.x version requires high amount of memory for the sparse computation. In CUDA 12, two new SpGEMM algorithms has been introduced to resolve the problem. I really hope that the new algorithms can be integrated to pytorch (Providing a solution to use the new algorithms is also exciting : ) ). Thanks.
Please see https://github.com/NVIDIA/CUDALibrarySamples/issues/38.

### Alternatives

_No response_

### Additional context

_No response_

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer @ptrblck
Add alias support for sparse tensors.
Labels: open source, release notes: sparse, topic: bug fixes, topic: not user facing, ciflow/periodic

This PR addresses the sparse tensors part of https://github.com/pytorch/pytorch/issues/99655

The PR introduces the following utility functions:
- `at::sparse_csr::alias_with_values(a_sparse_compressed_tensor, new_values)`
- `at::sparse::alias_with_values(a_sparse_tensor, new_values)`

These functions return a wrapper of a sparse tensor with new specified values that allow introducing alias support for sparse tensors and more (e.g. the most efficient way to resolve https://github.com/pytorch/pytorch/pull/99292#discussion_r1232201104 is to use `at::sparse_csr::alias_with_values(self, self.values().to(dtype))` as a replacement of `self.to(dtype)` to avoid the unnecessary copy of indices).

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #104145
* __->__ #103810
* #108512
Support `Sequence` type in JIT
Labels: oncall: jit, module: typing, triaged, enhancement

### 🚀 The feature, motivation and pitch

This is a sub-task of #103798.

We have functions annotated to accept `List[int]` but actually get `torch.Size` passed in, e.g.:
https://github.com/pytorch/pytorch/blob/918fe519a0f9bdb2dd213eaaffbcd420b7391280/torch/nn/functional.py#L2435
https://github.com/pytorch/pytorch/blob/918fe519a0f9bdb2dd213eaaffbcd420b7391280/torch/nn/functional.py#L2481

However, static type checkers like mypy does not like it because `torch.Size`, which is a subtype of `Tuple[int, ...]`, is not compatible with `List[int]`. This conflicts with JIT, which does not understand `Tuple[int, ...]` without a fixed length.

There are many cases of this problem, all involving functions taking in tensor sizes or similar (e.g., `Tensor.size` or `kernel_size`), where we only use them as `collections.abc.Sequence`.

Therefore, I propose to annotate them as `Sequence[int]` and ask JIT to interpret `Sequence[int]` as `List[int]`.

By annotating the sizes as `Sequence[int]`, we can i) make mypy happy because `List[int]` and `Tuple[int, ...]` (including `torch.Size`) are both `Sequence[int]`, and ii) borrow the JIT infrastructure for `ListType` which does not care about the length, bypassing the tuple length problem.

### Alternatives

1. _Adjust the implementation of `TupleType` in JIT to support `Tuple[int, ...]`_. This can be the best solution, but I don't know how to do it. Also, I assume it's not easy because otherwise it would already be implemented.
2. _Implement `SequenceType`, which differs from `ListType`_. This is also a good option, maybe second to the above, because it's safer than reusing the `ListType` -- some operations supported by `List` are not by `Sequence`, although JIT seems not to be using them.
3. _Add dozens of `type:ignore`s to suppress all related mypy warnings_. The good point is: the affected functions are for internal use (with `_` prefix) and will not affect public APIs (i.e., the users will not see any problem). However, this can leave the code error-prone.

### Additional context

Since we mark them as `List[int]` for now, even with `Tuple`s/`Size`s, directly reusing `List` for `Sequence` will not cause any problems. We actually keep how everything works under the hood, just switching to a different name that can satisfy mypy.

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel @ezyang @malfet @rgommers @xuzhao9 @gramster
Mis-annotated return for `F._no_grad_embedding_renorm_` (also JIT related)
Labels: oncall: jit, module: typing, triaged, bug

### 🐛 Describe the bug

This is a sub-task of #103761.

This is not user-facing as this only involves functions prefixed with an underscore.

`_no_grad_embedding_renorm_` has mismatched typing on the return value:
https://github.com/pytorch/pytorch/blob/918fe519a0f9bdb2dd213eaaffbcd420b7391280/torch/nn/functional.py#L2123-L2124

1. This function does not return anything at all. This is modified on purpose in #18684. (PS: the return value is not used anyway.)
2. The return now got annotated as a `Tuple[Tensor, Tensor]` in #78560. I don't know why it's changed in this way. @samdow Can you please explain this as the author of that PR?

---

Other occurrences of `_no_grad_embedding_renorm_`: (I'm not quite sure whether these are related, nor which is correct)
https://github.com/pytorch/pytorch/blob/918fe519a0f9bdb2dd213eaaffbcd420b7391280/torch/csrc/api/include/torch/nn/functional/embedding.h#L15-L22
https://github.com/pytorch/pytorch/blob/918fe519a0f9bdb2dd213eaaffbcd420b7391280/torch/csrc/jit/runtime/register_special_ops.cpp#L330
https://github.com/pytorch/pytorch/blob/918fe519a0f9bdb2dd213eaaffbcd420b7391280/torch/csrc/jit/runtime/serialized_shape_function_registry.cpp#L3196
https://github.com/pytorch/pytorch/blob/918fe519a0f9bdb2dd213eaaffbcd420b7391280/torch/jit/_shape_functions.py#L1109

### Versions

master

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel @ezyang @malfet @rgommers @xuzhao9 @gramster
[PT2] Return int32 indices in max_pool2d_with_indices
Labels: triaged, oncall: pt2

### 🚀 The feature, motivation and pitch

Another case of https://github.com/pytorch/pytorch/issues/103475. 

If we guard on the tensor having numel < 2^32, we can return int32 indices. On one max_pool2d_with_indices_backward kernel I tested in the backward, using int32 instead of int64 sped it up 27%. 

Related: https://github.com/pytorch/pytorch/issues/103111

Because we sometimes fallback to eager in lowering max_pool2d_backward, we'd either have to figure out a way not to fallback eager or add eager support for int32 in max_pool2d_backward.

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
[ONNX] Handle absence of `onnxscript` module in PyTorch requirements.txt
Labels: module: onnx, triaged, enhancement

### 🐛 Describe the bug

Currently, the new Dynamo-based ONNX Exporter heavily depends on ONNX Script (aka `onnxscript` module). However, PyTorch does not take dependency on it, officially

To prevent `ModuleNotFound` during `import torch`, the `torch.onnx` namespace is deferring `import onnxscript` into runtime, out of the initialization path.

We need to find a better way to handle this.

### Versions

main
Merge type stubs for `torch.nn.functional`
Labels: module: typing, triaged

### 🚀 The feature, motivation and pitch

From the discussion in https://github.com/pytorch/pytorch/issues/102918#issuecomment-1587576763 (cc @drisspg):

Similar to #102428, we would like to merge the .pyi type stub in the .py file for `torch.nn.functional`.

### Alternatives

Keep the .pyi file which is now generated by [`gen_pyi.py`](https://github.com/pytorch/pytorch/blob/main/tools/pyi/gen_pyi.py) from a .pyi.in template.

### Additional context

The pyi for `torch.nn.functional` consists of three parts:
- Functions imported from `torch` or `torch._C._nn` and passed through `_add_docstr`.
  - Functions in `torch` already have typing, #102918 has addressed the typing for `torch._C._nn`, and `_add_docstr` has been annotated to pass through the typing. Therefore, this part can be considered resolved.
- Functions generated by `torch._jit_internal.boolean_dispatch`. Their annotations are currently generated by `gen_pyi`.
  - We can move the generated overloads into the .py file. They should work directly.
  - I assume these do not involve `torch._jit_internal._overload` because they're already `boolean_dispatch`, **but need to be confirmed**.
- Functions directly defined in `nn/functional.py`.
  - They should already have annotation in the .py file, **but may need to be verified**.
  - By the following comment, these functions **may need special treatment to make both mypy and JIT work**, which I cannot imagine now (possibly blocks us from merging pyi): https://github.com/pytorch/pytorch/blob/5875a2fb3c4f21efec326dff6ba31d793a7194e1/torch/nn/functional.pyi.in#L30-L32

cc @ezyang @malfet @rgommers @xuzhao9 @gramster

---

This issue can be decomposed into several relatively self-contained sub-tasks, corresponding to the logs recorded below:
(I will open issues for each one and specify the sub-task in detail.)

```[tasklist]
### Tasks
- [ ] mypy does not recognize `BroadcastingList*`
- [ ] mypy does not recognize overloads decorated by `_jit_internal._overload`
- [ ] https://github.com/pytorch/pytorch/issues/103787
- [ ] https://github.com/pytorch/pytorch/issues/103806
- [ ] https://github.com/pytorch/pytorch/issues/103798
- [ ] https://github.com/pytorch/pytorch/issues/103800
```
SDPA produces NaN with padding mask
Labels: triage review, triaged, oncall: transformer/mha

### 🐛 Describe the bug

I'm implementing padding support directly on my LLM model. To do so, I add extra rows to the boolean attention mask with all `False` values.

However, calling `torch.nn.functional.scaled_dot_product_attention` returns NaNs.

An equivalent and naive implementation also does (see `naive_fsdpa_1`).
But if I use `torch.finfo(att.dtype).min` as the masked fill value, the result is correct (see `naive_fsdpa_2`)

Here's the repro:

```python
import torch

scale = 0.125

# 5 tokens, 1 padding
seq_length = 5
max_seq_length = 10

q = torch.randn(1, 2, seq_length + 1, 8)
k = torch.randn(1, 2, max_seq_length, 8)
v = torch.randn(1, 2, max_seq_length, 8)
mask = torch.tensor([[[
    [ True, False, False, False, False, False, False, False, False, False],  # regular mask
    [ True,  True, False, False, False, False, False, False, False, False],
    [ True,  True,  True, False, False, False, False, False, False, False],
    [ True,  True,  True,  True, False, False, False, False, False, False],
    [ True,  True,  True,  True,  True, False, False, False, False, False],
    [False, False, False, False, False, False, False, False, False, False]]]])  # padding mask



def torch_sdpa(q, k, v, mask):
    return torch.nn.functional.scaled_dot_product_attention(q, k, v, mask, dropout_p=0.0, scale=scale)


def naive_sdpa_1(q, k, v, mask):
    att = (q @ k.transpose(-2, -1)) * scale
    att = torch.masked_fill(att, ~mask, float("-inf"))
    att = torch.nn.functional.softmax(att, dim=-1)
    return att @ v


def naive_sdpa_2(q, k, v, mask):
    att = (q @ k.transpose(-2, -1)) * scale
    att = torch.masked_fill(att, ~mask, torch.finfo(att.dtype).min)
    att = torch.nn.functional.softmax(att, dim=-1)
    return att @ v

y = torch_sdpa(q, k, v, mask)
print(torch.isnan(y).any())

y = naive_sdpa_1(q, k, v, mask)
print(torch.isnan(y).any())

y = naive_sdpa_2(q, k, v, mask)
print(torch.isnan(y).any())
```

This seems to suggest that there's a precision issue. Is this expected or known? Could `scaled_dot_product_attention` be fixed?

Side question: do I lose flash attn if I append the padding mask like this?

Thank you for your help.

Maybe linked: https://github.com/pytorch/pytorch/issues/101967

### Versions

Current master

cc @ezyang @gchanan @zou3519 @jbschlosser @bhosmer @cpuhrsch @erichan1 @drisspg @msaroufim @wconstab @bdhirsh @anijain2305
[ONNX] Discuss improvements to Diagnostic public API
Labels: module: onnx, triaged

### 🐛 Describe the bug

Currently, when we use the diagnostic system through `@diagnostics.diagnose_call`, we need to create an extra utility function, that usually is only used once because it is function-signature-specific.

To diagnose
```def foo(a: int, b: float, c : bool)
      pass
```

We need to first create a helper:
```
def _foo_formatter(fx: Callable, diagnostic_context: diagnostics.DiagnosticContext,, a: int, b: float, c : bool)
    return "string_consuming_a_or_b_or_c"
```

Next we decorate the function
```
    @diagnostics.diagnose_call(
        diagnostics.rules.XXX,
        diagnostic_message_formatter=_foo_formatter,
    )
```

This is error prone because 
* every time the function/method signature changes, the utility signature must also change, which creates unnecessary coupling.
* beartype error message is not trivial to understand
* call site becomes cluttered with utlities flying around

A proposal is to replace the formatter as callable to string.
We can leverage how decorators are implemented and automatically create a hidden lambda/utility function that mimics the callable signature.
`diagnose_call` receives the callable, then it inspects the callable and find out the function/method signature. Next it builds a temporary function with the inspected signature and its implementation `return USER_FORMAT_STRING`

That would make the use of the decorator simpler to the user because they won't need to change the utility signature every time the callable also changes. They only pass the string they want to print, like python formatted strings already work.

### Versions

main
[dtensor] introduce experimental dmap
Labels: release notes: distributed (dtensor)

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #103686

This PR introduces `dmap`, this allows user to write function with local
shard view inside the function, i.e. write manual collectives. the dmap
can work together with DTensor operations.

i.e. We can do some DTensor computation first, then call a dmap function
to write some manual collectives, then use the results to do some
additional DTensor computations.

Given that we already have APIs to explicitly convert torch.Tensor to/from
DTensor, this API just serves as a convenient tool to write simple funcs
that works seamlessly with DTensor.

caveat: manual collectives always need to couple with autograd.Function so
that it works for backward. Ideally we should just make our functional collectives
to have gradient formulas, but that might be a bit tricky to implement.
fairseq distributed training dumps core with flash attention
Labels: triaged, oncall: transformer/mha

### 🐛 Describe the bug

When doing distributed training with Fairseq for an NMT model, flash attention core dumps. (Adding here for issue tracking, internal discussion for details).

The reported source location of the core dump is consistent across multiple repros:

```
Opening GPU coredump: /tmp/cudacoredump_6-gpu_A100.devgpu003.prn6.facebook.com.3918025.1686793663

CUDA Exception: Warp Illegal Address
The exception was triggered at PC 0x7f3279226060 (utils.h:819 in _ZN4fmha11Ldg_functorI5uint4Li1EE4loadEib inlined from utils.h:120)
[Current focus set to CUDA kernel 0, grid 200269, block (0,0,0), thread (0,0,0), device 0, sm 0, warp 1, lane 0]
#0  fmha_bwd_dot_do_o_kernel<FMHA_kernel_traits<256, 64, 16, 1, 8, 256u, __half> ><<<(2,16,3),(256,1,1)>>> ()
    at /re_cwd/./fbcode/third-party-buck/platform010/build/cuda/11.4.2/include/cuda_fp16.hpp:733 in _ZN54_INTERNAL_32_fmha_bwd_hdim64_cu_pic_o_cpp1_ii_a4e28c6c14__half22float2E7__half2 inlined from cuda_fp16.hpp:539
733 /re_cwd/./fbcode/third-party-buck/platform010/build/cuda/11.4.2/include/cuda_fp16.hpp: No such file or directory.
```

Repro internally by Wei Ho.
cc: @drisspg 

### Versions

fbcode trunk head

cc @jbschlosser @bhosmer @cpuhrsch @erichan1 @drisspg
Exported model with dropout incorrectly applies dropout during eval
Labels: module: nn, triaged, module: correctness (silent), oncall: pt2, module: export

Normally calling `model.eval()` on a model with dropout changes the behavior of the dropout ops into a noop. However, this is not true if model is already exported, since the aten dropout pattern remains in the graph. Using the resulting model for inference led to incorrect computation, since we continue to apply dropout to the inputs during eval, which is unexpected. This affects QAT for the PT2 flow, where all transformations are done on an exported model.

Simple repro:
```
import torch
import torch._dynamo

class MyModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.dropout = torch.nn.Dropout(p=0.5)
    def forward(self, x):
        return self.dropout(x)

model = MyModel()
model, _ = torch._dynamo.export(model, torch.randn(1), aten_graph=True)
model.eval()
print(model)
```

This prints:
```
def forward(self, x):
    arg0, = fx_pytree.tree_flatten_spec(([x], {}), self._in_spec)
    empty_like_default = torch.ops.aten.empty_like.default(arg0, memory_format = torch.contiguous_format)
    bernoulli__float = torch.ops.aten.bernoulli_.float(empty_like_default);  empty_like_default = None
    div__scalar = torch.ops.aten.div_.Scalar(bernoulli__float, 0.5);  bernoulli__float = None
    mul_tensor = torch.ops.aten.mul.Tensor(arg0, div__scalar);  arg0 = div__scalar = None
    return pytree.tree_unflatten([mul_tensor], self._out_spec)
```

But it's supposed to print:
```
def forward(self, x):
    arg0, = fx_pytree.tree_flatten_spec(([x], {}), self._in_spec)
    return pytree.tree_unflatten([arg0], self._out_spec)
```

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
[dynamo] Add config that turns on tracing through nn modules
Labels: Stale, module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #104360
* __->__ #103676
* #103987

After: https://gist.github.com/soulitzer/c747d3e9cb1e241f6c7b9b57c0f84a9b
Before: https://gist.github.com/soulitzer/4da95730e40d814aa0c64cdff5c48571

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @anijain2305 @ipiszy @msaroufim
detectron2_fcos_r_50_fpn and other models have enough graph breaks that we end up with multiple cache entries on module blocks
Labels: triaged, module: dynamic shapes

### 🐛 Describe the bug

If you run detectron2_fcos_r_50_fpn with automatic_dynamic_shapes, you'll notice that there are dynamic shapes recompilations, which aren't really true dynamic shape recompilations:

```
[2023-06-15 05:43:07,747] torch.fx.experimental.symbolic_shapes: [INFO] 12.0: create_env
[2023-06-15 05:43:08,412] torch.fx.experimental.symbolic_shapes: [INFO] 12.0: produce_guards
[2023-06-15 05:43:08,423] torch.fx.experimental.symbolic_shapes: [INFO] 12.1: create_env
[2023-06-15 05:43:08,425] torch._dynamo.variables.builder: [WARNING] automatic dynamic L['input'] size(1) 256 != 64
[2023-06-15 05:43:08,425] torch.fx.experimental.symbolic_shapes: [INFO] 12.1: create_symbol s0 = 256 for L['input'].size()[1]
[2023-06-15 05:43:08,525] torch.fx.experimental.symbolic_shapes: [INFO] 12.1: eval Eq(s0, 256) [guard added] at home/ezyang/local/b/pytorch-env/lib/python3.10/site-packages/detectron2/layers/wrappers.py:106 in forward (_subclasses/fake_tensor.py:612 in conv)
[2023-06-15 05:43:09,075] torch.fx.experimental.symbolic_shapes: [INFO] 12.1: produce_guards
```

Instead, they're due to the architecture of detectron2_fcos_r_50_fpn, which stacks a number of similar blocks together but with differing parameters. For Dynamo's intents and purposes, these are all using the same code block, and so each block with differing configuration all gets chucked in the same code cache.

I think if our policy is that we want to specialize on every unique parameter size (this seems reasonable to me), it would be best if we could arrange for our compiled code to NOT live on all the same code object.

@anijain2305 also mentioned to me that he noticed this could contribute to guard overhead, although ISTR that in the end it wasn't clear how much this actually mattered in the end.

### Versions

main
"Y.getIntrusivePtr()->set_storage(X.getIntrusivePtr()->storage()); " in C++ is not supported
Labels: module: cpp, triaged

### 🚀 The feature, motivation and pitch

There is no set_storage() support for getIntrusivePtr() object.
I want to copy from X to Y where X and Y are from different devices (i.e. X.device() != Y.device()).
However, two devices share the global memory address with each other, so I prefer just moving the storage point from X to Y to avoid expensive data copy.

The following code in C++ doesn't work for me, as it fails for undefined `set_storage()`:
```cpp
Y.getIntrusivePtr()->set_storage(X.getIntrusivePtr()->storage())
```

If I perform a shallow copy using:
```cpp
Y.getIntrusivePtr()->shallow_copy_from(X.getIntrusivePtr());
```

It is also not what I want, because it will change Y's device to be X's device as well, making following computation of Y not performed on original Y's expected device.

Any alternative way to share the storage link. or any supporting plans?

### Alternatives

_No response_

### Additional context

_No response_

cc @jbschlosser
MultiheadAttention should split embed_dim into four parameters
Labels: triaged, oncall: transformer/mha

### 🚀 The feature, motivation and pitch

The assertions around embed_dim are in `nn.MultiheadAttention` and `F.multi_head_attention_forward` too restrictive. 
The embed_dim currently seems to be a “catch-all” parameter, although the multi-head attention fundamentally supports four different dimensions to coexist, which are currently all noted as `embed_dim`:

1) The input dimension of the query - doesn’t need to be divisible by num_heads,
2) The embed_dim for query and key - this should be divisible by num_heads,
3) The embed_dim for value as well the input dimension of the out_projection - this should be divisible by num_heads,
4) The output dimension of out_proj - doesn’t need to be divisible by num_heads.

### Alternatives

Split up the embed_dim into four, i.e.
1) qdim,
2) qk_embed_dim,
3) vo_embed_dim,
4) out_dim,
and loosen the head_divisibility checks for qdim and out_dim. This would allow much more fine-grained adjustment of the MHA module, e.g. in the context of structured pruning.


### Additional context

_No response_

cc @jbschlosser @bhosmer @cpuhrsch @erichan1 @drisspg