gopikrsmscs/torch-issues huggingface dataset:
torchscript dataclasses have bad support for class types as fields
Labels: oncall: jit

### 🐛 Describe the bug

In `_dataclass_impls.py`, there's a step where the `__init__` function signature is collected and then serialized as a string. During that process, it seems like the serialized names are not always readable by PyTorch. In the example below, the type for `MyClass` gets serialized as `__main__.MyClass`. I'm assuming that the failure happens later on when Torchscript's RCB  isn't able to find `__main__.MyClass`.

A fix for this might be to find a different way to annotate the signature when it is serialized, e.g. by using the annotation types found on the dataclass fields themselves.

```python
from dataclasses import dataclass

from typing import Dict

import torch


class MyClass:
    x: int
    y: float

    def __init__(self, x: int, y: float):
        self.x = x
        self.y = y

    def eval(self) -> float:
        return self.y**self.x


@torch.jit.script
@dataclass
class Batch:
    float_features: Dict[int, torch.Tensor]
    mc: MyClass

    def __eq__(self, other: "Batch") -> bool:
        for k in self.float_features:
            if k in other.float_features:
                if not torch.equal(self.float_features[k], other.float_features[k]):
                    return False
            else:
                return False

        return self.mc == other.mc


class ASD(torch.nn.Module):
    def __init__(self) -> None:
        super().__init__()

    def forward(self, batch: Batch) -> torch.Tensor:
        return batch.float_features


model = ASD()
torch.jit.script(model)
```

### Versions

pytorch 2.0 / main branch as of submission

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Error when exporting to onnx for albert-base-v2, issue with attention_mask
Labels: module: onnx, triaged

### 🐛 Describe the bug

When attempting to export the following pretrained onnx model below, which uses [albert-base-v2](https://huggingface.co/albert-base-v2), I get an error about not being able to unsqueeze the attention mask even though it is clearly a list AND the model does original model outputs as expected.

Code:
```
input_ids = [2, 184, 20, 745, 2667, 11318, 18, 37, 4335, 4914,3]
token_type_ids = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
attention_mask = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

input_ids_tensor = torch.tensor([input_ids])
token_type_ids_tensor = torch.tensor([token_type_ids])
attention_mask_tensor = torch.tensor([attention_mask])

cur_state_dict = torch.load("./albert_model/checkpoint-69203/pytorch_model.bin", map_location="cpu")

model = AutoModelForSequenceClassification.from_pretrained("./albert_model/checkpoint-69203")

model.load_state_dict(cur_state_dict, strict=False)
model.eval()

with torch.no_grad():
    output = model(input_ids_tensor, token_type_ids_tensor, attention_mask_tensor)
    print("output: ", output)

    torch.onnx.export(model, 
                    (input_ids_tensor, token_type_ids, attention_mask), 
                    "new_onnx_model", 
                    export_params=True, 
                    opset_version=16,
                    do_constant_folding=True,
                    input_names = ["input_ids", "token_type_ids", "attention_mask"],
                    output_names = ["logits", "scores"])
```


The error message for the "torch.onnx.export" doesn't make sense because the "attention_mask" should be a list
<img width="598" alt="image" src="https://github.com/pytorch/pytorch/assets/135020113/54ad99f3-b711-490f-bad1-a9cdc4619195">


### Versions

Collecting environment information...
PyTorch version: 1.13.1+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.8.10 (default, Mar 13 2023, 10:26:41)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.0.0-1032-azure-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: 11.1.105
CUDA_MODULE_LOADING set to: LAZY

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] pytorch-nlp==0.5.0
[pip3] pytorch-pretrained-bert==0.6.2
[pip3] torch==1.13.1+cu116
[pip3] torchaudio==0.13.1+cu116
[pip3] torchvision==0.14.1+cu116
[conda] Could not collect
[inductor] Memory planning
Labels: topic: not user facing, module: inductor, module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #102625
* #111402
* #111117



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @anijain2305
Extremely slow will_fusion_create_cycle on nanogpt_generate
Labels: triaged, bug, oncall: pt2, module: inductor

### 🐛 Describe the bug

nanogpt_generate in the benchmark suite compiles extremely slowly. When looking at py-spy top, it looks like we spend a lot of time in will_fusion_create_cycle. I noticed the model is 40k nodes, which is probably triggering some bad asymptotics.

### Versions

main

cc @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225
Fix sparse windows
Labels: module: sparse, module: windows, module: mkl, open source, ciflow/binaries, release notes: releng, ciflow/periodic

Fix https://github.com/pytorch/pytorch/issues/97352.
This PR changes the way the linking to intel MKL is done and updating MKL on Windows to mkl-2021.4.0 .
There are for both conda and pip packages MKL  version with which you can link dynamically. mkl-devel contains the static versions of the dlls and MKL contains the needed dlls for the runtime. MKL dlls and static libs starting with  2021.4.0 have the version in their names( for MKL 2023 we have mkl_core.2.dll and for 2021.4.0 we have mkl_core.1.dll) so its possible to have multiple versions installed and it will work properly.
For the wheel build, I added dependency for whell MKL and on conda a dependecy for the conda MKL  and on libtorch I copied the MKL binaries in libtorch.
In order to test this PR I have to use custom builder https://github.com/pytorch/builder/pull/1467

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite
Support for activation checkpoint on demand in custom function
Labels: module: checkpoint, triaged

### 🚀 The feature, motivation and pitch

The pack hook always runs after the forward pass of a custom function, so ``Rule 5. Stop recomputation as soon as we've recomputed the saved tensors we know we need.`` does not work if we only need to save the inputs for the backward pass.

### Alternatives

_No response_

### Additional context

_No response_
Jetson NX with torch 1.12.0 :cannot import name 'ProcessGroup' from 'torch.distributed'.
Labels: oncall: distributed

### 🐛 Describe the bug

`from torch.distributed import ProcessGroup`

error: cannot import name 'ProcessGroup' from 'torch.distributed'.

![52d9b82de946a97d99f3122d66babf8](https://github.com/pytorch/pytorch/assets/86214036/e8a389f1-a341-460f-aca8-a5b6827f67bc)


### Versions

Device: jetson NX,
jetpack:5.1.1
torch: 1.12.0

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Dynamo should feed optimized upstream graph's output to downstream graph for DDP
Labels: oncall: distributed

### 🐛 Describe the bug

When a model is wrapped with DDP, dynamo will introduce graph breaks. Say we have g1, g2 due to graph breaks. Right now dynamo pass the original g1's output to g2 for compiling. But ideally we should use the optimized g1's output to compile g2.

BTW, the normal graph break (those not caused by DDP) do pass optimzed upstream graph's output to downstream graph for compiling.

To repro, 
1. patch https://github.com/pytorch/pytorch/pull/102376 or wait for it to be merged
2. ```TORCHINDUCTOR_KEEP_OUTPUT_STRIDE=0 python test/inductor/test_layout_optim.py -k test_2conv_with_graph_break``` . BTW disable `TORCHINDUCTOR_KEEP_OUTPUT_STRIDE` is needed to reveal the problem.

### Versions

https://gist.github.com/shunting314/59cfc853a417952ebdc4941bfd3c41c3

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @ezyang @msaroufim @wconstab @ngimel @bdhirsh @anijain2305
test_functional_autograd_benchmark.py::TestFunctionalAutogradBenchmark::test_fast_tasks passes with all NaNs
Labels: module: autograd, triaged, module: functorch

The test seems to compare results with and without functorch, but everything is printed as NaNs:

Results for model resnet18 on task vjp: nans (var: nan)
Results for model resnet18 on task vjp using Functorch: nans (var: nan)

See https://github.com/pytorch/pytorch/actions/runs/5123048143/jobs/9213382637 for example logs.

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @Chillee @samdow @kshitij12345 @janeyx99
[RFC] Add third-party malloc library to improve pytorch memory performance on Windows
Labels: module: performance, module: windows, module: cpu, triaged, intel

### 🚀 The feature, motivation and pitch

This doc is requesting comments for add third-party malloc library to improve pytorch memory performance on Windows.

During debug the issue: https://github.com/pytorch/pytorch/issues/62387 , We figure out the major performance gap between Windows to Linux is that, Windows has bad memory allocation performance. 

I also write a simple malloc benchmark project, [bench_malloc](https://github.com/xuhancn/bench_malloc). Which can proof the third-party malloc (tc_malloc) can improve the memory alloction performance on Windows.
![image](https://github.com/pytorch/pytorch/assets/8433590/83d48342-dae4-4f9a-b485-cc92e0b2ff9e)

After that, I tried to evaluate same popular third-party malloc library and make a brief summay here:
![image](https://github.com/pytorch/pytorch/assets/8433590/f872c353-8f39-4345-92d0-86bcb9fc3dfd)
From the summary, We can only select two candidate libraries:
# Option 1: tc_malloc from gperftools.
1. It is the best performance one, and it improve performance from original 11.1s to 2.9s.
2. It need upstream same code changes to gperftools. https://github.com/gperftools/gperftools/pull/1396. Actually I made it, but still not get response so far. 

> Note: I found the gperftools repo is inactivate more than one year, latest commit is https://github.com/gperftools/gperftools/commit/bf8b714bf5075d0a6f2f28504b43095e2b1e11c5 on May 31, 2022 

# Option 2: mimalloc
1. It could improve performance from original 11.1s to 3.9s. It not as good as tc_malloc, but still a huge improvement.
2. It better compatibility to pytorch, and can integrate to pytorch directly.
3. I have a PR for this option: https://github.com/pytorch/pytorch/pull/102595 

### Alternatives

# Option 3: Implement a caching memory allocator for CPU in PyTorch.
1. It is none additional depends on third party library.
2. Need a lot of effort to develop and test. 
3. Its principle is similar to tc_malloc and mimalloc. Optimize existing third party library is better.

### Additional context

---
My proposal
1. I'm not sure whether the PR for gperftools can be accecpted. We can't always wait on option 1.
2. We can enable option 2 (mimalloc) to optimize pytorch Windows firstly.
4. Design build option to switch malloc library, such as system malloc, mimalloc, and (further tc_malloc).
5. Maybe We can select option 2, enable the mimalloc. And then optimize mimalloc for pytorch.

cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @ngimel
[MPS] Fix MPS sorting issue with strided view tensors
Labels: triaged, open source, release notes: mps, ciflow/mps

Fixes #101878

The issue with the `sort_stable_out_mps` function is that it does not handle strided views of the input tensor correctly. Specifically, if the input tensor is a strided view, the function will produce incorrect results.

This fix ensures that the input tensor is contiguous before passing it to the mps calculation. test case also added. let me know if this is a proper solution
[feature request] PyTorch support for sub-interpreters with PEP 684 accepted and release in Python 3.12
Labels: triaged, module: python frontend

### 🚀 The feature, motivation and pitch

It appears that sub-interpretor support for Python in PEP 684 https://discuss.python.org/t/pep-684-a-per-interpreter-gil/ is advancing and will be merged soon (https://github.com/python/cpython/blob/main/Misc/NEWS.d/3.12.0b1.rst, https://github.com/python/cpython/pull/104210, https://github.com/ericsnowcurrently/multi-core-python/wiki/0-The-Plan)

Any updates on eventual support of sub-interpreters in PyTorch? 

Also, a question would be of zero-copy in-RAM IPC data exchange: maybe via named pipes? Some mmaps? shm? 

I found older issues asking the same:
- https://github.com/pytorch/pytorch/issues/23490
- https://github.com/pytorch/pytorch/issues/10950

This might be important for working in hosted multi-threaded environments where every thread runs its own Python / PyTorch code. This is also related to the current torch.compile / export / mini-interpreters story: https://discuss.pytorch.org/t/torch-compiles-deployment-story-to-non-python-host-processes/180943. This scenario was explicitly supported with TorchScript / TorchJit, but seems not yet developed for torch.compile, but it's an important usecase too (similar to API's OnnxRuntime is currently offering)

cc @albanD @colesbury 

### Alternatives

_No response_

### Additional context

_No response_
pytorch java api documentation is not clear and does not cover example 
Labels: module: docs, triaged, oncall: mobile, oncall: java

### 📚 The doc issue

The pytorch java api doc is unclear and does not include examples which makes it hard to understand and makes the documentation incomplete

### Suggest a potential alternative/fix

- more example can be included 
- how to's in java
- more description

cc @svekars @carljparker
[Inductor] [CPU] hf_Longformer performance regression > 10% on 2023-05-28 nightly release
Labels: triaged, oncall: pt2, module: cpu inductor

### 🐛 Describe the bug

Performance regression:  **hf_Longformer**  [TorchInductor CPU Performance Dashboard](https://github.com/pytorch/pytorch/issues/93531#issuecomment-1567662304) on 2023-05-28 as bellow:

|  2023-05-28  |   |     |   |  2023-05-25  |   |     |   |  Result Comp  |     |   |
|  ----  | ----  |  ----  | ----  |  ----  | ----  |  ----  | ----  | ----  |  ----  | ----  |
| batch_size  | speedup | inductor  | eager | batch_size  | speedup | inductor  | eager | speedup ratio | eager ratio  | inductor ratio |
|1	|1.172147	 |0.082288565	|0.096454295	|1	|1.358556	 |0.070512074	|0.095794601	|0.86	|0.99	|0.86

SW information:

SW	| Nightly commit	| Main commit
-- | -- | --
Pytorch|[3dedcb3](https://github.com/pytorch/pytorch/commit/3dedcb3)|[c3ea8cc](https://github.com/pytorch/pytorch/commit/c3ea8cc)
Torchbench|/|[987bbec3](https://github.com/pytorch/benchmark/commit/987bbec3)
torchaudio|[894388d](https://github.com/pytorch/audio/commit/894388d)|[72d3fe0](https://github.com/pytorch/audio/commit/72d3fe0)
torchtext|[3cf1db6](https://github.com/pytorch/text/commit/3cf1db6)| [0dc07eb](https://github.com/pytorch/text/commit/0dc07eb)
torchvision|[50fb988](https://github.com/pytorch/vision/commit/50fb988)|[abc40ef](https://github.com/pytorch/vision/commit/abc40ef)
torchdata|[a754743](https://github.com/pytorch/data/commit/a754743)|[ba31745](https://github.com/pytorch/data/commit/ba31745)
dynamo_benchmarks|[174d01b](https://github.com/pytorch/pytorch/commit/174d01b)|/

graph:
2023_05_28
[graph.txt](https://github.com/pytorch/pytorch/files/11595496/graph.txt)

2023_05_25
[graph.txt](https://github.com/pytorch/pytorch/files/11595499/graph.txt)


### Versions

repro:

```bash
python -m torch.backends.xeon.run_cpu --node_id 0 benchmarks/dynamo/torchbench.py --performance --inference --float32 -dcpu -n50 --inductor  --no-skip --dashboard --only hf_Longformer  --cold_start_latency

```

cc @ezyang @msaroufim @wconstab @ngimel @bdhirsh @anijain2305
[Utils][tensorboard]Enhancement: Include 'max_outputs' parameter in torch.utils.tensorboard.summary's 'image' method
Labels: triaged, module: tensorboard

### 🚀 The feature, motivation and pitch

## Issue Description ##
The `image` function in `torch.utils.tensorboard.summary` does not have a `max_outputs` parameter, even though it is mentioned in the function's documentation. The `max_outputs` parameter is important for controlling the number of images included in the summary, but it is currently missing in the implementation.

## code in origin tensorboard ##
  ```def image(name, data, step=None, max_outputs=3, description=None)```
https://github.com/tensorflow/tensorboard/blob/fd75a84d30b85bdc80986da01fbf014f363ca3b7/tensorboard/plugins/image/summary_v2.py#L27

## code in torch.utils.tensorboard ##
 ``` def add_image(self, tag, img_tensor, global_step=None, walltime=None, dataformats="CHW")```
https://github.com/pytorch/pytorch/blob/0e72ada9bba2693e55fd7177ab4d2af826d7d15f/torch/utils/tensorboard/writer.py#LL563C4-L565C7

According to the documentation, the `max_outputs` parameter should determine the maximum number of images to be included in the summary. However, in the current implementation, there is no handling or usage of this parameter.
https://github.com/pytorch/pytorch/blob/0e72ada9bba2693e55fd7177ab4d2af826d7d15f/torch/utils/tensorboard/summary.py#L424

## Expected Behavior ##
I would expect the `image` function in `torch.utils.tensorboard.summary` to accept and utilize the `max_outputs` parameter, similar to the functionality provided by the native TensorBoard.

## Steps to Reproduce (if applicable) ##
N/A

## Environment Information (if applicable) ##
N/A

## Additional Context ##
I believe adding the `max_outputs` parameter to the `image` function in `torch.utils.tensorboard.summary` would enhance the functionality and compatibility of the module with the native TensorBoard implementation.


### Alternatives

_No response_

### Additional context

## Additional Context ##
I believe the addition of the 'max_outputs' parameter to the 'image' method in torch.utils.tensorboard.summary would greatly enhance the flexibility and compatibility of the module. Currently, users are unable to control the number of images included in the summary, which can lead to excessive or insufficient visualizations in certain scenarios.

By introducing the 'max_outputs' parameter, users would have the ability to specify the maximum number of images to be included in the summary, aligning with the behavior of the native TensorBoard implementation. This would provide consistency and allow for better customization and interpretation of the summary results.

Additionally, the 'max_outputs' parameter is already mentioned in the function's documentation, indicating that it was likely intended to be implemented but was unintentionally omitted.

I believe this enhancement would greatly benefit PyTorch users who rely on the tensorboard.summary module for visualizations and analysis of their models.
[REQUEST] - Update Multiprocessing best practices with CPU device
Labels: module: multiprocessing, triaged, docs-hackathon, topic: docs

### 📚 The doc issue

The current documentation on `torch.multiprocessing` is primarily focusing on CUDA device. User is highly possible to encounter oversubscription issue when following the instructions on a CPU device. On a CPU device, it is usually necessary to do explicit thread affinity binding else well as numa control, just as explained at [Tuning Guide for AI on the 4th Generation Intel® Xeon® Scalable Processors](https://www.intel.com/content/www/us/en/developer/articles/technical/tuning-guide-for-ai-on-the-4th-generation.html)

Also the example from [mnist_hogwild](https://github.com/pytorch/examples/tree/main/mnist_hogwild) is better to be updated at the same time, to give a more clear hands-on practice of how to do multiprocessing on CPU device.

### Suggest a potential alternative/fix

We aim to complete the document as part of PyTorch Docathon 2023. cc @VitalyFedyunin @jingxu10
torch.onnx.errors.CheckerError: The model does not have an ir_version set properly.
Labels: module: onnx, triaged

### 🐛 Describe the bug

torch=1**.13.1
onnx=1.14.0

[libprotobuf ERROR /opt/conda/conda-bld/pytorch_1670525552411/work/third_party/protobuf/src/google/protobuf/message_lite.cc:457] onnx_torch.ModelProto exceeded maximum protobuf size of 2GB: 8266614040
Traceback (most recent call last):
  File "/data03/zhangxiaolei/anaconda3/envs/LLM/lib/python3.8/site-packages/torch/onnx/utils.py", line 1652, in _export
    _C._check_onnx_proto(proto, full_check=True)
RuntimeError: The model does not have an ir_version set properly.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "model_construct_conformer.py", line 399, in <module>
    cli_main()
  File "model_construct_conformer.py", line 395, in cli_main
    main(args)
  File "model_construct_conformer.py", line 203, in main**
    return _main(cfg, sys.stdout)
  File "model_construct_conformer.py", line 299, in _main
    torch.onnx.export(
  File "/data03/anaconda3/envs/LLM/lib/python3.8/site-packages/torch/onnx/utils.py", line 504, in export
    _export(
  File "/data03/anaconda3/envs/LLM/lib/python3.8/site-packages/torch/onnx/utils.py", line 1654, in _export
    raise errors.CheckerError(e)
torch.onnx.errors.CheckerError: The model does not have an ir_version set properly.

### Versions

torch=1**.13.1
onnx=1.14.0
Batching rule for `aten::_scaled_dot_product_efficient_attention`
Labels: triaged, module: vmap

### 🚀 The feature, motivation and pitch

Hi, I am trying to take batched gradients of a vector output given by `_scaled_dot_product_efficient_attention` but saw the error 

```
/site-packages/optimum/bettertransformer/models/attention.py:56: UserWarning: There is a performance drop because we have not yet implemented the batching rul
e for aten::_scaled_dot_product_efficient_attention. Please file us an issue on GitHub so that we can prioritize its implementation. (Triggered internally at ../aten/src/ATen/functorch/BatchedFallback.cpp:82.)
  sdpa_result = torch.nn.functional.scaled_dot_product_attention
```

when running my code. Implementing this would really increase throughput in our application! I would also be happy to take a stab at implementing it, if there is a document describing what I need to do at a high level.

### Alternatives

_No response_

### Additional context

_No response_

cc @zou3519
RuntimeError using torch.nn.functional.pad when using MPS
Labels: triaged, module: mps

### 🐛 Describe the bug

I'm having an issue with `torch.nn.function.pad` when using _MPS backend_ on an Apple Silicon macOS. I encountered this issue when I try to run a project made by others ([ECCV2022-RIFE](https://github.com/megvii-research/ECCV2022-RIFE)). Moreover, I have found a Closed Issue #87277  that reports a very similar bug to mine. The only difference is that I'm having this problem when using pad with `mode='replicate'` (instead of the default `'constant'`).

Here below is a minimal example to demonstrate the problem:
```
import torch
y = torch.rand((1, 1, 3, 32, 32))
y_pad = torch.nn.functional.pad(y, (5, 5, 5, 5, 5, 5), mode='replicate')
```
When the above code is run on CPU by default, everything works well and no error is raised.

But when the device is set to MPS and run the same code (as shown below):
```
import torch
y = torch.rand((1, 1, 3, 32, 32), device='mps')
y_pad = torch.nn.functional.pad(y, (5, 5, 5, 5, 5, 5), mode='replicate')
```
Below's error is raised:
```
Traceback (most recent call last):
  File "/Users/gordon/PycharmProjects/ECCV2022-RIFE/DEBUG.py", line 3, in <module>
    y_pad = torch.nn.functional.pad(y, (5, 5, 5, 5, 5, 5), mode='replicate')
RuntimeError: Argument #8: Padding size should be less than the corresponding input dimension, but got: padding (5, 5) at dimension 2 of input 5
```
Although I think this is probably a bug, I'm sorry for causing inconvenience if it turns out to be a misjudgment.

### Versions

PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.4 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.11 | packaged by conda-forge | (main, May 10 2023, 19:01:19) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-13.4-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Max

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==2.0.1
[pip3] torchvision==0.15.2
[conda] No relevant packages

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
Add additional "sigmoid" approximation to GeLu activation?
Labels: module: nn, triaged, enhancement, needs research

### 🚀 The feature, motivation and pitch

Hi,

I was just ranomly browsing [tinygrad](https://github.com/geohot/tinygrad) and saw the "quick_gelu" function, noticing also that this implementation is not a part of pytorch it self.

Pytorch only seems to have the `tanh` - approximation, so this is a feature request to add the second approximation also mentioned in the paper [Gaussian Error Linear Units (GELUs)](https://arxiv.org/abs/1606.08415v4)

I am not a AI expert, but this was just a simple thing I noticed and could gladly implement :)

### Alternatives

* Add a `sigmoid` approximation also, `x*sigmoid(1.702 * x)`
* Do nothing :)

### Additional context

Basically this commit:

https://github.com/pytorch/pytorch/commit/f7a1ca8476b4788265f7b4e2d9d088937dd1f033


cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
Add support ONNX Opset 19
Labels: module: onnx, triaged

### 🚀 The feature, motivation and pitch

Onnx v1.14.0 has just released opset 19 . Please add support for exporting DeformConv and AdaptiveAveragePool2d to opset 19. 

### Alternatives

_No response_

### Additional context

_No response_
Add default device function interface for device-aware apis
Labels: triaged, module: dispatch

_Originally posted by @mikaylagawarecki in https://github.com/pytorch/pytorch/pull/102155#pullrequestreview-1446953358_
     Hi @mikaylagawarecki , Some of the current community functions include c++ and pyhton, which have a default device of cuda. 
e.g.` _pin_memory` has a default dispatchkey of KCUDA, 
`DispatchKeySet _dk = c10::DispatchKeySet(c10::computeDispatchKey(c10::nullopt, self.layout(), device.value_or(at::kCUDA)));`
and` fork_rng `on the python side has a default device_type of` cuda`, 
`def fork_rng(devices=None, enabled=True, _caller="fork_rng", _devices_kw="devices", device_type="cuda")`etc. 
For third-party hardware devices, such as privateuse1, we need to set additional device parameters, `fork_rng(device_type='xxx')`,`pin_memory('xxx')`, when using the above functions. Some other functions have the same problem, the default device is cuda, which leads to incompatible api usage.
    We want to set a default device function in the community. On third-party devices, by setting the default device interface, the above function can be called directly, such as` fork_rng()`, without the need to set the device_type additionally. this will improve the compatibility of the community code on different devices.
_view_func but without keeping original view tensor alive
Labels: module: autograd, triaged, needs design, module: viewing and reshaping

### 🐛 Describe the bug

Tensor._view_func lets you reapply a view tensor's view func to a different tensor. That's good, but often what I want is to hold onto this view function **while freeing the original tensor** (e.g., if I'm using this to reconstruct a tensor later from a new base.) There's no way to do this right now.

### Versions

main

cc @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
[Composable] Unified way to check if modules are managed by composable API
Labels: oncall: distributed, release notes: distributed (composable)

### 🚀 The feature, motivation and pitch

Today to check if a top-level module is managed by a composable API (replicate, fully_shard, trec_shard), the `_get_registry(module)` API can be used. However, this is limited when considering the case of nested wrapping. i.e. if I have the following:

```
class UnitModule:
    self.lin1 = nn.Linear(...)

m = UnitModule()
fully_shard(m)
```

then I can check that `m` is managed by FSDP composable API, but there's no easy way for me to tell that `self.lin1` is also sharded. 

The generalization that if `_get_registry(module)` returns True then all of module's children are also managed by that composable API is not true in practice. For example, I could've called `replicate()` or `trec_shard` on `self.lin1` and it would be managed by another composable API. Or, I could've specified `fully_shard` to ignore `self.lin1` as well. 

The following API could work, but seems a bit cumbersome:

```
def get_fsdp_managed_modules(root_module):
    fsdp_mods = []
    fsdp_roots = [m for m in root_module.modules() if 'fully_shard' in _get_registry(m)]
    for m in fsdp_roots:
        if 'replicate' not in _get_registry(m) and 'trec_shard' not in _get_registry(m) and not _is_fsdp_ignored(m):
            fsdp_mods.append(m)
    return fsdp_mods
```

In general there are a couple pieces of information that we probably want:
- whether a particular module is the root module of a composable (fsdp, sharded, replicate) subtree
- whether a module is not (necessarily) the root, but still managed by a composable API
- whether a module is explicitly ignored by any of the composable APIs.

Would love suggestions / feedback on this issue.

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Unexpected Behavior when using torch.isclose()
Labels: triaged, module: NaNs and Infs, module: testing

### 🐛 Describe the bug

When using `self.assertEqual` within test_transformers, I was observing some weird behavior for certain test cases.  I tracked this down to `torch.isclose()` and have a minimal repro:

``` Python
import torch

def main():
    
    a = torch.tensor([0.0001], dtype=torch.float16, device="cuda")
    b = torch.tensor([0.0], dtype = torch.float16, device="cuda")
    rtol = (torch.abs(a - b) / torch.min(abs(a), abs(b))).item()
    print(rtol)
    # This will be torch.inf due to the division by zero
    print(torch.isclose(a, b, atol=1, rtol=rtol))
    # >> False
    

if __name__ == "__main__":
    main()
```

This test will fail regradless of what you set the atol. I think this is counter intuitive to what one would expect:
From the the docs: https://github.com/pytorch/pytorch/blob/0ed22fce978d7aae4a62ee86d9f85fa7314b1dd5/aten/src/ATen/native/TensorCompare.cpp#L287
``` C++
// (1) A is equal to B, with NaNs comparing equal when equal_nan is true.
// (2) The error abs(A - B) is finite and less than the max error
//      (atol + abs(rtol * B)).
```
I think the problem is that this 
In this case (A-B) is finite and `atol + abs(rtol *B) = 1 + abs(inf*0) `
The second term becomes NaN:
```
>>> import torch
>>> a = torch.tensor([torch.inf])
>>> b = torch.tensor([0.0])
>>> a
tensor([inf])
>>> b
tensor([0.])
>>> a * b
tensor([nan])
```

This might be truly a case of undefined behavior and we make no assumptions but I would think that someone using isclose with that atol thinks they are equal. I could be wrong. My argument for this would be add 1 to everything and shift the problem to not be centered around 0 and they results would be as expected.


### Versions

Nightly
Hooks not working in version 2.0.1+cu118
Labels: module: nn, triaged

### 🐛 Forward Hooks are not getting called

When writing forward hooks on a layer of a model, I realized they are not being called at all.

```python
# inside __init__ function
for name, module in self.model.named_modules():
    if attention_layer_name in name:
    print(f"Module hook registered for: {name}")
    module.register_forward_hook(lambda m, i, o: attentions.append(o))
```
Expected Behavior: When I forward an example through the model, the hooks should be called after specifically hooked layers are done forwarding the weights

I did try it on torch version 1.13.1, and it seems to work fine there.

### Versions

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==2.0.1+cu118
[pip3] torchaudio==2.0.2+cu118
[pip3] torchvision==0.15.2+cu118
[pip3] triton==2.0.0
[conda] Could not collect

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
"Examples" in "batch_isend_irecv" should be modified to get the correct results
Labels: oncall: distributed

https://github.com/pytorch/pytorch/blob/6c9b94dcda696984552769da6cf578ccca0cfbdd/torch/distributed/distributed_c10d.py#LL1709C13-L1709C13

    Examples:
        >>> # xdoctest: +SKIP("no rank")
        >>> send_tensor = torch.arange(2) + 2 * rank
        >>> recv_tensor = torch.randn(2)
        >>> send_op = dist.P2POp(dist.isend, send_tensor, (rank + 1)%world_size)
        >>> recv_op = dist.P2POp(dist.irecv, recv_tensor, (rank - 1 + world_size)%world_size)
        >>> reqs = batch_isend_irecv([send_op, recv_op])
        >>> for req in reqs:
        >>>     req.wait()
        >>> recv_tensor
Here "send_tensor" and "recv_tensor" are in different dtypes, which would make it fail to get the correct results on gpus.
For example, one might need to add explict dtype declaration
        >>> send_tensor = torch.arange(2, dtype=torch.float) + 2 * rank
        >>> recv_tensor = torch.randn(2, dytpe=torch.float)

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
[Dynamo] Better graph-break message for unsupported ctx managers
Labels: good first issue, triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

Repro:
```
import torch
import torch._dynamo
import logging
import warnings

torch._logging.set_logs(dynamo=logging.DEBUG)

class MyModule(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(10, 10)

    def forward(self, x):
        with warnings.catch_warnings():
            warnings.simplefilter('ignore')
            return self.layer(x)


x = torch.randn(10, 10)

m = MyModule()
print(m(x))

opt_m = torch.compile(backend="eager", fullgraph=True)(m)
print(opt_m(x))
```

Error:
```
torch._dynamo.exc.Unsupported: setattr(UserDefinedObjectVariable) <slot wrapper '__setattr__' of 'object' objects>
```

The root cause is we can't handle the context manager, so we fall back to eager. Need to figure out a better way to handle unsupported context managers or print better logs.

### Versions

N/A

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @aakhundov @ngimel
There is a memory leak in torch.load
Labels: module: memory usage, module: serialization, triaged

### 🐛 Describe the bug

test code:
```python
import torch
import gc

@profile
def my_func():
    ckpt = torch.load("best.ckpt", map_location=torch.device('cpu'))
    del ckpt
    gc.collect()
    gc.collect()


if __name__ == '__main__':
    my_func()
```
![image](https://github.com/pytorch/pytorch/assets/43675899/0c4a45d9-3085-4717-8991-72a478d3d8fc)

There is a memory leak in torch.load and cannot be freed


### Versions

Collecting environment information...
PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.6 (x86_64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.102)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.6 (default, Aug  5 2022, 15:21:02)  [Clang 14.0.0 (clang-1400.0.29.102)] (64-bit runtime)
Python platform: macOS-12.6-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.3
[pip3] torch==2.0.1
[pip3] torchaudio==2.0.2
[pip3] torchmetrics==0.9.2
[conda] Could not collect


cc @mruberry @mikaylagawarecki
Re-enable Low Memory Dropout
Labels: triaged, oncall: pt2, module: inductor

### 🐛 Describe the bug

https://github.com/pytorch/pytorch/pull/100064 caused 3% perf drop and 4% memory drop in HF training because of disabled low memory dropout

### Versions

master

cc @ezyang @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10
[Dynamo]Outdated logging setting
Labels: module: docs, triaged, oncall: pt2

### 📚 The doc issue

In https://pytorch.org/docs/stable/dynamo/troubleshooting.html, it says using 
 torch._dynamo.config.log_level = logging.INFO 
to enable debug/info logging, which no longer works.








### Suggest a potential alternative/fix

Per https://github.com/pytorch/pytorch/blob/0833f475ce7e42b1dd11af577f276b804fc2b158/torch/_dynamo/config.py#L12
we should use  torch._logging.set_logs(dynamo=<level)

cc @svekars @carljparker @ezyang @msaroufim @wconstab @ngimel @bdhirsh @anijain2305
after add /path_to_libtorch/libtorch/lib to LD_LIBRARY_PATH, I can't import torch_scatter.
Labels: module: build, module: cpp, module: abi, triaged

### 🐛 Describe the bug

To compile and use libtorch, I add libtorch/lib to $LD_LIBRARY_PATH. However, it causes error if I import torch_scatter. The following is the error:
>>> import torch_scatter
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/torch_scatter/__init__.py", line 16, in <module>
    torch.ops.load_library(spec.origin)
  File "/usr/lib/python3.8/site-packages/torch/_ops.py", line 643, in load_library
    ctypes.CDLL(path)
  File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /usr/lib/python3.8/site-packages/torch_scatter/_version_cuda.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSs

It looks like there is a conflict.
Once I remove the "/path_to_libtorch/libtorch/lib" from LD_LIBRARY_PATH, the problem will disappear. However, I will not be able to use libtorch anymore.

### Versions

python 3.8
torch == 2.0.0+cu117
torchvision==0.15.1+cu117
torchaudio==2.0.1+cu117

cc @malfet @seemethere @jbschlosser
ExponentialLR unexpectedly calls `step()` when init argument `last_epoch` is larger than -1
Labels: module: optimizer, triaged, actionable, module: LrScheduler

### 🐛 Describe the bug

Currently, the init function of ``torch.optim.lr_scheduler._LRSchedule`` will call ``self.step()`` once. This causes a mismatch between the learning rate used by the optimizer and the closed_form_lr of ExponentialLR, when init argument `last_epoch` is larger than -1.

```python
import torch

model = torch.nn.Linear(3, 3)
optim = torch.optim.AdamW(model.parameters())
sched = torch.optim.lr_scheduler.ExponentialLR(optim, gamma=0.999)

optim.step()
sched.step()
print("Optim & sched:")
print(optim.state_dict())
print(optim.param_groups[0]["lr"])
print(sched.state_dict())
print(sched._get_closed_form_lr())
print("")

# As if we are restoring from a checkpoint
optim2 = torch.optim.AdamW(model.parameters())
optim2.load_state_dict(optim.state_dict())
# Init scheduler with last_epoch=0
sched2 = torch.optim.lr_scheduler.ExponentialLR(optim2, gamma=0.999, last_epoch=0)

print("Optim2 & sched2:")
print(optim2.state_dict())
print(optim2.param_groups[0]["lr"])
print(sched2.state_dict())
print(sched2._get_closed_form_lr())
print("")

```

As the result shows, ``optim2`` has lr 0.000998001, but the closed form lr of ``sched2`` is 0.000999. This behavior causes confusion and inconsistency when one resumes training from a checkpoint.


### Versions

Collecting environment information...
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

cc @vincentqb @jbschlosser @albanD @janeyx99 @crcrpar
[ONNX] test_op_consistency.py doesn't support constant inputs
Labels: module: onnx, triaged, onnx-triaged

### 🐛 Describe the bug

The current op test design in TorchScript exporter assumes all inputs/args are model inputs, which JIT tracer would trace them all even if they are not tensors.

```python
class SingleOpModel(torch.nn.Module):
    """Test model to wrap around a single op for export."""

    def __init__(self, op, kwargs):
        super().__init__()
        self.operator = op
        self.kwargs = kwargs

    def forward(self, *args):
        return self.operator(*args, **self.kwargs)
```

This results in issue in ops like `scatter_reduce` which has `dim` in arguments. Usually, it is used like:
```python
input.scatter_reduce(0, index, x, reduce='amax')
```

So the dim wouldb't be traced. However, if it is passed in as model input, it will be traced, and inside symbolic function. We will not get prim::Constant of dim.

cc @justinchuby 

### Versions

Nightly
copy_'s functionalized operator keeps copied into tensor live
Labels: triaged, module: functionalization, oncall: pt2

### 🐛 Describe the bug

It's possible decomps fix this problem, but without decomps the graph is not as good as it could be:

```
import torch
from torch._functorch.aot_autograd import aot_export_joint_simple
    
def f(x, weight):
    def cb(grad_out):
        y = grad_out.sin()  # stand-in for collective result
        weight.data.copy_(y)
        # Better would be:
        #   weight.data = y
        return grad_out
    r = x @ weight
    r.register_hook(cb) 
    return (r,)

gm = aot_export_joint_simple(
    f,
    [torch.randn(2, 3, requires_grad=True), torch.randn(3, 4)],
    trace_joint=True,
)
gm.print_readable()
```

gives

```
class joint_helper(torch.nn.Module):
    def forward(self, primals, tangents):
        primals_1: f32[2, 3], primals_2: f32[3, 4], tangents_1: f32[2, 4], = fx_pytree.tree_flatten_spec([primals, tangents], self._in_spec)
        # No stacktrace found for following nodes
        mm: f32[2, 4] = torch.ops.aten.mm.default(primals_1, primals_2);  primals_1 = None
        sin: f32[2, 4] = torch.ops.aten.sin.default(tangents_1)
        copy: f32[3, 4] = torch.ops.aten.copy.default(primals_2, sin);  primals_2 = sin = None
        t_1: f32[4, 3] = torch.ops.aten.t.default(copy);  copy = None
        mm_1: f32[2, 3] = torch.ops.aten.mm.default(tangents_1, t_1);  tangents_1 = t_1 = None
        return pytree.tree_unflatten([mm, mm_1, None], self._out_spec)
```

I don't want the dependence on primals_2.

This example modeled based off of FSDP.

cc @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 

### Versions

main
aot_export_joint_simple on plain callable (not graph module) doesn't attach stack traces
Labels: triaged, better-engineering, oncall: pt2, module: aotdispatch

### 🐛 Describe the bug

Repro example:

```
import torch
from torch._functorch.aot_autograd import aot_export_joint_simple
    
def f(x, weight):
    def cb(grad_out):
        y = grad_out.sin()  # stand-in for collective result
        weight.data.copy_(y)
        # Better would be:
        #   weight.data = y
        return grad_out
    r = x @ weight
    r.register_hook(cb) 
    return (r,)

gm = aot_export_joint_simple(
    f,
    [torch.randn(2, 3, requires_grad=True), torch.randn(3, 4)],
    trace_joint=True,
)
gm.print_readable()
```

gives

```
        # No stacktrace found for following nodes
```

It would be nice to get the native stack traces in this case.

### Versions

main

cc @msaroufim @wconstab @ngimel @bdhirsh @anijain2305
scipy.ndimage.find_objects
Labels: feature, module: nn, triaged, needs research

### 🚀 The feature, motivation and pitch

This function a basic building block of any biomedical image analysis application.
It gives a list of tuple of slices of coordinates of labelled objects/cells within a mask image of dtype Uint16 or Uint8, assuming the image background is 0, and the labelled objects go from 1, 2, ..., max_label.

I was wondering it is possible to implement it in torch C++ using a simple TensorIterator.

Basically the simplest case would be it takes a 2D tensor of size (H, W) as input and
outputs a tensor of slices of size (N, 2) where N is the number of objects, and each row
is [slice(start,end,step), slice(start,end,step)]. 

### Alternatives

The implementation in C numpy can be found here:
https://github.com/scipy/scipy/blob/v1.10.1/scipy/ndimage/src/ni_measure.c

which uses Iterators defined here:
https://github.com/scipy/scipy/blob/v1.10.1/scipy/ndimage/src/ni_support.h

### Additional context

Can it also be extended to allow extract objects from a tensor of dimension (B, C, W, H) where B is the batch size, C the number of
channels and W is the width and H is the height.

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
Extend fake fast path to more situations
Labels: triaged, module: fakeTensor

### 🐛 Describe the bug

https://github.com/pytorch/pytorch/blob/d316a2dd5c8c89c1e0a18a0cce595a81dd9e52ab/torch/_subclasses/fake_tensor.py#L679-L824

Only applied to four ops right now. Apply it to more.

### Versions

main
torch/distributed/_spmd/api.py should aot_module_export instead of make_fx directly
Labels: oncall: distributed

### 🐛 Describe the bug

Self explanatory

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @bdhirsh @wanchaol @gmagogsfm 

### Versions

main
DISABLED test_build_tuple_unpack_dynamic_shapes (torch._dynamo.testing.DynamicShapesMiscTests)
Labels: triaged, module: flaky-tests, skipped, module: dynamo

Platforms: linux, mac

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_build_tuple_unpack_dynamic_shapes&suite=DynamicShapesMiscTests) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/undefined).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 2 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_build_tuple_unpack_dynamic_shapes`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `dynamo/test_dynamic_shapes.py` or `dynamo/test_dynamic_shapes.py`

cc @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx