gopikrsmscs/torch-issues huggingface dataset:
Infinite recursion when tracing through lift_fresh_copy OP in Adam optimizer
Labels: module: optimizer, triaged, oncall: pt2, module: fakeTensor

### 🐛 Describe the bug

Hello. I prepared code to test custom backends for torch.compile functionality. One of the things my backend is doing is using fx.Interpreter to go through all operations (via fake tensors) and propagate device info. Then I use this information to cluster some of the operations via CapabilityBasedPartitioner. For purpose of this bug I have simplified the code, it is available here: https://gist.github.com/kbadz-pl/53d6cf36ac467039a0fa3170e7d5157b

Issue is that when fx.Interpreter tries to run_node over **lift_fresh_copy** operation, it goes into infinite recursion and crashes.
![image](https://user-images.githubusercontent.com/122799060/212691735-68ec7688-4733-4b76-ab0b-030bac53c9f8.png)

I did not see such issue with other OPs. Issue does not occur for example for SGD (it does not contain such OP) or by disabling fake tensors. You can test it by changing to SGD in line 106 or disabling fake tensors in line 15 of my file.

### Versions

PyTorch version: 2.0.0.dev20230116+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Nov 14 2022, 12:59:47)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-126-generic-x86_64-with-glibc2.29
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] torch==2.0.0.dev20230116+cpu
[pip3] torchaudio==2.0.0.dev20230115+cpu
[pip3] torchvision==0.15.0.dev20230115+cpu
[conda] Could not collect


cc @vincentqb @jbschlosser @albanD @janeyx99 @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
Add torch::jit::ScriptModule to the C++ API
Labels: oncall: jit

### 🚀 The feature, motivation and pitch

I'd like to implement a model in C++ with some submodules defined in C++ being subclasses of `torch::nn::Module` and a submodule being a `torch::jit::Module`. This is a common use case when your design your own model based around some well known backbone for which a torchscripted version is available.

I'm not sure about the best way to do this, but I believe that I must build my own wrapper subclassing `torch::nn::Module` around the JIT module. Such wrapper would register the parameters of the JIT module, and build recursively wrappers around its submodules. The wrapper would also need to overide `train()`, `eval()`, `to()` ... and delegate them to the JIT module. A bit ugly  but it should work.

The thing is, I'm not really developing in C++, but in Java (or whatever language using a binding to the C++ API). In that case, writing a C++ subclass of `torch::nn::Module` that wraps the `torch::jit::Module` is not an available option.

So is it possible, either to make `torch::jit::Module` inherit from `torch:nn::Module` (I guess not) or provide a ready to be used C++ `torch::jit::ScriptModule` wrapper class, just like the Python counterpart ?


### Alternatives

_No response_

### Additional context

_No response_

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Hijacked package names from nightly repository
Labels: oncall: binaries, security

### 🐛 Describe the bug

In response to this
https://pytorch.org/blog/compromised-nightly-dependency/
I checked if there are other package names in the pytorch nightly package index. I found two package names that I could register on pypi (torchaudio-nightly, pytorch-csprng).

I reported this to facebook's bugbounty program (as your security policy says), however it seems facebook no longer is responsible here (see also #91570). Facebook's security team came to the conclusion that the issue is not severe.

Nevertheless I now have registered these package names on pypi and I am wondering what to do with them. I would prefer to transmit ownership of the account to the pytorch team so you can decide what to do with them and if you want to keep them registered to block the names.

### Versions

irrelevant/nightly

cc @ezyang @seemethere @malfet
Improve make_fx tracing speed
Labels: module: performance, triaged, module: ProxyTensor

### 🚀 The feature, motivation and pitch

We observed that tracing a model using make_fx is sometimes much slower than actually running a model, and the observation has been consistent across different model sizes. For our big model sometimes we see a 30s+ tracing time.

While we understand tracing is usually only done once throughout the training process, the slow tracing speed has a direct impact on startup time, which leads to a very unpleasant debug-test experience. 

```
import torch
from functorch import make_fx
import time

def main():
    model = ...
    input = ...
    start = time.perf_counter()
    model(input)
    end = time.perf_counter()
    print(f"time to execute {end - start}")
    start = time.perf_counter()
    traced = make_fx(model)(input)
    end = time.perf_counter()
    print(f"time to trace {end - start}")
```

```
time to execute 0.04433171300000005
time to trace 0.47819954299999967
```

### Alternatives

_No response_

### Additional context

_No response_

cc @ngimel @ezyang @SherlockNoMad @soumith @EikanWang @jgong5 @wenzhe-nrv
false INTERNAL ASSERT FAILED at "../c10/cuda/CUDAGraphsC10Utils.h":73, please report a bug to PyTorch. Unknown CUDA graph CaptureStatus32680
Labels: triaged, module: cuda graphs

### 🐛 Describe the bug

hello i used this code in colab:
seed = 72
torch.manual_seed(seed)         # Insert any integer
torch.cuda.manual_seed(seed)    # Insert any integer

i got a Error says:
RuntimeError: false INTERNAL ASSERT FAILED at "../c10/cuda/CUDAGraphsC10Utils.h":73, please report a bug to PyTorch. Unknown CUDA graph CaptureStatus32680


### Versions

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

cc @mcarilli @ezyang
RuntimeError: derivative for aten::mps_linear_backward is not implemented
Labels: module: autograd, triaged, actionable, module: mps

### 🐛 Describe the bug

when I calculate the second  derivative of the model, using my MacBook Air with M1, the bug is shown as below
<img width="1017" alt="image" src="https://user-images.githubusercontent.com/87766834/212471926-f4df0537-ac7e-42dc-b80e-a9d063314a14.png">



Another question: what amazes me most is that when I using the other computer with Windows PyTorch on cuda, the codes can run successfully...

I just want to know how to solve the problem on my MacBook shown in the picture above.

### Versions

     15 u_pred = model(f_tmp)
     16 du_df=torch.autograd.grad(outputs =u_pred, inputs = f_tmp,
     17                             grad_outputs = torch.ones_like(u_pred).to(device),
     18                             create_graph = True,
     19                             allow_unused=True
     20                             )[0]
---> 21 ddu_ddf=torch.autograd.grad(outputs =du_df, inputs = f_tmp,
     22                             grad_outputs = torch.ones_like(du_df).to(device),
     23                             create_graph = True,
     24                             allow_unused=True
     25                             )[0]

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @kulinseth @malfet @DenisVieriu97 @razarmehr @abhudev
Triton Autotuning Cache-Clearing Adds 256MB Memory Overhead
Labels: triaged, bug, oncall: pt2

### 🐛 Describe the bug

In inductor autotuning, we invoke triton's do_bench which internally allocates a [256mb tensor](https://github.com/openai/triton/blob/259f4c5f7d4ecb8c64c519fe669318a15d6f75f2/python/triton/testing.py#L162) that is used to clear the cache.

From a practical standpoint, 256 MB is insignificant on a 40GB machine, however for a 12GB RTX 3080 this is a 2% overhead. Additionally, it's a non-linear factor, which throws off memory usage stats for small batch sizes. It has a big effect on the torchbench performance dashboard because low batch sizes are being used. E.g. `resnet18` is at .69 compression with autotuning, and ~1 without.

We could pass a handle to other allocated memory that is not being used in the autotuned kernel to avoid the additional allocation.

### Error logs

_No response_

### Minified repro

_No response_

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
test_fx_passes generate bad test names
Labels: module: tests, oncall: fx

I am not sure how they get generated but some tests are named things like `test_partitioner_fn_<function TestPartitionFunctions_forward13 at 0x7f35079131a0>_expected_partition_[['add_2', 'add_1', 'add']]_bookend_non_compute_pass_False`.
I am amazed that this doesn't break anything very loudly. But this is definitely not going to work well with all the infra around detecting flaky tests and disabling broken things (the memory address there...).

These tests should be updated to have sane names!

cc @mruberry @ezyang @SherlockNoMad @EikanWang @jgong5 @wenzhe-nrv
"multi device" tests get skipped in standard CI
Labels: module: ci, module: tests, triaged

## Concern
The general concern is that people think they're adding tests to CI but these tests are getting skipped, which is bamboozling. Specifically, I wonder how many tests there are across our codebase that require multiple GPUs but are NOT on the multigpu shard, meaning they are always skipped (our runners only have 1 GPU). 

This stemmed from my noticing the `@deviceCountAtLeast(2)` decorator while reviewing a PR, and then a brief search revealed that there are 18 instances of this decorator throughout our codebase. There may other ways people guard/skip tests based on # devices (such as with if statements/with TEST_MULTIGPU). 

To add another layer to this onion, even if people knew about the multigpu shard and how to add a test to it, the multigpu config runs on a periodic cadence, so people may not realize that these tests they added never even ran on the CI of their PR.

## Potential Next Steps
1. Document in deviceCountAtLeast that these do NOT run in most CI, and that tests with this decorator should be added to https://github.com/pytorch/pytorch/blob/master/.jenkins/pytorch/multigpu-test.sh
2. Run existing unrun tests like `test_clip_grad_norm_multi_device` on the multigpu config
3. I understand that the multigpu shard was historically hardware-constrained. @pytorch/pytorch-dev-infra Is there a near future where multigpu can run on trunk instead of periodic?

cc @seemethere @malfet @pytorch/pytorch-dev-infra @mruberry
[Bug][Dataloader] unable to mmap 2048 bytes from file <filename not specified>: Cannot allocate memory (12)
Labels: module: multiprocessing, module: dataloader, triaged

### 🐛 Describe the bug

Hi,

Here is the results i observed when i was running my workload with PyTorch 1.13 on Ubuntu 20.04 to train RN50 with imageNet:
When i run 25 epochs, the error is thrown as below:

```
Epoch: [25][3096/5005]	Time  0.551 ( 0.548)	Data  0.000 ( 0.001)	Loss 1.9055e+00 (2.1268e+00)	Acc@1  57.81 ( 51.79)	Acc@5  79.69 ( 76.05)
Epoch: [25][3097/5005]	Time  0.547 ( 0.548)	Data  0.000 ( 0.001)	Loss 1.9593e+00 (2.1268e+00)	Acc@1  55.08 ( 51.80)	Acc@5  81.25 ( 76.06)
Exception in thread Thread-52:
Traceback (most recent call last):
  File "/home/gta/miniconda3/envs/zejun/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/home/gta/miniconda3/envs/zejun/lib/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/home/gta/miniconda3/envs/zejun/lib/python3.9/site-packages/torch/utils/data/_utils/pin_memory.py", line 52, in _pin_memory_loop
    do_one_step()
  File "/home/gta/miniconda3/envs/zejun/lib/python3.9/site-packages/torch/utils/data/_utils/pin_memory.py", line 29, in do_one_step
    r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL)
  File "/home/gta/miniconda3/envs/zejun/lib/python3.9/multiprocessing/queues.py", line 122, in get
    return _ForkingPickler.loads(res)
  File "/home/gta/miniconda3/envs/zejun/lib/python3.9/site-packages/torch/multiprocessing/reductions.py", line 294, in rebuild_storage_fd
    storage = cls._new_shared_fd(fd, size)
RuntimeError: falseINTERNAL ASSERT FAILED at "/home/gta/zejun/pytorch/aten/src/ATen/MapAllocator.cpp":323, please report a bug to PyTorch. unable to mmap 2048 bytes from file <filename not specified>: Cannot allocate memory (12)
```

This Runtime Error is thrown from torch dataloader. I found the community has had this issue: https://discuss.pytorch.org/t/pytorch-cannot-allocate-memory/134754.

I set the dataloader workers to be 0 and the error is missing, but the training time increases too much when using the single process to fetch and decode the dataset. I also check the host memory usage and no oom happen. The fd limitation of one process is also not exceeded. Thus we wonder if the issue can be fixed, otherwise it may block the model training with PyTorch.

### Versions

PyTorch 1.13
Python 3.9
Torchvision 1.14
Running BS: 256
Model: ResNet50
Dataset: ImageNet

Ubuntu 20.04
Host Mem 128G
CPU: Intel Xeon Gold 6342

cc @SsnL @VitalyFedyunin @ejguan @NivekT
Torchrun seems to have problem with virtual environment
Labels: oncall: distributed, triaged, oncall: r2p

## Issue description

Use torchrun (inside a virtual environment) to launch a Python script. The script can not import modules installed in that virtual environment. Changing to use torch.distributed.launch to launch works well but that method has been depreciated.

## Code example

* Create a venv, activate the venv
* Install a module, say "pip install datasets"
* Write a python script (test.py) to load datasets
```
# This is test.py
import datasets
```
* Do the following
```
torchrun \
    --nnodes=1 \
    --nproc_per_node=8 \
    --max_restarts=0 \
    --rdzv_id=123456 \
    --rdzv_backend=c10d \
    --rdzv_endpoint=localhost \
    test.py
```
You will get "ModuleNotFoundError: No module named 'datasets'"
* Changing from 'torchrun' to 'python3 -m torch.distributed.launch', then it all works.
```
python3 -m torch.distributed.launch \
    --standalone \
    --nnodes=1 \
    --nproc_per_node=8 \
    test.py
```

## System Info

Python 3.10.6
PyTorch 1.12.0
Linux (CentOS 7)

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
DISABLED test_cuda_variable_sharing (__main__.TestMultiprocessing)
Labels: triaged, module: flaky-tests, skipped, module: unknown

Platforms: rocm

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_cuda_variable_sharing&suite=TestMultiprocessing) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/10617465804).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 2 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT BE ALARMED IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_cuda_variable_sharing`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.
DISABLED test_cdist_large_batch (__main__.TestMPS)
Labels: triaged, module: flaky-tests, skipped, module: mps

Platforms: mac, macos

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/failure/test_cdist_large_batch) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/10604384288).

Over the past 72 hours, it has flakily failed in 2 workflow(s).

**Debugging instructions (after clicking on the recent samples link):**
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Grep for `test_cdist_large_batch`


cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
compilig MultiHeadAttention
Labels: oncall: jit

### 🐛 Describe the bug

If `vdim` or `kdim` is different from `embed_model` the `nn.MultiheadAttention` module throws error when calling `torch.jit.scritp` on it.

```python
import torch

m = torch.nn.MultiheadAttention(10, 10, vdim=5)

torch.jit.script(m)
```

### Versions

Collecting environment information...
PyTorch version: 1.13.0+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.5 (default, Sep  4 2020, 07:30:14)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 10.1.243
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Quadro RTX 6000
Nvidia driver version: 510.47.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.19.2
[pip3] numpydoc==1.1.0
[pip3] open-clip-torch==2.8.2
[pip3] torch==1.13.0
[pip3] torchaudio==0.9.0a0+33b2469
[pip3] torchvision==0.10.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] mkl                       2020.2                      256  
[conda] mkl-service               2.3.0            py38he904b0f_0  
[conda] mkl_fft                   1.2.0            py38h23d657b_0  
[conda] mkl_random                1.1.1            py38h0573a6f_0  
[conda] numpy                     1.19.2           py38h54aff64_0  
[conda] numpy-base                1.19.2           py38hfa32c7d_0  
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1  
[conda] open-clip-torch           2.8.2                     dev_0    <develop>
[conda] torch                     1.13.0                   pypi_0    pypi
[conda] torchaudio                0.9.0                      py38    pytorch
[conda] torchvision               0.10.0               py38_cu102    pytorch

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Implement forward AD with grid_sampler_2d 
Labels: triaged, module: forward ad

### 🚀 The feature, motivation and pitch

I have a use case where I need to calculate the singular vectors of the jacobian of a function which generates and images and sends it through the vgg network used in stylegan2-ada for the [PPL implementation](https://github.com/NVlabs/stylegan2-ada-pytorch/blob/main/metrics/perceptual_path_length.py) 

I use `torch.autograd.functional.jacobian`and would like to use  `vectorize=True`,  `strategy='forward-mode` but run into the following

```
RuntimeError: Trying to use forward AD with grid_sampler_2d that does not support it because it has not been implemented yet.
Please file an issue to PyTorch at https://github.com/pytorch/pytorch/issues/new?template=feature-request.yml so that we can prioritize its implementation.
Note that forward AD support for some operators require PyTorch to be built with TorchScript and for JIT to be enabled. If the environment var PYTORCH_JIT=0 is set or if the library is not built with TorchScript, some operators may no longer be used with forward AD.
```
jit testing fails on 3.11 debug build
Labels: oncall: jit

test_jit fails very loundly on 3.11 debug build:
It prints thousands of
```
gc:0: ResourceWarning: Object of type torch._C.ScriptFunction is not untracked before destruction
```
And then crash with
```
Modules/gcmodule.c:442: update_refs: Assertion "gc_get_refs(gc) != 0" failed
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7f7f48661800
object refcount : 0
object type     : 0x6a5a2c0
object type name: torch._C.ScriptFunction
object repr     : <refcnt 0 at 0x7f7f48661800>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized
```

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Add plots of LRSchedulers to doc to make it easier to read
Labels: module: docs, triaged, actionable, module: LrScheduler

We can do something similar to what we have for activation functions https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html?highlight=relu#torch.nn.ReLU

A simple script similar to the one shared in https://github.com/pytorch/pytorch/issues/91765 can be used a the basis for generating these plots.

cc @svekars @carljparker
autograd.functional.jacobian : Imaginary part is lost for functions with real input and complex output.
Labels: module: autograd, triaged, has workaround

### 🐛 Describe the bug

As per title, similar to #90499 

```python
import torch
import functorch

m = torch.tensor([[0.30901699437494745, 0.9510565162951535], [-0.9510565162951535, 0.30901699437494745]], dtype=torch.double)

def fn(m):
    return m * 0.5j

jacfwd_o = functorch.jacfwd(fn)(m)
jacobian_o = torch.autograd.functional.jacobian(fn, m)

print(jacfwd_o)
print(jacobian_o)
```

Output:
```
tensor([[[[0.+0.5000j, 0.+0.0000j],
          [0.+0.0000j, 0.+0.0000j]],

         [[0.+0.0000j, 0.+0.5000j],
          [0.+0.0000j, 0.+0.0000j]]],


        [[[0.+0.0000j, 0.+0.0000j],
          [0.+0.5000j, 0.+0.0000j]],

         [[0.+0.0000j, 0.+0.0000j],
          [0.+0.0000j, 0.+0.5000j]]]], dtype=torch.complex128)

tensor([[[[0., 0.],
          [0., 0.]],

         [[0., 0.],
          [0., 0.]]],


        [[[0., 0.],
          [0., 0.]],

         [[0., 0.],
          [0., 0.]]]], dtype=torch.float64)
```

### Versions

master

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
export does not support boolean tensor indexing
Labels: triaged, module: dynamic shapes, module: dynamo

```
import torch._dynamo
torch._dynamo.config.dynamic_shapes=True
def foo(images, handedness):
    right_hand_mask = handedness == 1
    images[right_hand_mask] = images[right_hand_mask].flip(-1)

torch._dynamo.export(foo, torch.randint(0, 256, (512, 1, 1, 96, 96)), torch.randint(0, 1, (512, 1, 1)))
```
raises:
```
dynamic shape operator: aten.index.Tensor
```

My ideal behavior would be that we are capable of returning a tensor with "new" shapes that are unbacked symints, similar to what we want from nonzero. 

Internal command to reproduce the issue: P595150412

cc @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
Torch's affinity setting lead to openvino using only one core.
Labels: triaged, module: intel, intel priority

### 🐛 Describe the bug
Torch's affinity setting lead to openvino using only one core. 
I export a pytorch resnet50 model to OpenVINO IR Model with following steps.
1. export resnet50 to onnx model
```
from torchvision.models import resnet50
from openvino.runtime import Core
import torch
from pathlib import Path

model = resnet50()
model.eval()

dummy_input = torch.randn((1, 3, 224, 224),
                             generator=None,
                             device="cpu",
                             dtype=torch.float32)
onnx_path = Path("resnet50.onnx")

if not onnx_path.exists():
   torch.onnx.export(
           model,
           dummy_input,
           onnx_path,
           opset_version=11,
           do_constant_folding=False,
       )
```
2. Convert onnx model to openvino IR model：
```
mo --input_model "resnet50.onnx" --input_shape "[1,3, 224, 224]" --mean_values="[123.675, 116.28 , 103.53]" --scale_values="[58.395, 57.12 , 57.375]" --data_type FP32 --output_dir model
```
3. Export Intel KMP affinity env:
```
export LD_PRELOAD=conda_env_home/lib/libiomp5.so # like /usr/local/envs/test/lib/libiomp5.so
export KMP_AFFINITY=granularity=fine,compact,1,0
```

4. At last, run openvino inference:
```
from openvino.runtime import Core
from pathlib import Path
import numpy as np

dummy_input = np.random.randn(1, 3, 224, 224)

config = {
        "CPU_THREADS_NUM": "48"
    }

ie = Core()
classification_model_xml = "model/resnet50.xml"
model = ie.read_model(model=classification_model_xml)

import torch # if import torch here, openvino will use only one core.

compiled_model = ie.compile_model(model, "CPU", config)

# import torch # if import torch here, openvino will use 48 cores.

input_layer = compiled_model.input(0)
output_layer = compiled_model.output(0)
ir = compiled_model.create_infer_request()
import time
s = time.time()
for i in range(200):
    ir.infer(inputs={input_layer.any_name: dummy_input})

print("time cost: " + str(time.time() - s))
```

BTW: gomp affinity `export OMP_PROC_BIND=CLOSE` has the same hehaviour.

### Versions

torch: 1.13.1
python: 3.7.10
cpu: Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
Set up env with 
```
conda create -n test -y python==3.7.10 setuptools==58.0.4 
conda activate test
conda install intel-openmp
pip3 install torch torchvision
pip3 install openvino openvino-dev
```

cc @frank-wei @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
sympy failure on model when dynamic_shapes=True
Labels: triaged, module: dynamic shapes, module: dynamo

Error looks like: 
```
Traceback (most recent call last):
  File "torch/_dynamo/utils.py", line 1078, in run_node
    return node.target(*args, **kwargs)
RuntimeError: TypeError: cannot determine truth value of Relational
At:
  sympy/core/relational.py(297): __nonzero__
  torch/fx/experimental/symbolic_shapes.py(943): _maybe_guard_eq
  torch/fx/experimental/symbolic_shapes.py(989): evaluate_expr
  torch/fx/experimental/symbolic_shapes.py(209): bool_
```
This is happening on an internal model. Full stack trace: P595091810.

Repro: patch D42443834 and run the command in the comment: https://fburl.com/diff/f5nphw9a

cc @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
Torchdynamo with onnxrt backend generating fake tensor errors
Labels: triaged, bug, oncall: pt2

### 🐛 Describe the bug

Hi,
  I'm trying to use torchdynamo with onnxrt (both onnxrt_cuda and onnxrt_cpu backends) following the resnet example described in  https://pytorch.org/tutorials/intermediate/dynamo_tutorial.html but I'm hitting an error

"torch._dynamo.exc.BackendCompilerFailed: onnxrt_cuda raised Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten.convolution.default(*(FakeTensor(FakeTensor(..., device='meta', size=(16, 3, 128, 128)), cuda:0), Parameter containing:
tensor([[[[...]]]], device='cuda:0', requires_grad=True),"

I've tried many examples from torchbench and all are hitting same error. It does not happen with other backends e.g. aot_cudagraphs etc. I've also force set the FakeTensorMode in fake_tensor.py but that hit another error down the line. Am I missing something obvious here?

Torch version: 2.0.0a0+gitdf46ba4
onnxrt version: 1.13.1

I was thinking something obvious is missing here before I dig into this in detail.

### Error logs

"torch._dynamo.exc.BackendCompilerFailed: onnxrt_cuda raised Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten.convolution.default(*(FakeTensor(FakeTensor(..., device='meta', size=(16, 3, 128, 128)), cuda:0), Parameter containing:
tensor([[[[...]]]], device='cuda:0', requires_grad=True),"

### Minified repro

import torch
import torch._dynamo as dynamo

# Returns the result of running `fn()` and the time it took for `fn()` to run,
# in seconds. We use CUDA events and synchronization for the most accurate
# measurements.
def timed(fn):
    start = torch.cuda.Event(enable_timing=True)
    end = torch.cuda.Event(enable_timing=True)
    start.record()
    result = fn()
    end.record()
    torch.cuda.synchronize()
    return result, start.elapsed_time(end) / 1000

# Generates random input and targets data for the model, where `b` is
# batch size.
def generate_data(b):
    return (
        torch.randn(b, 3, 128, 128).to(torch.float32).cuda(),
        torch.randint(1000, (b,)).cuda(),
    )

from torchvision.models import resnet18
def init_model():
    return resnet18().to(torch.float32).cuda()

def eval(mod, inp):
    return mod(inp)
torch._dynamo.config.verbose=True
model = init_model()
eval_opt = dynamo.optimize("onnxrt_cuda")(eval)

inp = generate_data(16)[0]
print("eager:", timed(lambda: eval(model, inp))[1])
print("dynamo:", timed(lambda: eval_opt(model, inp))[1])


cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
Hessian produces wrong results, but works if I add a perturbation
Labels: triaged

I have the following function I would like the Hessian of: 

```python
def rbf(x1, x2, l):
    diff = x1/l - x2/l
    dist = torch.linalg.norm(diff)
    return torch.exp(-dist * dist)
```

Taking the Hessian with `functorch.jacfwd` and `functorch.grad` seems to get me an incorrect result on a naive example. According to my derivation by hand, this should return `2.0 / l**2 * torch.eye(16)`. If I add a small number to one of the terms, the Hessian function returns the correct result:

```python
jacfwdgrad_rbf = jacfwd(grad(rbf, argnums=0), argnums=1)  # doing d^2f/dx1dx2

a = torch.randn(16)
l = 1.0

print(torch.allclose(jacfwdgrad_rbf(a, a, l), 2/l/l * torch.eye(16)))  # prints False

print(torch.allclose(jacfwdgrad_rbf(a, a+1e-8, l), 2/l/l * torch.eye(16)))  # prints True
```
Is there some bug that means the former check doesn't work, or is there something I'm missing?
Proxy/cache server option/hooks for downloading model checkpoints and dataset archive files in cloud environment
Labels: triaged, module: hub

### 🚀 The feature, motivation and pitch

Often in cloud environments, we may want to cache the pretrained model checkpoints (e.g. certain torchvision resnet50) and dataset archives (e.g. COCO tar files) on a custom endpoint (e.g. S3 bucket with http access) for faster downloads and avoiding overloading public servers, especially given problems with over-downloading in all DDP replicas:  https://github.com/pytorch/pytorch/issues/68320

It would be nice to have some mechanisms / hooks in torch domain libraries for rewriting the URLs. (maybe in torch.hub?)

It would also be nice to generalize content-addressability of checkpoints (at least as an option). Currently some checkpoints have hashes in their file names in torchvision, but maybe it's not made obligatory.

An advanced hook could by itself avoid race-conditioned downloads and could by itself download to S3 bucket (via boto3) and then return the fixed path

### Alternatives

_No response_

### Additional context

_No response_

cc @nairbv @NicolasHug @vmoens @jdsgomes
[PT2.0 Feature Proposal] GNN inference and training optimization on CPU
Labels: feature, triaged, oncall: pt2

### 🚀 The feature, motivation and pitch

This ticket is per request from Ibrahim Siddiqui as part of PT2.0 feature proposal process. The feature proposal has also been submitted @ https://docs.google.com/spreadsheets/d/1TzGkWuUMF1yTe88adz1dt2mzbIsZLd3PBasy588VWgk/edit#gid=790902532 

Graph Neural Networks (GNNs) is a powerful tool to analyze graph structure data. This proposed feature is targeted at improving GNN inference and training performance on CPU. PyG is a very popular library built upon PyTorch for GNN.  Performance of PyG strongly depends on PyTorch. Currently on CPU, GNN models of PyG run extremely slow due to the lack of GNN-related sparse matrix multiplication ops (i.e., spmm_reduce) and the lack of several critical kernel level optimizations (scatter/gather, etc.) tuned for GNN compute. 
We submitted PRs to optimize the more critical hot spots, and demonstrated a 2-5x performance speedup on PyG benchmarks/examples, OGB benchmarks for single node inference and training. 
To be specific, the following kernels are optimized from torch side (for inference and training stage, targeting at torch 2.0):

- scatter_add and scatter_reduce: this is the hotspot in Message Passing when the edge index is stored in COO, which takes 49% of runtime in SAGE + Reddit.
- spmm_reduce: this is the hotspot in Message Passing when the edge index is stored in CSR, which takes 97% of runtime in GCN + ogbn-products.
- Gather: backward of scatter_add. Current implementation from ATen is not efficient when the index is an expanded tensor.
- sampled_addmm: this one can be used for feature aggregation when the edge index is in CSR, the current CPU impl is a reference design which converts the sparse tensor back to dense for computation.

The following will be nice to have:

- segment_reduce: used when the edge index is logically CSR but physically stored in individual dense tensors. The current impl in torch is a reference (single thread and non-vectorized).
- Multi-Aggregation mode: for scatter_reduce or spmm_reduce, the graph might want to initiate multiple ‘reduce’ types spontaneously. Allowing multi-aggregation mode would greatly save time for such scenarios since the src feature matrix (which is usually very large and accessed in an in-direct order) could be only read once.


### Alternatives

_No response_

### Additional context

**Here are the PRs that are required for this feature:**
* #82703
* #83727 
* #87586 

**There PRs are nice to have for PT2.0.**
* #90978 
* #91500 


cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
elastic job failed when scale down
Labels: oncall: distributed, triaged, oncall: r2p

### 🐛 Describe the bug

i run one job with three worker in k8s cluster,just like this command:
```shell
torchrun --nnodes=1:3 --nproc_per_node=1 --rdzv_id=1 --rdzv_backend=c10d --max_restarts=100 --rdzv_endpoint=172.22.154.120:60001 /wdk/elasticjob/pytorch/main.py --arch resnet50 --epochs 100 --batch-size 128 /wdk/imagenet/
```
when i delete a worker,the  job should keep running .but in my test,the job is terminated,the worker0 job is like this
![image](https://user-images.githubusercontent.com/5036200/211448842-76590e23-ab00-48fd-9bcf-614f890f1417.png)
![image](https://user-images.githubusercontent.com/5036200/211448861-2f367af1-875b-44e0-a0f2-035862926986.png)


the worker 1 log is like this:
![image](https://user-images.githubusercontent.com/5036200/211448941-1f99b21a-5fa2-4148-b393-c0c0939ee8a7.png)


any suggestions? thanks


### Versions

1.13

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
torchrun elastic always “address already in use” error
Labels: oncall: distributed, triaged, oncall: r2p

### 🐛 Describe the bug

i run the model in k8s pod, there is no other process in the pod. but this problem occurs frequently. the torch version is 1.13. i submit the job use this command：
```shell
torchrun --nnodes=1:3 --nproc_per_node=1 --rdzv_id=1 --rdzv_backend=c10d --max_restarts=100 --rdzv_endpoint=172.22.154.120:60001 /wdk/elasticjob/pytorch/main.py --arch resnet50 --epochs 100 --batch-size 128 /wdk/imagenet/
```
the error is like this 
![image](https://user-images.githubusercontent.com/5036200/211445837-a4eabcd5-c261-42ba-b81c-f8f2c4da8cb5.png)


after check the pytorch code, the code will bind to get a port,then release this port as the master port. maybe this is the reason.
``` python
def get_socket_with_port() -> socket.socket:
    """
    Returns a free port on localhost that is "reserved" by binding a temporary
    socket on it. Close the socket before passing the port to the entity
    that requires it. Usage example

    ::

    sock = _get_socket_with_port()
    with closing(sock):
        port = sock.getsockname()[1]
        sock.close()
        # there is still a race-condition that some other process
        # may grab this port before func() runs
        func(port)
    """

    addrs = socket.getaddrinfo(
        host="localhost", port=None, family=socket.AF_UNSPEC, type=socket.SOCK_STREAM
    )
    for addr in addrs:
        family, type, proto, _, _ = addr
        s = socket.socket(family, type, proto)
        try:
            s.bind(("localhost", 0))
            s.listen(0)
            return s
        except OSError as e:
            s.close()
            log.info("Socket creation attempt failed.", exc_info=e)
    raise RuntimeError("Failed to create a socket")
````

Any method to slove this problem ?thanks.


### Versions

1.13

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
ToTensor deadlock in subprocess
Labels: module: multiprocessing, triaged


🐛 Describe the bug


I load a checkpoint in main process, then I start a subprocess to call ToTensor and the subprocess hang forever.

the following codes will reproduce my problem:
```python
import numpy as np
from PIL import Image
from multiprocessing import Process
import torchvision
from torchvision.transforms import ToTensor

a = np.ones((224, 224, 3), dtype=np.uint8)
a = Image.fromarray(a, mode='RGB')
model = torchvision.models.resnet50(True)
p = Process(target=ToTensor(), args=(a,))
p.start()
p.join()
print('end')
```


### Versions

torch version: 1.12.0
torchvision version: 0.13.0

cc @VitalyFedyunin @ejguan
No setting to allow collecting the first trace early.
Labels: oncall: profiler

The default schedule behavior is to 'wait' immediately following the skip_first steps. As a consequence, if we want to wait 5k steps between traces, the earliest possible trace we can collect is after 5k steps. We'd like to be able to record a trace early in training, like at step 100, and then subsequently once every 5k steps.

cc @robieta @chaekit @aaronenyeshi @ngimel @nbcsm @guotuofeng @guyang3532 @gaoteng-git @tiffzhaofb
ddp vs fsdp
Labels: oncall: distributed, module: fsdp

### 🐛 Describe the bug


I used fsdp+ShardedGradScaler to train my model. Compared with apex. amp+ddp, the precision of my model has decreased.

The ddp is like
```
model, optimizer = amp.initialize(model, optimizer,
                                      num_losses=len(task2scaler),
                                      enabled=opts.optimizer["fp16"], opt_level='O2')

model = DDP(model, device_ids=[get_local_rank()], output_device=get_local_rank(), find_unused_parameters=False)
with amp.scale_loss(loss, optimizer, delay_unscale=delay_unscale,
                    loss_id=task2scaler[name]) as scaled_loss:
    
    scaled_loss.backward()
```
and the fsdp is like
```
model = FSDP(model,
                 auto_wrap_policy=t5_auto_wrap_policy,
                 mixed_precision=MixedPrecision(
        param_dtype=torch.bfloat16,
        # Gradient communication precision.
        reduce_dtype=torch.bfloat16,
        # Buffer precision.
        buffer_dtype=torch.bfloat16,
    ),
                 device_id=torch.cuda.current_device(),
                 sharding_strategy=ShardingStrategy.SHARD_GRAD_OP,  # ZERO2
                 backward_prefetch=BackwardPrefetch.BACKWARD_PRE)
```

What is possible reason?

### Versions

。

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
torch.Categorical samples indexes with 0 probability when given logits as argument
Labels: module: distributions, triaged

### 🐛 Describe the bug
`torch.distributions.Categorical` samples indexes with 0 probability when given logits as argument.

## To reproduce

```python
import torch
from torch.distributions import Categorical

logits = torch.ones((1_000_000, 500))
logits[:, 1:] = -1e5 
probs = Categorical(logits=logits)

sample = probs.sample()
torch.sum(sample > 0)
```

### Expected behaviour
The sum should be 0, because probs.sample() should only extract the index 0, because the other logits are $-10^5$ while the logit for the index 0 is 1

### Versions

[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] numpydoc==1.4.0
[pip3] torch==2.0.0.dev20230105
[pip3] torchaudio==2.0.0.dev20230105
[pip3] torchvision==0.15.0.dev20230105
[conda] blas                      1.0                         mkl
[conda] cpuonly                   2.0                           0    pytorch-nightly
[conda] mkl                       2021.4.0           haa95532_640
[conda] mkl-service               2.4.0            py39h2bbff1b_0
[conda] mkl_fft                   1.3.1            py39h277e83a_0
[conda] mkl_random                1.2.2            py39hf11a4ad_0
[conda] numpy                     1.21.5           py39h7a0a035_3
[conda] numpy-base                1.21.5           py39hca35cd5_3
[conda] numpydoc                  1.4.0            py39haa95532_0
[conda] pytorch                   2.0.0.dev20230105     py3.9_cpu_0    pytorch-nightly
[conda] pytorch-mutex             1.0                         cpu    pytorch
[conda] torchaudio                2.0.0.dev20230105        py39_cpu    pytorch-nightly
[conda] torchvision               0.15.0.dev20230105        py39_cpu    pytorch-nightly

cc @fritzo @neerajprad @alicanb @nikitaved
torchrun --help is too slow
Labels: oncall: distributed, triaged, oncall: r2p, topic: improvements, topic: performance

### 🚀 The feature, motivation and pitch

```
❯ hyperfine -Nw3 -r10 'torchrun --help'
Benchmark 1: torchrun --help
  Time (mean ± σ):     943.2 ms ±   7.1 ms    [User: 855.5 ms, System: 84.6 ms]
  Range (min … max):   933.9 ms … 959.5 ms    10 runs
```

`torchrun --help` need to wait 1 second -- I advise to move the parse_args part to `torch/distributed/run/__main__.py`, And don't import any slow code in `__main__.py`, then if not `--help` after parse_args, import true main function and pass parsed args to it. It should fasten `torchrun --help`.

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
torchrun default value of command line options
Labels: oncall: distributed, triaged, oncall: r2p

### 🚀 The feature, motivation and pitch

1. Now `torchrun`'s `--nproc_per_node`'s default value is `1`. Why not use `auto`? `auto` decides the value automatically, which seems to be better than fixed integer.

> --nproc_per_node NPROC_PER_NODE
>                       Number of workers per node; supported values: [auto,
>                       cpu, gpu, int].

2. Now `torchrun`'s `--start_method`'s default value is `spawn`. I can understant its reasom: windows only support spawn while unix support spawn, fork, forkserver. So use `spawn` as default value can make it work by default in more OSs. However, according to <https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods>

> spawn
>   ... Starting a process using this method is rather slow compared to using fork or forkserver.

So why not provide a value named `auto`, when the OS is unix, it chooses `fork/forkserver` which is faster, when the OS is windows, it chooses `spawn`? It seems to be better than fixed method, maybe it can be the default value of `--start_method`?

TIA for discussion!

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
jacrev over huber function
Labels: needs reproduction, triaged, module: functorch

I get the following error when trying to jacrev through this:

```
def huber(input_, delta=1.0):
    out = torch.zeros_like(input_)

    out[input_ < delta] = .5 * input_[input_ < delta] **2

    out[input_ >= delta] = delta*(torch.abs(input_[input_ >= delta]) - .5*delta)
    return out
```

Here's the error:

```
 File "/home/drake/drake-stacking/src/comparisons/newton_raphson_equality.py", line 85, in <module>
    from copy import deepcopy
  File "/home/drake/drake-stacking/src/utils/trajopt_class.py", line 299, in get_newton_with_equality_step
    res = jacrev(jacrev(self.lagrangian_from_flat, argnums=0), argnums=(0, 1))(self.param_model.get_primal_var_param_flat(),
  File "/usr/local/lib/python3.9/dist-packages/functorch/_src/eager_transforms.py", line 459, in wrapper_fn
    vjp_out = _vjp_with_argnums(func, *args, argnums=argnums, has_aux=has_aux)
  File "/usr/local/lib/python3.9/dist-packages/functorch/_src/vmap.py", line 35, in fn
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/functorch/_src/eager_transforms.py", line 289, in _vjp_with_argnums
    primals_out = func(*primals)
  File "/usr/local/lib/python3.9/dist-packages/functorch/_src/eager_transforms.py", line 474, in wrapper_fn
    results = vmap(vjp_fn)(basis)
  File "/usr/local/lib/python3.9/dist-packages/functorch/_src/vmap.py", line 362, in wrapped
    return _flat_vmap(
  File "/usr/local/lib/python3.9/dist-packages/functorch/_src/vmap.py", line 35, in fn
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/functorch/_src/vmap.py", line 489, in _flat_vmap
    batched_outputs = func(*batched_inputs, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/functorch/_src/eager_transforms.py", line 323, in wrapper
    result = _autograd_grad(flat_primals_out, flat_diff_primals, flat_cotangents,
  File "/usr/local/lib/python3.9/dist-packages/functorch/_src/eager_transforms.py", line 113, in _autograd_grad
    grad_inputs = torch.autograd.grad(diff_outputs, inputs, grad_outputs,
  File "/usr/local/lib/python3.9/dist-packages/torch/autograd/__init__.py", line 300, in grad
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: values.dim() >= num_leading_nones + max_index_dim INTERNAL ASSERT FAILED at "../aten/src/ATen/functorch/BatchRulesScatterOps.cpp":571, please report a bug to PyTorch.

```


cc @zou3519 @Chillee @samdow @soumith @kshitij12345 @janeyx99
Adding a page for subfolder/subfile overview/descriptions in the developer wiki
Labels: module: docs, triaged

### 🚀 The feature, motivation and pitch

Hi! 

As noted here (https://dev-discuss.pytorch.org/t/what-should-we-do-about-developer-documentation/491/10), the developer documentation could be improved upon by adding a few more things.
I want to add a page in the developer wikis which discuss the overview/intentions of each subfolder/subfiles in the main repo (a brief example of what I mean here : https://dear-opera-8cc.notion.site/Pytorch-c16c4b333e9648139e8448aa0171744f )

### Alternatives

_No response_

### Additional context

_No response_

cc @svekars @carljparker
Missing python 3.11 on anaconda for torch 1.13.1
Labels: oncall: binaries

### 🐛 Describe the bug

There is a torch 1.13.1 package with Python 3.11:

https://download.pytorch.org/whl/cpu/torch-1.13.1%2Bcpu-cp311-cp311-linux_x86_64.whl

However, there are no packages for Python 3.11 at
https://anaconda.org/pytorch/pytorch/files

### Versions

torch 1.13.1

cc @ezyang @seemethere @malfet