gopikrsmscs/torch-issues huggingface dataset:
assert callable(unaltered_fn)
Labels: high priority, triaged, oncall: pt2

### 🐛 Describe the bug

This is a bug generated from https://github.com/pytorch/pytorch/issues/97078

To reproduce, check out transformers and patch (I tested on a515d0a77c769954ac2f0151a2a99c04d8d6cf95)

```
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
index 2eb081af7..886df74c1 100755
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -1572,8 +1572,7 @@ class Trainer:
 
         # torch.compile() needs to be called after wrapping the model with FSDP or DDP
         # to ensure that it accounts for the graph breaks required by those wrappers
-        if self.args.torch_compile:
-            model = torch.compile(model, backend=self.args.torch_compile_backend, mode=self.args.torch_compile_mode)
+        model = torch.compile(model, backend=self.args.torch_compile_backend, mode=self.args.torch_compile_mode, dynamic=True)
 
         return model
 
```

Then, run

```
pytest tests/trainer/test_trainer.py --tb=native -k test_adafactor_lr_none
```

It fails with

```
______________________________________________ TrainerIntegrationPrerunTest.test_adafactor_lr_none ______________________________________________
Traceback (most recent call last):
  File "/data/users/ezyang/a/transformers/tests/trainer/test_trainer.py", line 465, in setUp
    trainer.train()
  File "/data/users/ezyang/a/transformers/src/transformers/trainer.py", line 1658, in train
    return inner_training_loop(
  File "/data/users/ezyang/a/transformers/src/transformers/trainer.py", line 1745, in _inner_training_loop
    model = self._wrap_model(self.model_wrapped)
  File "/data/users/ezyang/a/transformers/src/transformers/trainer.py", line 1575, in _wrap_model
    model = torch.compile(model, backend=self.args.torch_compile_backend, mode=self.args.torch_compile_mode, dynamic=True)
  File "/data/users/ezyang/a/pytorch/torch/__init__.py", line 1600, in compile
    return torch._dynamo.optimize(backend=backend, nopython=fullgraph, dynamic=dynamic, disable=disable)(model)
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/eval_frame.py", line 499, in optimize
    return _optimize_catch_errors(
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/eval_frame.py", line 401, in _optimize_catch_errors
    return OptimizeContext(
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/eval_frame.py", line 330, in __init__
    compiler_fn = innermost_fn(callback)
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/eval_frame.py", line 149, in innermost_fn
    assert callable(unaltered_fn)
AssertionError
```

It's possible that HF is misusing the torch.compile API (there's some sort of repeated wrapping going on, it seems like), but even if that's true, it shouldn't assert error.

cc @gchanan @zou3519 @soumith @msaroufim @wconstab @ngimel @bdhirsh @stas00 

### Versions

master
Backwards graph is labeled incorrectly when dynamic=True
Labels: triaged, oncall: pt2, module: dynamic shapes

### 🐛 Describe the bug

Run `TORCH_COMPILE_DEBUG=1 python tt.py` with

```
#!/usr/bin/env python3

import time
import torch
import torch._dynamo as dynamo
import torchvision.models as models

model = models.alexnet()

optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
compiled_model = torch.compile(model, dynamic=True)

x = torch.randn(16, 3, 224, 224)
optimizer.zero_grad()

epoches=10
count = []
for epoch in range(epoches):
    start = time.time()

    #out = model(x)
    out = compiled_model(x)
    out.sum().backward()
    optimizer.step()

    end = time.time()
    count.append(end - start)
    print(f"Epoch {epoch}/{epoches} time: {end - start}")

print(f"Epoch avg time: {sum(count)/len(count)}")
```

but really any training script will work. Inspect the inductor directory:

```
$ ls torch_compile_debug/run_2023_04_05_07_17_04_323873-pid_27994/torchinductor/
aot_model___0_debug.log  model__0_forward_1.0  model__0_inference_2.1
```

The backwards graph is incorrectly reported as an inference graph.

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh @Chillee 

### Versions

master
TORCH_COMPILE_DEBUG and TORCH_LOGS interact badly
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

Here's an example log: https://gist.github.com/ezyang/6e2904c8ecbd863eefcbee7456ada544 for this run:

```
TORCH_COMPILE_DEBUG=1 PYTHONUNBUFFERED=1 WANDB_DISABLED=true TORCH_LOGS=dynamo,inductor,guards CUDA_VISIBLE_DEVICES=3 PYTHONPATH=src pp python examples/pytorch/translation/run_translation.py --model_name_or_path t5-base --do_train --source_lang en --target_lang de --source_prefix 'translate English to German: ' --dataset_name stas/wmt14-en-de-pre-processed --output_dir /tmp/tst-translation --num_train_epochs 1 --per_device_train_batch_size=1 --max_train_samples 1000 --overwrite_output_dir --seed 1137 --per_device_eval_batch_size 1 --fp16 --torch_compile 2>&1 | tee comp.log
```

Some things to note:

1. Despite not asking for it, I'm still getting inductor DEBUG logs printed to stderr:

```
[2023-04-05 06:43:18,087] torch._inductor.codegen.triton.__schedule: [DEBUG] Schedule:
```

My intention for TORCH_COMPILE_DEBUG was to get the directory dump; I think it shouldn't interact with console output

2. It keeps repeatedly printing this:

```
04/05/2023 06:43:27 - WARNING - torch._logging._internal - Using TORCH_LOGS environment variable for log settings, ignoring call to set_logs
04/05/2023 06:43:27 - WARNING - torch._logging._internal - Using TORCH_LOGS environment variable for log settings, ignoring call to set_logs
04/05/2023 06:43:27 - WARNING - torch._logging._internal - Using TORCH_LOGS environment variable for log settings, ignoring call to set_logs
```

### Versions

master

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh
`torch.Tensor.layout` is not documented
Labels: module: docs, triaged, module: python frontend

### 🐛 Describe the bug

This looks like a public function, so it should have been documented, but it is not:
```
% python -c "import torch;print(torch.Tensor.layout.__doc__)"   
None
```

I wonder if this intentional. If not, we should add the documentation.

### Versions

2.0/nightly

cc @svekars @carljparker @albanD
Contribute to the privateuse1 backend.
Labels: module: internals, triaged, module: backend

### 🚀 The feature, motivation and pitch

This issue is used to discuss how to improve the PrivateUse1 backend to facilitate third-party manufacturers to access Pytorch.
With the popularity of pytorch and the evolution of computing acceleration hardware, the strong coupling between pytorch and cuda has become a serious problem. So the completeness of PrivateUse1 is what third-party hardware manufacturers need. After all, we can't add more enumerated types to DeviceType unless we are a big company like Apple or Intel (just kidding🫡).
For each feature, I will summarize it into this issue.
Please join us, thank you.

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @bhosmer @smessmer @ljk53 @bdhirsh
[PTD][Checkpoint] Enable single_file_per_rank for fsspec storage read/write
Labels: oncall: distributed, triaged

### 🚀 The feature, motivation and pitch

With our current setup, single_file_per_rank is not supported for fsspec StorageWriter and StorageReader. This means we can only write single file per tensor/blob, which will significantly affect our performance. 

We need to support single file per rank in fsspec and add the option back.

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
pip doesn't install the right version of pytorch when torchtext is involved
Labels: oncall: binaries

### 🐛 Describe the bug

I encountered some weird installation problems while installing the nightly version.

```bash
pip3 install --force-reinstall --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cu118
```
This is fine.

```bash
pip3 install --force-reinstall --pre torch torchtext torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cu118
```
This is going to install torch2.0.0 instead of the nightly version.

I’m sure it worked a few days ago.

### Versions

PyTorch version: 2.1.0.dev20230404+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

CMake version: version 3.26.1
Libc version: glibc-2.31

Versions of relevant libraries:
[pip3] torch==2.1.0.dev20230404+cu118
[pip3] torchaudio==2.1.0.dev20230404+cu118
[pip3] torchdata==0.6.0
[pip3] torchtext==0.15.1
[pip3] torchvision==0.16.0.dev20230404+cu118
[pip3] triton==2.1.0


cc @seemethere @malfet
Intermittent failure of mobilenet_v3_large
Labels: triaged, module: flaky-tests, oncall: pt2

Repro:
```
python benchmarks/dynamo/torchbench.py --training --accuracy --device cuda --inductor --amp --only mobilenet_v3_large
```
See more details at https://github.com/pytorch/pytorch/pull/98314

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
[cpu] Fix div with rounding_mode="floor" when division overflows
Labels: module: cpu, open source, release notes: python_frontend, topic: bug fixes

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #98330
* #98329

Fixes #77742

Sleef_fmod returns NaN when the division overflows, but we should be
returning inf here. So lets just use the direct division result
whenever it's nonfinite.

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
Inconsistent nn.KLDivLoss behavior: 0s in target OK on cpu, but gives nan on mps
Labels: triaged, module: mps

### 🐛 Describe the bug

KLDivLoss is supposed to take the log of a probability distribution as target, sometimes this target contains 0s. This is handled correctly when device='cpu', but when device='mps' we get nans. Current workaround is to add some small eps to the target.

```python

torch.manual_seed(1)

x = torch.rand(10, 8, device='mps')
x = x / x.sum(dim=1, keepdim=True)
x = log_softmax(x, dim=-1)

y = torch.rand(10, 8, device='mps')
y = y / y.sum(dim=1, keepdim=True)

criterion = nn.KLDivLoss(reduction="sum")
print(criterion(x, y), criterion(x.to('cpu'), y.to('cpu')))

# mask out random entries of y
mask = torch.rand(10, 8, device='mps') < 0.5
y = y * mask
print(criterion(x, y), criterion(x.to('cpu'), y.to('cpu')))
```

Outputs:
```tensor(1.6974, device='mps:0') tensor(1.6974)```
```tensor(nan, device='mps:0') tensor(1.0370)```

### Versions

```
PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.2.1 (x86_64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.202)
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.10 (main, Mar 21 2023, 13:41:39) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-10.16-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==2.0.0
[pip3] torchaudio==2.0.0
[pip3] torchvision==0.15.0
[conda] blas                      1.0                         mkl  
[conda] ffmpeg                    4.3                  h0a44026_0    pytorch
[conda] mkl                       2021.4.0           hecd8cb5_637  
[conda] mkl-service               2.4.0           py310hca72f7f_0  
[conda] mkl_fft                   1.3.1           py310hf879493_0  
[conda] mkl_random                1.2.2           py310hc081a56_0  
[conda] numpy                     1.23.5          py310h9638375_0  
[conda] numpy-base                1.23.5          py310ha98c3c9_0  
[conda] pytorch                   2.0.0                  py3.10_0    pytorch
[conda] torchaudio                2.0.0                 py310_cpu    pytorch
[conda] torchvision               0.15.0                py310_cpu    pytorch```

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
hf_Longformer regression caused by https://github.com/pytorch/pytorch/pull/98119
Labels: triaged, oncall: pt2

Bisect points to https://github.com/pytorch/pytorch/pull/98119 and reverting it makes the error go away.

Repro:
```
python benchmarks/dynamo/torchbench.py  --inductor --amp --training --accuracy  --device cuda --only hf_Longformer
```

Error msg:
```
ValueError: Cannot view a tensor with shape torch.Size([4, 12, 1024, 513]) and strides (6303744, 513, 6156, 1) as a tensor with shape (48, 4, 256, 513)!
```

Q: Why is this not caught by CI?
A: Because `hf_Longformer` uses more than 24GB GPU memory, it is currently skipped on CI. I will fix this by running larger models on A100 instances.

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
Broken mypy check in test_type_hints.py::TestTypeHints::test_doc_examples
Labels: module: typing, triaged

### 🐛 Describe the bug

There are two problems with this test that surfaces today:

1. The `mypy==0.960` version we are using has a bug with Python 3.10.6+ https://github.com/python/mypy/issues/13627.  It breaks for Python 3.11 in CI.  The fix requires an upgrade to 0.981 or newer.  The bug is that mypy not only checks stuffs in PyTorch, but also PyTorch dependencies.  Specifically, it's numpy `/opt/conda/envs/py_3.11/lib/python3.11/site-packages/numpy/__init__.pyi:638:48: error: Positional-only parameters are only supported in Python 3.8 and greater  [syntax]`.  For example, https://hud.pytorch.org/pytorch/pytorch/commit/73b06a0268bb89c09a86f16fa0f72818baa4b250.  I found a PR https://github.com/pytorch/pytorch/pull/91983 to upgrade mypy.  Could we go ahead with that? (cc @ezyang @malfet @rgommers @xuzhao9 @gramster)
2. The test has been broken on ASAN after https://github.com/pytorch/pytorch/pull/94525 (cc @wanchaol) with the following error `mypy.ini:5:1: error: Error importing plugin "numpy.typing.mypy_plugin": No module named 'numpy.typing.mypy_plugin'  [misc]`, for example https://github.com/pytorch/pytorch/actions/runs/4601886643/jobs/8130489457

So why today you ask?  As these changes have been in trunk for a while.  It turns out that `test_doc_examples` was slow, so it wasn't run in these jobs till now.

```
test_type_hints.py::TestTypeHints::test_doc_examples SKIPPED [0.0002s] (test is slow; run with PYTORCH_TEST_WITH_SLOW to enable test)
```

The test is run as part of [slow jobs](https://github.com/pytorch/pytorch/blob/master/.github/workflows/slow.yml) but they don't cover 3.11 nor ASAN to trigger those bugs (Ouch!)

I have disabled the test in the mean time while we take actions https://github.com/pytorch/pytorch/issues/98259

### Versions

PyTorch CI
DISABLED test_doc_examples (__main__.TestTypeHints)
Labels: module: typing, triaged, skipped

Platforms: linux

This test was disabled because it is failing on master ([recent examples](https://hud.pytorch.org/failure/%2Fopt%2Fconda%2Fenvs%2Fpy_3.11%2Flib%2Fpython3.11%2Fsite-packages%2Fnumpy%2F__init__.pyi%3A638%3A48%3A%20error%3A%20Positional-only%20parameters%20are%20only%20supported%20in%20Python%203.8%20and%20greater%20%20%5Bsyntax%5D)).

This looks like an upstream mypy issue https://github.com/python/mypy/issues/13627 as documented in https://github.com/pytorch/pytorch/pull/94255.  Disable the test to investigate further

cc @ezyang @malfet @rgommers @xuzhao9 @gramster
[CUDA][MAGMA][Linalg] Remove MAGMA from CUDA linear algebra dependencies
Labels: triaged, open source, ciflow/trunk, release notes: releng, ciflow/periodic, no-stale, keep-going

# wip

linear algebra on CUDA (non-ROCM) devices will only use cuSOLVER library; magma will be removed

Todo:
- [ ] remove this has_magma check in pytorch/builder for cuda build https://github.com/pytorch/builder/blob/3df3313b12e91a9feae40f15681f505b9f3d1354/check_binary.sh#L392-L393, and make sure https://github.com/pytorch/pytorch/pull/98256/commits/34194b8f3b42f0a63098cddefa565cb20aa0b171 is reverted
[Dynamo] Enable `dynamo.export` for huggingface models w/ `ModelOutput`
Labels: feature, triaged, module: pytree, oncall: pt2, module: dynamo, module: export

### 🚀 The feature, motivation and pitch

Initially reported at #96386, `dynamo.export` raises for any huggingface model that returns a subclass of `ModelOutput`.

Here is an experimental patch I have done to unblock `dynamo.export` based ONNX export. The idea is to extend `pytree` to be able to flatten/unflatten `ModelOutput` and its subclasses.

https://github.com/pytorch/pytorch/blob/95de585a7f208476ae06894acbcc897a0bf9abed/torch/onnx/_internal/fx/exporter.py#L21-L68

I wonder if it is a good idea to bring this native into `pytree`? The context of this patch needs to cover not only `dynamo.export` but any other subsequent fx pass that may interact with the formatted output, otherwise it won't recognize the output structure.

Feedbacks and suggestions are welcomed. Feel free to let me know if there are other more suitable solutions.

@jansel @voznesenskym @wconstab  

### Alternatives

_No response_

### Additional context

_No response_

cc @zou3519 @ezyang @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @soumith @desertfire
inductor `compile_fx_inner()` segfaults on `torch.isinf`
Labels: triaged, oncall: pt2, module: inductor

minimal repro:
```
import torch
from torch._inductor.compile_fx import compile_fx_inner
from torch._subclasses import FakeTensorMode
from torch.fx.experimental.proxy_tensor import make_fx

def f(x):
    isinf = torch.isinf(x)
    return [isinf]

with FakeTensorMode():
    inp = torch.ones(2048, device='cuda')
    fx_g = make_fx(f)(inp)
    fx_inner = compile_fx_inner(fx_g, [inp])
```

This fails with:
```
  File "/tmp/torchinductor_hirsheybar/3h/c3hkvftvlgct3scdmbgllet56gcfozfyloqidc57nlxmixrk5ioq.py", line 45, in <module>
    async_compile.wait(globals())
  File "/scratch/hirsheybar1/work/pytorch/torch/_inductor/codecache.py", line 876, in wait
    scope[key] = result.result()
  File "/scratch/hirsheybar1/work/pytorch/torch/_inductor/codecache.py", line 734, in result
    self.future.result()
  File "/scratch/hirsheybar1/work/py38/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/scratch/hirsheybar1/work/py38/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
```

cc @ezyang @soumith @msaroufim @wconstab @ngimel @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire
aten::_linalg_solve_ex.result' is not currently implemented for the MPS
Labels: feature, triaged, module: mps

### 🐛 Describe the bug

I would like to request an implementation to fix "aten::_linalg_solve_ex.result' is not currently implemented for the MPS"

### Versions

aten::_linalg_solve_ex.result

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
Wrong results for GELU forward pass (CPU vs MPS) while  inferencing a GLPN model from huggingface
Labels: high priority, triaged, module: correctness (silent), module: mps

### 🐛 Describe the bug

Hi, there. I've discovered strange behaviour when using MPS device. When infer the same model with the same input on "mps" device, the result is numerically wrong and meaningless. 
@amyeroberts from huggingface narrowed down the scope of the problem to GELU layer, more information is here: https://github.com/huggingface/transformers/issues/22468

The issue makes it impossible to use GLPN architecture with MPS device.

Here are some examples:

MPS            |  CPU
:-------------------------:|:-------------------------:
![](https://user-images.githubusercontent.com/126671893/228804909-71194936-a266-4511-9ae0-1c667f762faa.png)  |  ![](https://user-images.githubusercontent.com/126671893/228804919-9916e01f-f4cd-43a2-be74-a7cb8eeb9c6f.png)

Thanks in advance for the help!

### Versions

PyTorch version: 2.1.0.dev20230329
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.3 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.6 (default, Mar 10 2023, 20:16:38)  [Clang 14.0.3 (clang-1403.0.22.14.1)] (64-bit runtime)
Python platform: macOS-13.3-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M2 Max

Versions of relevant libraries:
[pip3] numpy==1.24.2
[pip3] torch==2.1.0.dev20230329
[pip3] torchaudio==2.1.0.dev20230329
[pip3] torchvision==0.16.0.dev20230329
[conda] Could not collect

cc @ezyang @gchanan @zou3519 @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
torch.jit.script + legacy executor mode has diff in some pattern
Labels: oncall: jit

### 🐛 Describe the bug

# there is output diff between origin and script

``` python
import torch
import os
os.environ['TORCH_JIT_DISABLE_NEW_EXECUTOR'] = '1'
torch._C._jit_set_nvfuser_enabled(False)
class TempModule(torch.nn.Module):
  def __init__(self):
    super().__init__()
    self.head_size = 64
  def forward(self, x):
    score_scale1 = torch.rsqrt(torch.tensor(self.head_size) * 3)
    return score_scale1

module = TempModule()
x = module(torch.randn(4, 5))
print(x) # print 0.0722
ts_module = torch.jit.script(module)
y = ts_module(torch.randn(4, 5))
print(y) # print 0, unexpectly
```

### Versions

torch==2.0.0

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Add a deterministic version of reflection_pad2d_backward_cuda
Labels: module: nn, triaged, enhancement, module: determinism, actionable

Running ResViT model in deterministic mode i face

UserWarning: reflection_pad2d_backward_cuda does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True, warn_only=True)'. You can file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation. (Triggered internally at ../aten/src/ATen/Context.cpp:82.)

pytorch version 1.13.1

Thank you!


### Alternatives

_No response_

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @kurtamohler
AssertionError: was expecting embedding dimension of 22, but got 1320
Labels: oncall: transformer/mha

### 🐛 Describe the bug

I am new to pytorch on Colab. I used Transformer encoder as feature extractor to build DQN. The input shape of DQN is (60,22), and the output is one of the three numbers of 0,1,2. model.learn completed successfully. But model.predict is wrong. How can I change it?

code:
class TransformerFeaturesExtractor(BaseFeaturesExtractor):
    def __init__(self, observation_space, features_dim=128):
        super(TransformerFeaturesExtractor, self).__init__(observation_space, features_dim)

        self.transformer_encoder = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d_model=features_dim, nhead=2), num_layers=6
        )
        self.flatten = nn.Flatten()

    def forward(self, observations):
        x = self.flatten(observations)
        x = self.transformer_encoder(x.unsqueeze(0))
        return x.squeeze(0)

policy_kwargs = dict(
    features_extractor_class=TransformerFeaturesExtractor,
    features_extractor_kwargs=dict(features_dim=22), #, action_space=env.action_space),
)

# Create an instance of the DQN agent
model = DQN("MlpPolicy", env,policy_kwargs=policy_kwargs, verbose=1)
#model = DQN('MlpPolicy', stock_trade_env, verbose=1)
#model = PPO('MlpPolicy', env, verbose=1)
# Train the agent
# pdb.set_trace()
model.learn(total_timesteps=10_000, progress_bar=True)

obs = env.reset()
for i in range(100):
    action, _states = model.predict(obs)
    #action = env.action_space.sample()
    obs, rewards, dones, info = env.step(action)
    if dones:
        break
    env.render()



### Versions

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-8-b1d23f2bc043> in <cell line: 366>()
    365 obs = env.reset()
    366 for i in range(100):
--> 367     action, _states = model.predict(obs)
    368     #action = env.action_space.sample()
    369     obs, rewards, dones, info = env.step(action)

16 frames
/usr/local/lib/python3.9/dist-packages/torch/nn/functional.py in multi_head_attention_forward(query, key, value, embed_dim_to_check, num_heads, in_proj_weight, in_proj_bias, bias_k, bias_v, add_zero_attn, dropout_p, out_proj_weight, out_proj_bias, training, key_padding_mask, need_weights, attn_mask, use_separate_proj_weight, q_proj_weight, k_proj_weight, v_proj_weight, static_k, static_v, average_attn_weights)
   5024             raise AssertionError(
   5025                 "only bool and floating types of key_padding_mask are supported")
-> 5026     assert embed_dim == embed_dim_to_check, \
   5027         f"was expecting embedding dimension of {embed_dim_to_check}, but got {embed_dim}"
   5028     if isinstance(embed_dim, torch.Tensor):

AssertionError: was expecting embedding dimension of 22, but got 1320

cc @jbschlosser @bhosmer @cpuhrsch @erichan1
torch.nn.init functions with `generator` argument
Labels: module: nn, triaged, module: random, actionable

### 🚀 The feature, motivation and pitch

I might have a custom random generator (`torch.Generator`). All the low-level random functions support a `generator` argument for this. All the functions in `torch.nn.init` just call the low-level functions.

It would be useful if I can pass `generator` to those `torch.nn.init` functions and it would just pass it on to the low-level functions.

In my case, I'm mostly interested in `torch.nn.init.trunc_normal_`, as there is no such low-level `trunc_normal_` function.

### Alternatives

For my use case, it would also be fine if you add a `Tensor.trunc_normal_` low-level function, next to the `Tensor.normal_` function.

However, I think having `generator` as an argument for the `torch.nn.init` functions can anyway be helpful.

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @pbelevich
how can i load seperate pytorch_model.bin?
Labels: module: serialization, triaged

If you have a question or would like help and support, please ask at our
[forums](https://discuss.pytorch.org/).

If you are submitting a feature request, please preface the title with [feature request].
If you are submitting a bug report, please fill in the following details.

## Issue description

Provide a short description.

## Code example

Please try to provide a minimal example to repro the bug.
Error messages and stack traces are also helpful.

## System Info
Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
```

- PyTorch or Caffe2:
- How you installed PyTorch (conda, pip, source):
- Build command you used (if compiling from source):
- OS:
- PyTorch version: 1.13.1
- Python version: 3.8
- CUDA/cuDNN version:
- GPU models and configuration:
- GCC version (if compiling from source):
- CMake version:
- Versions of any other relevant libraries:

I have a separate binary file (pytorch_model-00001-of-00002.bin, pytorch_model-00002-of-00002.bin, pytorch_model.bin.index.json). 
When I substitute path into torch.load, I get an error.
How can I get this into torch.load? 


cc @mruberry
The operator 'aten::_weight_norm_interface' is not currently implemented for the MPS device.
Labels: feature, triaged, module: mps

### 🐛 Describe the bug

NotImplementedError: The operator 'aten::_weight_norm_interface' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764. As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.


### Versions

NotImplementedError: The operator 'aten::_weight_norm_interface' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764. As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.


cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
forward AD implimentation : _scaled_dot_product_efficient_attention 
Labels: triaged, module: forward ad

### 🚀 The feature, motivation and pitch

Hi there,

I encountered an error message that requests me to file an issue regarding a feature implementation. The error message is as follows:

NotImplementedError: Trying to use forward AD with _scaled_dot_product_efficient_attention that does not support it because it has not been implemented yet.
Please file an issue to PyTorch at https://github.com/pytorch/pytorch/issues/new?template=feature-request.yml so that we can prioritize its implementation.
Note that forward AD support for some operators require PyTorch to be built with TorchScript and for JIT to be enabled. If the environment var PYTORCH_JIT=0 is set or if the 
library is not built with TorchScript, some operators may no longer be used with forward AD.

I would appreciate it if you could prioritize the implementation of this feature. Thank you for your help.

### Alternatives

_No response_

### Additional context

I ran forward AD of Stable-Diffusion with diffusers library, dtype = torch.float32, device=cuda.
Request to cherrypick a fix into v1.13.1 (v1.8 has a CVE)
Labels: oncall: binaries, triaged

### 🐛 Describe the bug

We ran into a CVE on v1.8, details at https://nvd.nist.gov/vuln/detail/CVE-2022-45907. The CVE got fixed in v1.13.1 but this version has a bazel bug which stops us from using it: https://github.com/pytorch/pytorch/issues/92096#issuecomment-1492771044. 

Since, 1.8 -> 2.0.0 is a major version upgrade (which is a risk for us), the request is to cherrypick the fix for the bazel issue to 1.13: Fix is https://github.com/pytorch/pytorch/pull/92122. 

### Versions

N/A

cc @seemethere @malfet
GroupNorm cpu/gpu parity tests fail with pretty large differences
Labels: module: nn, triaged, needs research

### 🐛 Describe the bug

Differences look too big to be tolerance errors,  investigate why this is the case. Skipping the test for now but filing this issue to track

```
 =================================== FAILURES ===================================
2023-03-31T17:42:23.8945917Z _________ TestModuleCUDA.test_cpu_gpu_parity_nn_GroupNorm_cuda_float32 _________
2023-03-31T17:42:23.8946180Z Traceback (most recent call last):
2023-03-31T17:42:23.8946431Z   File "test_modules.py", line 578, in test_cpu_gpu_parity
2023-03-31T17:42:23.8946690Z     check_backward(cpu_outputs, gpu_outputs)
2023-03-31T17:42:23.8946944Z   File "test_modules.py", line 570, in check_backward
2023-03-31T17:42:23.8947185Z     self.assertEqual(cpu_p.grad, gpu_p.grad)
2023-03-31T17:42:23.8947635Z   File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/testing/_internal/common_utils.py", line 3023, in assertEqual
2023-03-31T17:42:23.8947942Z     raise error_metas[0].to_error(
2023-03-31T17:42:23.8948212Z AssertionError: Tensor-likes are not close!
2023-03-31T17:42:23.8948359Z 
2023-03-31T17:42:23.8948451Z Mismatched elements: 6 / 6 (100.0%)
2023-03-31T17:42:23.8948716Z Greatest absolute difference: 9.029479503631592 at index (4,) (up to 0.05 allowed)
2023-03-31T17:42:23.8949112Z Greatest relative difference: 1.9686480656500926 at index (2,) (up to 1.3e-06 allowed)
```

### Versions

master

cc @albanD @mruberry @jbschlosser @walterddr
[dynamo] hf_Reformer's graph break has increased
Labels: low priority, triaged, oncall: pt2, module: dynamo

https://github.com/pytorch/pytorch/pull/98003 updated the torchbench version that CI uses. While it does pick up a transformers upstream change which fixes graph break in Bert, it increase hf_Reformer's graph break from 5 to 26 for inference and from 51 to 67 for training. 

cc: @ngimel 

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @soumith @ngimel
Is there a recommended implementation of yuv2RGB for the current torch?
Labels: module: onnx, module: nn, triaged, module: python frontend

### 🚀 The feature, motivation and pitch

I need to convert the YUV420SP image to RGB, and then I built a network by torch. However, when I turned to the onnx frame model at last, I found it was very large. Is there a good way to do this?

![YUV420SP2RGB_BY_TORCH](https://user-images.githubusercontent.com/37217594/229105058-f548d1ec-445c-4437-934b-c6ea53ec277c.png)



### Alternatives

_No response_

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
Add PrivateUse1 folder in aten/src/ATen
Labels: triaged, module: backend

### 🚀 The feature, motivation and pitch

I am going to add a PrivateUse1 folder to contain all the feature adaptations for PrivateUse1 under Aten,For example GetGeneratorPrivate which is used for the three-party backend to register his own Generator implementation.This makes it easier for us to centrally manage these features, and it will increase the convenience of adaptation for different back-end manufacturers.This is an immature idea, please give your valuable advice.

### Alternatives

_No response_

### Additional context

_No response_
Request custom backend device memory Allocator.
Labels: module: memory usage, triaged, module: backend, module: CUDACachingAllocator

### 🚀 The feature, motivation and pitch

For newly registered backend users, the framework is expected to support a device lining pool mechanism. Users only need to register malloc and free functions related to devices to reduce the development cost of the new backend.

### Alternatives

I think the support for this feature can be divided into the following steps
1. Currently, the cachingAllocator mechanism of the CUDA is mature. We can extract classes from the cachingAllocator mechanism and provide them for users to inherit during implementation, such as DeviceStats, Block, BlockInfo, THCcachingAllocator and DeviceCachingAllocator.
2. We can pre-register a device CachingAllocator for the privateuse1 backend. New users only need to register the free and malloc functions of their backend.
3. The CachingAllocator policy can be replaced and extended for capable users.

### Additional context

_No response_
Module 'Sequential' has no attribute '_modules' :
Labels: oncall: jit

### 🐛 Describe the bug
```pythons
import torch
import torch.nn as nn
class MyModule(nn.Module):
    def __init__(self):

        super(MyModule, self).__init__()

        self.seq = nn.Sequential(
            nn.Linear(10, 20),
            nn.ReLU(),
            nn.Linear(20, 5)
        )

def forward(self, x):
    return self.seq(x)

my_module = MyModule()
scripted_module = torch.jit.script(my_module)
```

```
RuntimeError: 
Module 'Sequential' has no attribute '_modules' :
  File "/home/zhou/.local/lib/python3.10/site-packages/torch/nn/modules/container.py", line 251
    @property
    def modules(self):
        return self._modules
               ~~~~~~~~~~~~~ <--- HERE

```


### Versions

__version__ = '2.0.0+cu117'
debug = False
cuda = '11.7'
git_version = 'c263bd43e8e8502d4726643bc6fd046f0130ac0e'
hip = None


cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
DISABLED test_scatter_1d (__main__.DeviceMeshCollectiveTest)
Labels: oncall: distributed, module: flaky-tests, skipped

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_scatter_1d&suite=DeviceMeshCollectiveTest) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/12412570135).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 3 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_scatter_1d`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `distributed/_tensor/test_device_mesh.py` or `distributed/_tensor/test_device_mesh.py`

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
functional collective should respect the whole mesh
Labels: oncall: distributed, triaged

### 🚀 The feature, motivation and pitch

See discussion of this PR, when passing the whole mesh to the functional collective (without meshdim), it should be able to see the whole mesh ranks and run collective on all the ranks, instead of only 0 mesh dim.

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Relax version dependencies on CUDA pip wheels?
Labels: oncall: binaries, module: cuda

### 🐛 Describe the bug

A JAX user noted that `torch` and `jax` have conflicting requirements for NVIDIA's CUDA pip wheels.

Torch specifies exact version pins in its wheels:
```
Requires-Dist: nvidia-cuda-nvrtc-cu11 (==11.7.99) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cuda-runtime-cu11 (==11.7.99) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cuda-cupti-cu11 (==11.7.101) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cudnn-cu11 (==8.5.0.96) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cublas-cu11 (==11.10.3.66) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cufft-cu11 (==10.9.0.58) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-curand-cu11 (==10.2.10.91) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cusolver-cu11 (==11.4.0.1) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cusparse-cu11 (==11.7.4.91) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-nccl-cu11 (==2.14.3) ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-nvtx-cu11 (==11.7.91) ; platform_system == "Linux" and platform_machine == "x86_64"
```

These are non-overlapping with some of JAX's dependencies. JAX currently ships wheels that require:
* CUDA 11.8+ and cudnn 8.6+, or
* CUDA 12.0+ and cudnn 8.8+.

For CUDA 11 wheels, JAX requires:
```
          "nvidia-cublas-cu11>=11.11",
          "nvidia-cuda-nvcc-cu11>=11.8",
          "nvidia-cuda-runtime-cu11>=11.8",
          "nvidia-cudnn-cu11>=8.6",
          "nvidia-cufft-cu11>=10.9",
          "nvidia-cusolver-cu11>=11.4",
          "nvidia-cusparse-cu11>=11.7",
```

I'm wondering if you would be open to relaxing some or all of the CUDA version pins in `torch` to be `>=` constraints. NVIDIA promise backward compatibility for CUDA and CUDNN minor releases, so in theory this is safe. Theory and practice may differ, however.

My fall back is to tell users to do the following, which works because `pip` doesn't actually enforce the version constraints:
```
pip install torch
pip install --upgrade nvidia-cublas-cu11 nvidia-cuda-runtime-cu11 # ... etc.  
# then install JAX
```
but it's not optimal: it means one cannot really release a package that depends on both `jax` and `torch`.



### Versions

2.0

cc @seemethere @malfet @ngimel
Can libtorch be used for quantization-aware training of models?
Labels: oncall: quantization, triaged

### 🚀 The feature, motivation and pitch

We want to use C++ to develop a software for deep learning model training and quantification. Users can fine-tune and train the model by themselves. In order to facilitate deployment, we don't want to use python language, so we want to implement it based on libtorch.
However, libtorch does not seem to support quantization-aware training at present. Is it possible to add this function to libtorch?

### Alternatives

_No response_

### Additional context

_No response_

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel
Dynamo doesn't report accurate line numbers for <resume> in some situations
Labels: triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

Steps to reproduce:
1. Patch in https://github.com/pytorch/pytorch/pull/97884
2. Run `TORCH_LOGS=dynamo benchmarks/dynamo/huggingface.py --devices cuda --inductor   --accuracy  --print-graph-breaks --only  PegasusForConditionalGeneration`

Here are the full logs https://gist.github.com/ezyang/01254a7e14d17ad9f9843c22676cb1fc

In the logs, you will see:

```
[2023-03-30 05:27:57,270] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing file "<string>", line 3, in <resume in __init__>
[2023-03-30 05:27:57,270] torch._dynamo.output_graph: [WARNING] Graph break: call_method DataClassVariable() __setitem__ [ConstantVariable(str), ConstantVariable(NoneType)] {} from user code at:
  File "<string>", line 4, in <resume in __init__>

[2023-03-30 05:27:57,277] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing file "<string>", line 4, in <resume in __init__>
[2023-03-30 05:27:57,277] torch._dynamo.output_graph: [WARNING] Graph break: call_method DataClassVariable() __setitem__ [ConstantVariable(str), ConstantVariable(NoneType)] {} from user code at:
  File "<string>", line 5, in <resume in __init__>

[2023-03-30 05:27:57,284] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing file "<string>", line 5, in <resume in __init__>
[2023-03-30 05:27:57,284] torch._dynamo.output_graph: [WARNING] Graph break: call_method DataClassVariable() __setitem__ [ConstantVariable(str), ConstantVariable(NoneType)] {} from user code at:
  File "<string>", line 6, in <resume in __init__>

[2023-03-30 05:27:57,291] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing file "<string>", line 6, in <resume in __init__>
[2023-03-30 05:27:57,291] torch._dynamo.output_graph: [WARNING] Graph break: call_method DataClassVariable() __setitem__ [ConstantVariable(str), ConstantVariable(NoneType)] {} from user code at:
  File "<string>", line 7, in <resume in __init__>

[2023-03-30 05:27:57,298] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing file "<string>", line 7, in <resume in __init__>
[2023-03-30 05:27:57,298] torch._dynamo.output_graph: [WARNING] Graph break: call_method DataClassVariable() __setitem__ [ConstantVariable(str), TensorVariable()] {} from user code at:
  File "<string>", line 8, in <resume in __init__>
```

This is really bad. Really want real file line numbers here.

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire @jansel

### Versions

master
torch.randn signature is missing generator
Labels: module: docs, triaged, enhancement, module: random, module: python frontend

### 📚 The doc issue

[torch.randn](https://pytorch.org/docs/stable/generated/torch.randn.html) lacks `generator` argument in the signature `torch.randn(*size, *, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False, pin_memory=False)`.

### Suggest a potential alternative/fix

Add `generator`, like in [torch.rand](https://pytorch.org/docs/stable/generated/torch.rand.html#torch.rand): 
`torch.randn(*size, *, generator=None, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False, pin_memory=False)`

cc @svekars @carljparker @pbelevich @albanD
[CI/Infra] Record keeping: runner shutdown spike
Labels: triaged

### 🐛 Describe the bug

https://hud.pytorch.org/failure/The%20runner%20has%20received%20a%20shutdown%20signal.%20This%20can%20happen%20when%20the%20runner%20service%20is%20stopped%2C%20or%20a%20manually%20started%20runner%20is%20canceled. shows skyrocketing number of instances "runner shutdown" on Mar 29, 2023. Because they got self-disconnected via "sudo ./svc.sh stop" usually after main action step's failure. When the actions was already failing, there was logic to determine whether the error was "unrecoverable" or not. If the error was indeed unrecoverable, then disconnecting is good because it prevents pollution of future jobs. Though such disconnection should be aligned with the underlying infra (e.g. as it currently stands, a courtesy notification may be needed for GCP runners). If the error was not "unrecoverable", we should make sure the runner does not get disconnected from GHA (or maybe this does not matter, but developers would be redirected to runner cancellation error first).  

<img width="797" alt="image" src="https://user-images.githubusercontent.com/109318740/228754623-4524e2a1-2a7b-4450-8b0e-577972ec44bd.png">


We know the related PR (#97585), and a follow-up forward fix (#97929) ,. but I want to make sure we address the following issues: 
  
1) For GCP A100 runners that were affected, the error should not be categorized "unrecoverable", e.g. in this action https://github.com/pytorch/pytorch/actions/runs/4560162853/jobs/8045132304 , it was transient SSH error, should be recoverable. And for this https://github.com/pytorch/pytorch/actions/runs/4558849401/jobs/8042466592  
that was software caused core-dump and is non-deterministic. The runner should not be disconnected from the GHA. Though after the disconnection, manual intervention was needed as the auto-scaler does not yet have the knowledge that the still online busy runner decided to silently disconnect from GHA without telling auto-scaler (e.g. set /tmp/runner_status to "idle" for auto-scaler to shut it down) 
 
2) For AWS runners that were affected, we need to also make sure those instances go through "garbage collection" procedure so they do not become at large. cc @jeanschmidt 

3) If we allow runner self-disconnect, depending on how we implemented auto-scaler, we should make sure the auto-scaler get the opportunity to reclaim them and take care of the runners. 

cc @huydhn @seemethere @malfet 

### Versions

After #97585, maybe resolved by now, or maybe not. Needs confirmation.