gopikrsmscs/torch-issues huggingface dataset:
LayerNorm freeze processes using torch multiprocessing
Labels: module: multiprocessing, triaged

### 🐛 Describe the bug

LayerNorm operation is freezing my processes when i launch 1 or more process using torch multiprocessing. I did a trivial network only containing a layernorm and my process all freezes in the forward pass. I am on linux and i did not hve this problem on MacOS. The code is :

```
import sys, os

import torch
import torch.multiprocessing as mp




def worker(rank, model, input_action):
    """Worker function"""
    print(f"Worker {rank} received model")
    out = model(input_action)
    
#Create a model using torch layernorm
class LayerNormModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.layernorm = torch.nn.LayerNorm(4)
        
    def forward(self, x):
        return self.layernorm(x)


if __name__ == '__main__':
    # Set the multiprocessing start method
    #mp.set_start_method('spawn')
    num_threads = torch.get_num_threads()
    print("Number of threads:", num_threads)
    
    model = LayerNormModel()
    input_action = torch.randn(1, 2, 4)
    #Working properly
    test = model(input_action)
    
    torch.set_num_threads(5)
    num_threads = torch.get_num_threads()
    print("Number of threads:", num_threads)
    
    # Create a list of values
    values = [1, 2, 3, 4, 5]

    # Create a process for each value
    processes = []
    for i, value in enumerate(values):
        p = mp.Process(target=worker, args=(i, model, input_action))
        processes.append(p)
        p.start()

    # Wait for all processes to finish
    for p in processes:
        p.join()
        
    print("Done!")
```



### Versions

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==2.0.1
[pip3] triton==2.0.0
[conda] numpy                     1.24.3                   pypi_0    pypi
[conda] torch                     2.0.1                    pypi_0    pypi
[conda] triton                    2.0.0                    pypi_0    pypi

cc @VitalyFedyunin
Typing missing on arithmetic ops on `Tensor`
Labels: module: typing, triaged

### 🐛 Describe the bug

This is related to #103375 and #103376, but I assume it's better to split into smaller fixes.

Some of the dunder ops are not defined in `_C._TensorBase` but directly in `Tensor`:
https://github.com/pytorch/pytorch/blob/03101a227f6639d5a9ad628d1dc300f9f99a8812/torch/_tensor.py#L850-L902
However, as seen, there's no typing for these methods.

### Versions

master

cc @ezyang @malfet @rgommers @xuzhao9 @gramster
Inplace binary ops on tensor subclasses can cause mypy error
Labels: module: typing, triaged

### 🐛 Describe the bug

Using inplace binary ops on a subclass of `Tensor` will cause mypy error (e.g. `*=` shown below, same for other ops `+=` etc.)
```python
import torch
a = torch.nn.Parameter(torch.Tensor())
a *= 2
```
run `mypy`:
```
a.py:3: error: Result type of * incompatible in assignment
```
This is because of pyi definition `def __imul__(self, other: Any) -> Tensor: ...`, which means it always returns `Tensor` but not the subclass.

---

Although mypy does not give any error with `a.mul_(2)` instead of `a*=2`, they should be the same, and inplace methods like `mul_` should return `Self` instead of `Tensor`.
```python
import torch
a = torch.nn.Parameter(torch.Tensor())
print(a.mul_(2).__class__)
```
The result is indeed `torch.nn.parameter.Parameter`.

### Versions

master

cc @ezyang @malfet @rgommers @xuzhao9 @gramster
torchgen/gen_backend_stubs.py compatibility with DispatchStubs
Labels: triaged, module: dispatch, module: codegen, module: structured kernels

### 🚀 The feature, motivation and pitch

For our out-of-tree backend, I would like to support many structured kernels similar to how is done in CUDA, i.e. registering `DispatchStub` per operation, similar to how is done [here](https://github.com/pytorch/pytorch/blob/03101a227f6639d5a9ad628d1dc300f9f99a8812/aten/src/ATen/native/cuda/BinaryMulKernel.cu#L46), for `PrivateUse1`, the support for registering stubs was added in [this PR](https://github.com/pytorch/pytorch/pull/99611). 

This PR however, does not tackle how to implement structured kernels in the same manner. It would be nice if [`get_backend_stubs.py`](https://github.com/pytorch/pytorch/blob/03101a227f6639d5a9ad628d1dc300f9f99a8812/torchgen/native_function_generation.py) could support implementation paths of this form instead.

I'm not certain as to whether `torchgen` could support the reuse of these kernels with overridden headers, which is how we've implemented a method to redirect CUDA kernel launches from our backend successfully with various unstructured kernels.

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @bhosmer @bdhirsh
RuntimeError: CUDA error: unknown error
Labels: oncall: distributed, triaged, module: fsdp

### 🐛 Describe the bug

![image](https://github.com/pytorch/pytorch/assets/39661319/da0f9a22-de47-4942-8319-d279c6a682a6)


### Versions

RuntimeError: CUDA error: unknown error
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 8193) of binary: /home/win10-ubuntu/anaconda3/envs/vicuna-7b/bin/python3.10
Traceback (most recent call last):
  File "/home/win10-ubuntu/anaconda3/envs/vicuna-7b/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/win10-ubuntu/anaconda3/envs/vicuna-7b/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/win10-ubuntu/anaconda3/envs/vicuna-7b/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/home/win10-ubuntu/anaconda3/envs/vicuna-7b/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/win10-ubuntu/anaconda3/envs/vicuna-7b/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/win10-ubuntu/anaconda3/envs/vicuna-7b/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
fastchat/train/train_mem.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-06-10_20:45:09
  host      : DESKTOP-LA7GLEG.localdomain
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 8193)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
![image](https://github.com/pytorch/pytorch/assets/39661319/ab44828a-414b-4fe6-99ca-28c0f0fa2253)


cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Add a requirements.txt for windows pip packages
Labels: triaged, module: devx

**Goal**: Enable pytorch devs to add/update pip packages for windows builds and tests in a single PR

### Context
Today, if you're introducing a PR that takes a dependency on a new python package, for Windows you need to:
1. Add a manual, temporary pip install somewhere in the workflow so that your PR passes
2. Create a test-infra PR that adds that pip package to the AMI here: https://github.com/pytorch/test-infra/blob/main/aws/ami/windows/scripts/Installers/Install-Pip-Dependencies.ps1
3. Remove the manual pip install line from step 1

That's pretty tedious

### The Ask
Instead we should do the following:

1. Define a dedicated requirements.txt file for windows in python/python, following the pattern used for https://github.com/pytorch/pytorch/blob/main/.github/requirements/pip-requirements-macOS.txt
3. In that [ps1 file](https://github.com/pytorch/test-infra/blob/main/aws/ami/windows/scripts/Installers/Install-Pip-Dependencies.ps1), download the req.txt file from the prev step (you'll need to get it from the py/py repo) and install the packages it specifies. 
4. In the windows build & test workflows, we also always install packages from the requirements.txt file if they’re not already installed.

### Benefits
1. This would make future installations/upgrades easy by putting all the package changes in a single folder
2. We have dependencies defined in one place 
3. We have safe guard in case the dependencies are not found in the AMI (which is expected for a small time window when a new package/upgrade is freshly added)

cc @kit1980 @huydhn @clee2000
[feature request] Native method for iterating Python items of tensors: `iteritems()` and a new `tensor.item(i, j, k, ...)` method
Labels: feature, triaged, module: python frontend

### 🚀 The feature, motivation and pitch

without having to call `.item()`. OP: https://github.com/pytorch/pytorch/pull/103339#discussion_r1224826554

so this is needed for 1d tensors, although could be useful in the future in other contexts if string arrays are supported: https://github.com/pytorch/pytorch/issues/101699

Only relevant for large tensors/loops, where materializing a python list first takes too many python objects

Related on slow item indexing: https://github.com/pytorch/pytorch/issues/29973 and proposal of `tensor.item(i, j, k, ...)` fast indexing method returning Python int/float objects without `tensor[i, j, k, ...].item()` first creating an extra tensor object and only then upacking it

Related on supporting `memoryview` on tensors: https://github.com/pytorch/pytorch/issues/43949, then this method could be implemented by simply returning memoryview which supports iteration in python

### Alternatives

_No response_

### Additional context

_No response_

cc @albanD
mps and cpu give far different results when training a transformer.
Labels: triaged, module: mps

### 🐛 Describe the bug

when training a transfromer model with mac CPU this is the resulting loss function log
994,tensor(13.4385),2.598125166893005,2.676572799682617
1988,tensor(13.2934),2.5872674012184143,2.6137495040893555
2000,tensor(13.3104),2.5885423374176026,2.628082513809204
2982,tensor(12.8851),2.5560683608055115,2.675567865371704
3976,tensor(12.6861),2.54050742149353,2.4734363555908203
4000,tensor(12.8042),2.549770863056183,2.584007501602173
4970,tensor(12.6474),2.537447814941406,2.528282403945923


when using the MPS, same code and data
994,tensor(1.4541),0.3743850642442703,0.42320576310157776
1988,tensor(1.4568),0.3762083804607391,0.3572617173194885
2000,tensor(1.4540),0.37433584868907926,0.3778816759586334
2982,tensor(1.4524),0.3732233664393425,0.3656735122203827
3976,tensor(1.4476),0.36990807741880416,0.3881590962409973
4000,tensor(1.4447),0.36788938045501707,0.3148590922355652
4970,tensor(1.4531),0.373729208111763,0.3972059488296509



[test.zip](https://github.com/pytorch/pytorch/files/11710031/test.zip)


### Versions

Collecting environment information...
PyTorch version: 2.1.0.dev20230608
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.4 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: Could not collect
Libc version: N/A

Python version: 3.11.3 (main, Apr 19 2023, 18:49:55) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-13.4-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M2 Pro

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==2.1.0.dev20230608
[pip3] torchaudio==2.1.0.dev20230608
[pip3] torchvision==0.16.0.dev20230608
[conda] numpy                     1.24.3          py311hb57d4eb_0  
[conda] numpy-base                1.24.3          py311h1d85a46_0  
[conda] pytorch                   2.1.0.dev20230608        py3.11_0    pytorch-nightly
[conda] torchaudio                2.1.0.dev20230608       py311_cpu    pytorch-nightly
[conda] torchvision               0.16.0.dev20230608       py311_cpu    pytorch-nightly

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
python test/inductor/test_split_cat_fx_passes.py -k test_consecutive_split_merge fails, but running all tests together succeeds
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

Something wrong with the harness. I plan to disable these tests for now.

cc @msaroufim @wconstab @bdhirsh @anijain2305 @devashishshankar 

### Versions

main
Improve `_group_tensors_by_device_and_dtype`
Labels: module: optimizer, triaged, better-engineering, actionable, module: mta

follow-up https://github.com/pytorch/pytorch/pull/100007

https://github.com/pytorch/pytorch/blob/6fa2d41dc7dfcbff37df7fb5517e6644eb3d74ab/torch/csrc/Module.cpp#L1745-L1746 should be able to be cleaner by implementing an appropriate type caster for `at::ScalarType`

cc @vincentqb @jbschlosser @albanD @janeyx99 @mcarilli
RuntimeError: torch.vmap a function that includes in-place arithmetic operations on a zero-initialized tensor, an error "vmap: inplace arithmetic(self, *extra_args) is not possible" is raised.
Labels: triaged, module: functorch

### 🐛 Describe the bug

When using torch.vmap to batch-process a function that includes inplace arithmetic operations on a zero-initialized tensor, an error "vmap: inplace arithmetic(self, *extra_args) is not possible" is raised.
Specifically, an error is raised when initializing the zero tensor using torch.zeros(x.shape), but no error is raised if using torch.zeros_like(x). 


```
import torch

def func(x, y):
    mat = torch.zeros(x.shape)  # Error occurs on this line
    # mat = torch.zeros_like(x)  # No error occurs on this line
    mat[0, 0] = mat[0, 1] + x[0, 0]
    return mat

input = torch.ones((10, 5, 6))
inputy = torch.ones((10, 5, 6))

batched_func = torch.vmap(func, in_dims=(0, 0))
batched_func(input, inputy)

```

```
Error message:
RuntimeError: vmap: inplace arithmetic(self, *extra_args) is not possible because there exists a Tensor `other` in extra_args that has more elements than `self`. This happened due to `other` being vmapped over but `self` not being vmapped over in a vmap. Please try to use out-of-place operators instead of inplace arithmetic. If said operator is being called inside the PyTorch framework, please file a bug report instead
```

### Versions

PyTorch version: 2.0.1
Operating system: Mac Monterey 12.6.2
Python version: 3.10.4

cc @zou3519 @Chillee @samdow @kshitij12345 @janeyx99
Disabling ALL TestOptim on the dynamo config
Labels: high priority, module: optimizer, triaged, skipped, module: dynamo

### 🐛 Describe the bug

https://github.com/pytorch/pytorch/pull/102640 introduced flakiness into the dynamo optim tests.

There was a followup PR https://github.com/pytorch/pytorch/pull/103066 to try to disable the tests in attempt to restore health, but that did not sufficiently cover all forms of flakiness as there were further reports of flaky tests after. 

This issue tracks the fact that we currently disable ALL test_optim tests in the dynamo shard, which is probably undesirable, but is a necessary stopgap to restore CI health. The alternative is reverting https://github.com/pytorch/pytorch/pull/102640 and its forward fixes, like https://github.com/pytorch/pytorch/pull/103121. 

### Versions

main

cc @ezyang @gchanan @zou3519 @vincentqb @jbschlosser @albanD @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @aakhundov
binary_cross_entropy (loss) seems to be giving incorrect values for very negative logits
Labels: module: nn, triaged

### 🐛 Describe the bug

Please see the following code which computes the loss manually (using PyTorch ops) and using the `F.binary_cross_entropy` API.

```
import torch
from torch import nn
from torch.nn import functional as F

x = nn.Parameter(torch.tensor([-50., -10., -5., -2., 0., 2., 5., 10.]), requires_grad=True)

optimizer = torch.optim.Adam([x])
optimizer.zero_grad()

xs = x.sigmoid()
loss = -xs.log().mean()

loss.backward()
print(x.grad)

optimizer.zero_grad()
loss = F.binary_cross_entropy(x.sigmoid(), torch.ones_like(x))
loss.backward()
print(x.grad)
```

This code prints.

```
tensor([-1.2500e-01, -1.2499e-01, -1.2416e-01, -1.1010e-01, -6.2500e-02,
        -1.4900e-02, -8.3660e-04, -5.6773e-06])
tensor([-2.4109e-11, -1.2499e-01, -1.2416e-01, -1.1010e-01, -6.2500e-02,
        -1.4900e-02, -8.3660e-04, -5.6773e-06])
```

Note that for the input `-50`, the `binary_cross_entropy` API returns a very small negative value `-2.4109e-11`, instead of a larger negative value, namely `-1.2500e-01`.


### Versions

1.12.1

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
Add Half support for softmax and log_softmax on CPU
Labels: module: cpu, open source, ciflow/trunk, release notes: nn, ciflow/periodic, ciflow/mps, module: inductor

Add Half support for softmax and log_softmax on CPU.
Note: This introduces a correctness issue with MPS https://github.com/pytorch/pytorch/issues/111416 and https://github.com/pytorch/pytorch/issues/111479.

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @voznesenskym @penguinwu @EikanWang @Guobing-Chen @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @ngimel
Fast kernels for low rank matrix multiplication
Labels: triaged, module: linear algebra

### 🚀 The feature, motivation and pitch

I'm working on the robotics perception field and we have been exploring to use pytorch for production more on the control side where the most common used primitives are camera poses, projections, quaternion arithmetic, which in the real-time production side can be identified by a subset of matmul operators of sizes of 3x1, 4x1, 3x3, 4x4.

With some colleagues we put together in kornia a small package for [Lie Algebra](https://github.com/kornia/kornia/tree/master/kornia/geometry/liegroup), Quaternion, etc based on a colab with the [Sophus](https://github.com/strasdat/Sophus) team. We did some internal benchmarks and ended up turning down the kornia/pytorch implementation because of slowness for small matrix multiplications operators.

Sophus is based on c++ Eigen, however, in parallel I did a quick public [benchmark](https://discuss.pytorch.org/t/matmul-slow-for-small-matrices/168425) comparing pytorch/numpy and (unless something missing; and happy to expand/improve) numpy is still a clear winner that prevents using pytorch in real time production robotics platforms.

### Alternatives

Alternatives I'm thinking to solve my problem:
- Numba (which removes pytorch out of the equation)
- Custom c++/cuda kernels to mimic Eigen (which in kornia we don't have bandwidth to support but happy to contribute (with guidance) to pytorch core.
- Custom Python/Triton kernels (which we can experiment/maintain from kornia side)

### Additional context

Just screenshot of the [benchmark](https://discuss.pytorch.org/t/matmul-slow-for-small-matrices/168425) mentioned above

![image](https://github.com/pytorch/pytorch/assets/5157099/2cadc328-19ab-430e-b621-1753920c9855)


cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano
DTensor uneven sharding corner cases.
Labels: oncall: distributed, triaged

### 🐛 Describe the bug

While debugging activation checkpointing's unit I found one interesting corner case.

For example:
```
x = torch.rand(5, 10)
y = DTensor.from_local(x).redistribute(device_mesh=mesh, placements=[Shard(0)]).redistribute(device_mesh=mesh, placements=[Replicate()]).to_local()

y.size is not equal to x.
```

### Versions

If world size = 4, y sizes are:
```
y size:  torch.Size([8, 10]) (rank 1)
y size:  torch.Size([8, 10]) (rank 2)
y size:  torch.Size([8, 10]) (rank 0)
y size:  torch.Size([8, 10]) (rank 3)
```

So this is a bug of uneven sharding.

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
distributed.gather shape constraints
Labels: oncall: distributed, triaged, topic: docs

### 📚 The doc issue

For distributed.gather(tensor, tensorlist, **kwargs) with backend NCCL
It is not documented what are the shape constraints for gather tensorlist.
I thought for the destination call, the tensors in tensor_list must simply be of the size of the tensor input of each corresponding rank,
but this is not the case, 
I had an incorrect shape error with on rank 0 tensor of shape (70, 1), on rank 1 tensor of shape (76, 1) 
and a tensor_list on rank 1 with in position 0 a tensor of shape (70, 1), in position 1 a tensor of shape (76, 1)
After searching for the cause of the error, I found a forum message on a somewhat similar issue with all_gather that stated the shape of the tensor in tensor_list must all be equal ? Is that correct ? Or non-increasing (so that last tensor can have less than full size) ? I worked around the issue using asynchronous send and receive instead of a gather, but this could do with better specification.


### Suggest a potential alternative/fix

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Dynamo trouble shooting dead link
Labels: good first issue, triaged, topic: docs, oncall: pt2, module: dynamo

### 📚 The doc issue

The dynamo page https://pytorch.org/docs/stable/dynamo/troubleshooting.html#accuracy-debugging link to TROUBLESHOOTING results in a dead link for me.

### Suggest a potential alternative/fix

_No response_

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @aakhundov
Misaligned address error with torch.cat
Labels: high priority, triaged, oncall: pt2, module: inductor

### 🐛 Describe the bug

The following repro raises "RuntimeError: Triton Error [CUDA]: misaligned address".

https://gist.github.com/zou3519/0de4a64f5f612531133d04a8f59400eb

### Error logs

_No response_

### Minified repro

_No response_

### Versions

main, A100

cc @ezyang @gchanan @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @davidberard98
Warn / deprecate / remove ProcessGroupNCCL._group_start(), _group_end() APIs
Labels: oncall: distributed, triaged

### 🐛 Describe the bug

As reported by @lw, these APIs are error prone and can easily result in correctness issues if the appropriate synchronization is not done by the user. As a result, we should discuss whether to remove these APIs (and just use the `coalescing_manager`), or add a warning saying users have to do explicit sync.

### Versions

main

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Issue-103101:  Refactor dimensionality check in tuned_mm_plus_mm to pattern matching phase.
Labels: triaged, open source, module: inductor, ciflow/inductor

Fixes #103101

I am a new contributor and this is my first attempt at solving the issue. Looking forward to feedback.

Thanks,
Sid

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @aakhundov
Image Processing with Pytorch
Labels: triaged

### 🚀 The feature, motivation and pitch

Hello, I would like to use it in Image Processing for calculating the similarity between two photos. I am thinking Pytorch is really well for this process. But I couldn't see any issue therefore I would like to open that issue. I would like to submit a code as an example of usage for PyTorch in this area. And I would like to know how and where can I do this.

### Alternatives

_No response_

### Additional context

My codes calculate the similarity between the reference image and the target image. For the data, I got some of the example pictures but the photo could change, it is necessary.
Benchmark --quick with huggingface runs almost indefinitely on CPU
Labels: triaged, oncall: pt2

Minimal repro is the same as triggering a run on `Albert`.
```
python benchmarks/dynamo/huggingface.py --performance --float32 -dcpu --output=tmp.csv --inference -n5 --inductor  --no-skip --dashboard -k Albert --batch-size 1
```

Not a bug per se, it might be just that slow for CPU. However it does hinder --quick sanity check on cpu.

CPU: Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
[help] did torch.distributed.launch can be applied on k8s cluster with pytorch-operator
Labels: oncall: distributed, module: elastic

### 🐛 Describe the bug

Hi, i have been successfuly in starting distributed training via torch.distributed.launch on baremetal servers.
But recently, i need to using k8s cluster, and "apply -f operator.yaml" on kubernetes is recommanded .
I wonder does pytorch-operator or other training-operator support torch.distributed.launch ?
because pytorch-operator does not import args like "ADDR_MASTER" or "NODE_RANK" etc to starts distributed train, but these args should be manually initialized to input to torch.distributed.launch, which really confuses me.
Thank you in advance~ any hints would be helpful to me

### Versions

kubernetes cluster

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @dzhulgakov
Undeterministic behavior in testing in dynamo.
Labels: triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

TorchDynamo uses object ID to figure out which functions/modules to be traced (https://github.com/pytorch/pytorch/blob/main/torch/_dynamo/allowed_functions.py#L283). 
This is problematic when you are running unittests (python test/dynamo/test_misc.py) where one unittest uses allow_in_graph API which directly adds entry to this global allowed_functions tracker. (https://github.com/pytorch/pytorch/blob/main/torch/_dynamo/__init__.py#L90). The reason is once the test that uses allow_in_graph finishes, we don't delete the object id from it directly. As a result, another test that uses different object with same object id will be assumed to be in this allowlist incorrectly. 

### Versions

main

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @aakhundov
breakpoint() in torch.compile region behaves oddly
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

I expect breakpoint() to induce a graph break and let me inspect the frame

```
import torch

@torch.compile(backend="eager")
def f(x):
    breakpoint()
    return x + 1

f(torch.randn(1))
```

However, when I run this, I actually get dropped in some weird, synthetic catch_errors frame

```
(/home/ezyang/local/a/pytorch-env) [ezyang@devgpu019.ftw1 ~/local/a/pytorch (f15e82f7)]$ python z.py
--Call--
> /data/users/ezyang/a/pytorch/torch/_dynamo/eval_frame.py(407)catch_errors()
-> @functools.wraps(callback)
(Pdb) bt
  /data/users/ezyang/a/pytorch/z.py(8)<module>()
-> f(torch.randn(1))
  /data/users/ezyang/a/pytorch/torch/_dynamo/eval_frame.py(286)_fn()
-> return fn(*args, **kwargs)
  /data/users/ezyang/a/pytorch/z.py(5)f()
-> breakpoint()
> /data/users/ezyang/a/pytorch/torch/_dynamo/eval_frame.py(407)catch_errors()
-> @functools.wraps(callback)
```

I can get to the frame I care about by typing `up`, but this threw me for a loop the first time it happened to me.

### Versions

main

cc @msaroufim @wconstab @bdhirsh @anijain2305
Allow overriding __repr__ to call dataclass_repr (infinite recursion right now)
Labels: triaged, better-engineering, module: codegen

### 🐛 Describe the bug

dataclass_repr is great and I want it as the default for most complicated dataclasses I define. However, I cannot actually define `__repr__` to call into `dataclass_repr` because this triggers an infinite loop.

### Versions

master

cc @bhosmer @bdhirsh
Optimize the copy of Half to Float and Float to Half on CPU
Labels: module: cpu, open source, ciflow/trunk, topic: not user facing, ciflow/periodic

### Description
Optimize the copy of Half to Float and Float to Half on CPU.

### Testing

Single core:
shape | fp16 -> fp32 / ms | fp32 -> fp16 / ms | bf16 -> fp32 / ms | fp32 -> bf16 / ms
-- | -- | -- | -- | --
size: (1, 777) | 0.00345 | 0.00344 | 0.00411 | 0.00410
size: (2, 512) | 0.00355 | 0.00344 | 0.00431 | 0.00400
size: (10, 555) | 0.00473 | 0.00391 | 0.00562 | 0.00477
size: (1, 2048, 1024) | 0.488 | 0.480 | 0.498 | 0.499
size: (32, 100, 777) | 0.584 | 0.568 | 0.571 | 0.587

28 cores:
shape | fp16 -> fp32 / ms | fp32 -> fp16 / ms | bf16 -> fp32 / ms | fp32 -> bf16 / ms
-- | -- | -- | -- | --
size: (10, 555) |  0.00472 | 0.00369 | 0.00576 |  0.00481
size: (1, 2048, 1024) |  0.0189 | 0.0188 | 0.0173 | 0.0251
size: (64, 512, 1024) | 3.159 | 2.375 |  3.152 | 2.358
size: (32, 100, 777) | 0.0225 | 0.0195 | 0.0193 | 0.0261

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
error: ‘aligned_alloc’ was not declared in this scope        static_cast<char*>(aligned_alloc(FLATBUFFERS_MAX_ALIGNMENT, size)), free);
Labels: module: build, triaged

### 🐛 Describe the bug

[100%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/init_flatbuffer_module.cpp.o
/data/meda_home/hongbin/pytorch/torch/csrc/init_flatbuffer_module.cpp: In function ‘std::shared_ptr<char> copyStr(const string&)’:
/data/meda_home/hongbin/pytorch/torch/csrc/init_flatbuffer_module.cpp:37:26: error: ‘aligned_alloc’ was not declared in this scope
       static_cast<char*>(aligned_alloc(FLATBUFFERS_MAX_ALIGNMENT, size)), free);
                          ^~~~~~~~~~~~~
gmake[2]: *** [caffe2/torch/CMakeFiles/torch_python.dir/build.make:1630: caffe2/torch/CMakeFiles/torch_python.dir/csrc/init_flatbuffer_module.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:4151: caffe2/torch/CMakeFiles/torch_python.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2


### Versions

(base) -bash-4.1$ python torch/utils/collect_env.py 
Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: CentOS release 6.9 (Final) (x86_64)
GCC version: (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5)
Clang version: Could not collect
CMake version: version 3.21.3
Libc version: glibc-2.10

Python version: 3.7.4 (default, Aug 13 2019, 20:35:49)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-2.6.32-696.el6.x86_64-x86_64-with-centos-6.9-Final
Is CUDA available: N/A
CUDA runtime version: 10.1.105
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.12.0a0+git664058f
[conda] numpy                     1.21.6                   pypi_0    pypi
[conda] torch                     1.12.0a0+git664058f          pypi_0    pypi

cc @malfet @seemethere
Turn on Inductor Max Pool2d Backward Lowering For Channels Last
Labels: feature, good first issue, triaged, oncall: pt2, module: inductor

### 🚀 The feature, motivation and pitch

We currently disable max_pool2d_with_indices_backward when [an input is in channels last](https://github.com/pytorch/pytorch/blob/main/torch/_inductor/lowering.py#L3063-L3066) due to slow kernel generation. 

This is reproducible in https://github.com/pytorch/pytorch/blob/main/benchmarks/dynamo/microbenchmarks/operatorbench.py with a few local changes, shared below.

`python /scratch/eellison/work/pytorch/benchmarks/dynamo/microbenchmarks/operatorbench.py --suite=timm --op=aten.max_pool2d_with_indices_backward.default --max-samples=5 --dtype=float16 --channels-last`

> Inductor Speedups : [0.8040992530735804, 0.8699831653183436, 1.059330068525701]

However, when we run with `TORCHINDUCTOR_COORDINATE_DESCENT_TUNING=1` those regressions turn into speedups:

> Inductor Speedups : [1.1356843331778068, 1.2388101486725653, 1.5219576909790553]

We should investigate updating our pointwise heuristics when running channels last kernels and turn back on kernel. 


You'll need to get the changes from https://github.com/pytorch/pytorch/pull/103110 and disable the [fallback](https://github.com/pytorch/pytorch/blob/main/torch/_inductor/lowering.py#L3063-L3066). 

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225
Increased / more verbose type aliases for improved readability of user defined content
Labels: module: typing, triaged, enhancement, needs research

### 🚀 The feature, motivation and pitch

Right now PyTorch exports a few classes and has a few type aliases under `torch.types`. 
While these are useful, for those working with pytorch there may be a case to use `TypeAlias` to help clarify arguments and return statements. For example consider just a few of the following aliases:

```python
TorchDevice: TypeAlias = Union[Device, TorchDeviceTypes]


TorchTensor: TypeAlias = Union[tuple(torch._tensor_classes)] # could be just torch.Tensor, but doesn't offer all the hints
TorchTensors: TypeAlias = Sequence[TorchTensor]
TorchLayer: TypeAlias = torch.nn.Module
TorchLayers: TypeAlias = Sequence[TorchLayer]

TorchLoss: TypeAlias = torch.nn.Module
TorchLosses: TypeAlias = Sequence[TorchLoss]

StateDict: TypeAlias = dict
MaskTensor: TypeAlias = Union[torch.BoolTensor, torch.IntTensor]

HiddenState: TypeAlias = TorchTensor
CellState: TypeAlias = TorchTensor

GRUState: TypeAlias = HiddenState
LSTMStates: TypeAlias = Tuple[HiddenState, CellState]

RNNStates: TypeAlias = Union[GRUState, LSTMStates]
```

Suppose someone is defining custom layer that users can initialize with either a `GRU` or `LSTM`.  Showing that the forward call returns `Tuple[Tensor, RNNStates]` may help developer more readily remember that `RNNStates` is either a single `tensor`, or a `tuple`, which could otherwise be overlooked. 

Likewise `TorchLayer` and `TorchLoss`, while both an alias for `nn.Module` would help clarify if someone is passing a loss function or a layer into a model a wider variety of what expect input is. 


Does PyTorch need an `torch.aliases` submodule  with a bunch of different aliases to use? Not technically. I do, however, find working with others on projects where doing so can make things much more clear for collaboration. One could also just use longer type annotations, but I find those eventually clutter code e.g.

```
def foo(..., rnn_states: Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]], ...):
    pass
```

It is just a preference.

### Alternatives

_No response_

### Additional context

I put it in a [PR](https://github.com/pytorch/pytorch/pull/102914#issuecomment-1578935718)

cc @ezyang @malfet @rgommers @xuzhao9 @gramster
PyTorch should not use `windows.8xlarge.nvidia.gpu` to test binary builds
Labels: module: ci, triaged

### 🐛 Describe the bug

Because binary tests are tiny, so smallest machine we can get is better, and also `windows.8xlarge.nvidia.gpu` are often queued. From [hud/metircs](https://hud.pytorch.org/metrics):
<img width="668" alt="image" src="https://github.com/pytorch/pytorch/assets/2453524/27bc37f2-6e51-4a04-a084-b4ef76f16b96">


### Versions

CI

cc @seemethere @pytorch/pytorch-dev-infra
Refactor mm_plus_mm to check conditions upfront
Labels: feature, good first issue, triaged, oncall: pt2, module: inductor

### 🚀 The feature, motivation and pitch

We always lower `x @ y + a @ b` to [tuned_mm_plus_mm](https://github.com/pytorch/pytorch/blob/main/torch/_inductor/fx_passes/post_grad.py#L138-L139) even though we only lower to the fused op in special conditions, see https://github.com/pytorch/pytorch/blob/08c4a442fd589f380ac0dd6f1126d6e144f449e5/torch/_inductor/kernel/mm_plus_mm.py#L154-L159. 


We could refactor this check to occur in the pattern matching pass so that these mms/adds aren't prevented from participating in other patterns or fusions.

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225
torch.compile specializes on output name
Labels: high priority, triaged, oncall: pt2

### 🐛 Describe the bug

x-posting https://discuss.pytorch.org/t/pytorch-compile-requires-fixed-input-naming/181323

```python
import torch
torch._dynamo.config.verbose=True

class MyModule(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.lin = torch.nn.Linear(100, 10)

    def forward(self, x, step):
        return {f'output_{step}': self.lin(x[f'input_{step}'])}

mod = MyModule()
opt_mod = torch.compile(mod)

my_input = {
    'input_0': torch.ones([100]),
    'input_1': torch.ones([100])
}

for step in range(2):
    output = opt_mod(my_input, step)
    print(output.keys())
```


### Error logs


### Expected Output

```
dict_keys(['output_0'])
dict_keys(['output_1'])
```

### Actual Output

```
dict_keys(['output_0'])
dict_keys(['output_0'])
```

### Minified repro

n/a

### Versions

.

cc @ezyang @gchanan @zou3519 @wconstab @bdhirsh @anijain2305
Ambiguitiy in causal-mask in scaled_dot_product_attention
Labels: oncall: transformer/mha

### 🐛 Describe the bug

When `is_causal=True`, `torch.nn.functional.scaled_dot_product_attention` is defined as an efficient implementation of the following

```
# Q: (N, ..., L, E)
# K: (N, ..., S, E)
# V: (N, ..., S, Ev)
attn_mask = torch.ones((L, S), dtype=torch.bool).tril(diagonal=0)
attn_mask = attn_mask.maksed_fill(not attn_mask, -float('inf'))
attn_weight = torch.softmax((Q @ K.transpose(-2, -1) / math.sqrt(Q.size(-1))) + attn_mask, dim=-1)
out = attn_weight @ V
```

If `L != S`, the definition of attn_mask may be ambiguous. 

For example, during the auto-regressive generation process of a GPT-like model, when kv-cache is enabled, usually `L=1`. In this case, 

```python
In [1]: attn_mask = torch.ones((1, S), dtype=torch.bool).tril(diagonal=0); attn_mask
Out[1]: tensor([[ True, False, False, False, False, False, False, False]])
``` 

However, the correct `attn_mask` in this scenario should be `[True, True, ..., True]`.

I'm not sure it's a bug or feature. I guess there must be a reason why the causal mask is designed in the current way. It would be helpful to discuss.

### Versions

2.0

cc @jbschlosser @bhosmer @cpuhrsch @erichan1 @drisspg
Unexpected failure in LLVM JIT when running TorchScript model in C++
Labels: oncall: jit

### 🐛 Describe the bug

I have a [TorchScript model](https://github.com/pytorch/pytorch/files/11661294/model.zip) that I exported from Python and loaded in C++ via the following snippet (network is member of a class)
```c++
network = torch::jit::load(request_msg.torch_script_path);
network.eval();
```

The model is a standard feedforward model with [1024, 512, 256, 256] linear units and ELU activations and producing 4 output values with a final linear layer. When I try to execute the loaded model via 
```c++
std::vector<c10::IValue> inputs;
inputs.emplace_back(torch::zeros({1, 275}, torch::kFloat32));
auto network_effort_internal = network.forward(inputs);
```
I get the following error on the second time I try to call the network
```
terminate called after throwing an instance of 'c10::Error'
  what():  valOrErr INTERNAL ASSERT FAILED at "../torch/csrc/jit/tensorexpr/llvm_jit.h":33, please report a bug to PyTorch. Unexpected failure in LLVM JIT: Unable to find target for this triple (no targets are registered)
```
Interestingly, I can prevent the error from appearing by cloning the network prior to each call
```
network = network.clone();
```
which is however not a solution since this can be a rather costly operation compared to the execution of the model. This error has also[ been mentioned in the PyTorch discussion forum](https://discuss.pytorch.org/t/calling-forward-on-torchscript-model-multiple-times-leads-to-error/154990), however has not seen much discussion so far.


### Versions

I am using libtorch in Version 2.0.1. The model has been exported from PyTorch 1.11.0 but I also checked with Python version 2.0.1 with no change in behavior.

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
[Pytorch 2.0] torch::nn::Dropout output is incorrect on Windows  
Labels: oncall: binaries, module: windows, module: cpp, triaged, module: regression

### 🐛 Describe the bug

Reproduce steps:
1. refer to https://pytorch.org/cppdocs/installing.html
2. download pytorch 2.0.1 windows libtorch package from https://pytorch.org/
3. update the example-app.cpp
```cpp
#include <torch/torch.h>
#include <iostream>
int main() {
	auto m = torch::nn::Dropout(torch::nn::DropoutOptions().p(0.42));
	std::cout << "module : " << std::endl << m << std::endl << std::endl;
	std::cin.get();
}
```
4. build and run
```
mkdir build
cd build
cmake -G "Visual Studio 16 2019" -DCMAKE_PREFIX_PATH=/absolute/path/to/libtorch ..
cmake --build . --config Release
Release\example-app.exe
```
5. the p in the output is incorrect
  if libtorch is libtorch-win-shared-with-deps-2.0.1+cu117, the p is a random value.
    ```
    module :
    torch::nn::Dropout(p=8.749e+99, inplace=false)
    ```
    if libtorch is libtorch-win-shared-with-deps-2.0.1+cpu, the p is zero
    `torch::nn::Dropout(p=0, inplace=false)`


The result is correct on Linux or Libtorch1.13 on Windows.
```
module :
torch::nn::Dropout(p=0.42, inplace=true)
```

### Versions

Stable(2.0.1) LibTorch
The output isn't correct with CPU or CUDA binaries.


cc @seemethere @malfet @peterjc123 @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite @jbschlosser @ezyang @gchanan @zou3519
lit-llama lora fine tuning NetworkXUnbounded: Infinite capacity path, flow unbounded above
Labels: high priority, triaged, oncall: pt2, module: dynamic shapes

### 🐛 Describe the bug

lit-llama version 8aa65ba33e844c283c0a84b9758445fe0c6aab2d

Enable torch.compile with

```
diff --git a/finetune/lora.py b/finetune/lora.py
index 1873701..2d18ee8 100644
--- a/finetune/lora.py
+++ b/finetune/lora.py
@@ -73,6 +73,8 @@ def main(
     
     mark_only_lora_as_trainable(model)
 
+    model = torch.compile(model, dynamic=True)
+
     optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)
     model, optimizer = fabric.setup(model, optimizer)
     train(fabric, model, optimizer, train_data, val_data, tokenizer_path, out_dir)
```

Work around unrelated bug with

```
diff --git a/lit_llama/lora.py b/lit_llama/lora.py
index 0f644e2..3f46d1a 100644
--- a/lit_llama/lora.py
+++ b/lit_llama/lora.py
@@ -320,7 +320,7 @@ class MergedLinear(nn.Linear, LoRALayer):
                     self.lora_B.unsqueeze(-1),  # (256, 2) -> (256, 2, 1)
                     groups=sum(self.enable_lora)
                 ).transpose(-2, -1)  # (64, 4, 64) @ (256, 2, 1) -> (64, 256, 64) -> (64, 64, 256)
-                result += self.zero_pad(after_B) * self.scaling  # (64, 64, 256) after zero_pad (64, 64, 384)
+                result = result + self.zero_pad(after_B) * self.scaling  # (64, 64, 256) after zero_pad (64, 64, 384)
             return result
```

Patch in Triton fix https://github.com/openai/triton/pull/1741

Use standard setup instructions, `python finetune/lora.py`

Fails with

```
NetworkXUnbounded: Infinite capacity path, flow unbounded above
```

The graph that was generated: https://gist.github.com/ezyang/71460c07dfc86d297090888e077bd88e

I think this is the corresponding joint graph https://gist.github.com/ezyang/65e55a66c9cee4120c1ba242bbaef358

This also affects finetune/adapter.py, so it's probably an issue in the llama model itself

cc @gchanan @zou3519 @msaroufim @wconstab @bdhirsh @anijain2305 @Chillee 

### Versions

main
MPS bug: padding_idx in nn.Embedding does not prevent gradient accumulation 
Labels: triaged, module: mps

### The `padding_idx` in `nn.Embedding` does not prevent gradient accumulation when run on MPS

Expected behaviour:
```python
import torch

embedding = torch.nn.Embedding(5, 3, padding_idx=0)
input_ids = torch.LongTensor([[0, 1, 1, 2]]) # padding_token, token_1, token_1, token_2
loss = torch.sum(embedding(input_ids))
loss.backward()
# expect to see zeros for the padding token, 2 for token_1, 1 for token_2, and 0 for other tokens
expected = torch.tensor([0, 2, 1, 0, 0]).repeat(3, 1).T
assert torch.all(torch.eq(embedding.weight.grad, expected)), embedding.weight.grad  # all good
```

On MPS the gradient accumulates for the padding_idx:
```python

embedding = torch.nn.Embedding(5, 3, padding_idx=0).to('mps')
input_ids = torch.LongTensor([[0, 1, 1, 2]]).to('mps') # padding_token, token_1, token_1, token_2
loss = torch.sum(embedding(input_ids))
loss.backward()
# expect to see zeros for the padding token, 2 for token_1, 1 for token_2, and 0 for other tokens
expected = torch.tensor([0, 2, 1, 0, 0]).repeat(3, 1).T.to('mps')
assert torch.all(torch.eq(embedding.weight.grad, expected)), embedding.weight.grad  # fails
```
```python
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[10], line 7
      5 # expect to see zeros for the padding token, 2 for token_1, 1 for token_2, and 0 for other tokens
      6 expected = torch.tensor([0, 2, 1, 0, 0]).repeat(3, 1).T.to('mps')
----> 7 assert torch.all(torch.eq(embedding.weight.grad, expected)), embedding.weight.grad

AssertionError: tensor([[1., 1., 1.],
        [2., 2., 2.],
        [1., 1., 1.],
        [0., 0., 0.],
        [0., 0., 0.]], device='mps:0')
```
This would lead to the padding vector being updated unexpectedly during training on MPS. 

### Versions

Collecting environment information...
PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.3.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 20:12:31) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-13.3.1-arm64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Max

Versions of relevant libraries:
[pip3] flake8==4.0.1
[pip3] numpy==1.22.4
[pip3] torch==2.0.1
[conda] numpy                     1.22.4          py310h0a343b5_0    conda-forge
[conda] pytorch                   2.0.1                  py3.10_0    pytorch

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev