gopikrsmscs/torch-issues huggingface dataset:
Preserve weight_g/weight_v accessors on new weight_norm
Labels: module: nn, triaged, module: nn.utils.parametrize

### 🐛 Describe the bug

Parametrizations don't let you control what the original parameters are called; they're always original0, original1, etc. For weight_norm, this new naming is a bit obtuse; the original naming of g/v was better. Not sure if this is actually worth fixing, holler if you think it is.

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @lezcano 

### Versions

main
raise `RuntimeError` faster when loading an object with a torch CUDA tensor on a CPU-only machine
Labels: module: nn, module: serialization, triaged, actionable

### 🚀 The feature, motivation and pitch

Currently when loading an object with a torch CUDA tensor on a CPU-only machine, torch will raise a `RuntimeError` like https://github.com/pytorch/pytorch/issues/16797 mentioned. A problem in this situation is that torch will spend time (maybe trying to load the tensor bytes) before raise that `RuntimeError`. This extra time is proportional to the size of the tensor. Hence when loading a huge tensor, the extra time may be noticeable for performance-critical applications.

To give a concrete example, I have the following codes:

```python
def _safe_torch_tensor_loads(bs: bytes) -> t.Any:
    import torch

    f = io.BytesIO(bs)
    if not torch.cuda.is_available():
        return torch.load(f, map_location="cpu")
    else:
        return torch.load(f)


class FixTorchUnpickler(pickle.Unpickler):
    def find_class(self, module: str, name: str) -> t.Callable[[bytes], t.Any]:
        if module == "torch.storage" and name == "_load_from_bytes":
            return _safe_torch_tensor_loads
        else:
            return super().find_class(module, name)


def _fix_torch_loads(bs: bytes) -> t.Any:
    f = io.BytesIO(bs)
    unpickler = FixTorchUnpickler(f)
    return unpickler.load()


def loads_or_fix_torch(bs: bytes):
    try:
        return pickle.loads(bs)
    except RuntimeError:
        return _fix_torch_loads(bs)
```

where `_fix_torch_loads` will directly loads a object containing cuda tensors even on a CPU-only machine and `loads_or_fix_torch` will first try to load the object and only use `_fix_torch_loads` when it met the `RuntimeError`. In my benchmark (I actually have a benchmark repo [here](https://github.com/larme/pytorch_unpickler_benchmark)), `loads_or_fix_torch` will be 60%-70% slower than `_fix_torch_loads` for different sizes of objects. For a large enough tensor  (like (25000, 25000) fp32 tensor), the difference could be 500-600ms. My application is distributed model inference serving so 500-600ms is quite a lot from end user's point of view.

I'd like to help improving this situation but I need some input from more experienced pytorch developer. I tried to dive in `torch.serialization` and `torch.storage` but they are a little bit complex so if some pointer will be helpful!

### Alternatives

_No response_

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
Discussion and Design  for Masked Loss Functions which can be used with PackedSequence training (but not exclusively)
Labels: module: loss, triaged, module: masked operators

### 🚀 The feature, motivation and pitch

Original idea for these come from [Github Gist] that helps reproduce [Issue 102911][packed seqs on mps] ([PackedSequences on MPS accelerator yields grad_y missing or crashes the kernel][packed seqs on mps])


See: [PR][Pull Request]

[packed seqs on mps]: https://github.com/pytorch/pytorch/issues/102911
[packed sequence failure]: https://github.com/pytorch/pytorch/issues/97552
[loss error on m1]: https://github.com/pytorch/pytorch/issues/96416
[grad_y missing fix]: https://github.com/pytorch/pytorch/pull/96601
[gru nan]: https://github.com/pytorch/pytorch/issues/94691
[Github Gist]: https://gist.github.com/dsm-72/1cea0601145a8b92155d8d08c90bf998
[Results]: https://github.com/pytorch/pytorch/issues/94691#issuecomment-1574365231
[Pull Request]: https://github.com/pytorch/pytorch/pull/102915#issuecomment-1576748710

### Alternatives

_No response_

### Additional context

[packed seqs on mps]: https://github.com/pytorch/pytorch/issues/102911

The [Issue 102911][packed seqs on mps] stated above is related to:
- [Issue 96416][loss error on m1] ([Loss.backward() error when using MPS on M1 #96416][loss error on m1]), 
- [Issue 94691][gru nan] ([Nan is output by GRU on mps #94691][gru nan]), 
- [Issue 97552][packed sequence failure] ([PackedSequence failure with MPS #97552][packed sequence failure]), and 
- [PR 96601][grad_y missing fix] ([[MPS] LSTM grad_y missing fix #96601][grad_y missing fix]).

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev


### Description

When training it can be useful to either pad input **_or_** mask parts of the expected output to increase efficiency and / or robustness. The example above demonstrates this with a RNN for time series where each sequence may have variable length utilizes padded sequences and `PackedSequence`  to improve throughput. The use of padded output (since this was a Seq2Seq model the padded input is the output) or masked output motivates a loss function that can handle this, without having to adjust the dataset / dataloaders / training method.  

Such occurs enough to motivate even a base `MaskedLoss` class that can be used to, in a standardized way, quickly and easily generate other masked loss functions e.g. `MaskedMSELoss`, `MaskedL1Loss`, etc.

The [PR][Pull Request] offers an example of how one might achieve this.
[Inductor] add debugging tools
Labels: triaged, open source, Stale, topic: not user facing, intel, module: inductor, ciflow/inductor

Add debugging tools for Inductor.

### Description

- Graph merger: Merge FX graphs of a model into a single large FX graph. Thus, graphs can be compared quickly between different versions of PyTorch.
- Graph matching: In order to know what each kernel does, this tool matches cpp kernel with FX graph operators and adds corresponding operators before each kernel in cpp output code.


#### Example result of graph merger:
```
GRAPH_INDEX:0
class <lambda>(torch.nn.Module):
    def forward(self, arg0_1: f32[2, 3]):
        # File: test_cpu_repro.py:1845, code: x = x.pow(5)
        pow_1: f32[2, 3] = torch.ops.aten.pow.Tensor_Scalar(arg0_1, 5);  arg0_1 = None
        return (pow_1,)

GRAPH_INDEX:1
class <lambda>(torch.nn.Module):
    def forward(self, arg0_1: f32[2, 3]):
        # File: test_cpu_repro.py:1847, code: return torch.sin(x)
        sin: f32[2, 3] = torch.ops.aten.sin.default(arg0_1);  arg0_1 = None
        return (sin,)
```

#### Example result of graph matching:
```
#line 42: pow_1 = torch.ops.aten.pow.Tensor_Scalar(arg0_1, 5);  arg0_1 = None
#line 43: sin = torch.ops.aten.sin.default(pow_1);  pow_1 = None
cpp_fused_pow_sin_0 = async_compile.cpp('''
#include "/tmp/torchinductor_root/mq/cmqzxwuyo7ryvun3egqos5jq5ak4fue7d2jbopbqs7pgpkhdpfh4.h"
extern "C" void kernel(const float* in_ptr0,
                       float* out_ptr0)
{
    {
        for(long i0=static_cast<long>(0L); i0<static_cast<long>(48L); i0+=static_cast<long>(16L))
        {
            auto tmp0 = at::vec::Vectorized<float>::loadu(in_ptr0 + static_cast<long>(i0));
            auto tmp1 = tmp0 * tmp0;
            auto tmp2 = tmp1 * tmp1;
            auto tmp3 = tmp2 * tmp0;
            auto tmp4 = tmp3.sin();
            tmp4.store(out_ptr0 + static_cast<long>(i0));
        }
        #pragma omp simd simdlen(8)
        for(long i0=static_cast<long>(48L); i0<static_cast<long>(60L); i0+=static_cast<long>(1L))
        {
            auto tmp0 = in_ptr0[static_cast<long>(i0)];
            auto tmp1 = decltype(tmp0)(tmp0 * tmp0);
            auto tmp2 = decltype(tmp1)(tmp1 * tmp1);
            auto tmp3 = decltype(tmp2)(tmp2 * tmp0);
            auto tmp4 = std::sin(tmp3);
            out_ptr0[static_cast<long>(i0)] = tmp4;
        }
    }
}
''')

```


cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @voznesenskym @penguinwu @EikanWang @Guobing-Chen @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @Xia-Weiwen @ngimel
TypeError: (): incompatible function arguments
Labels: oncall: distributed, triaged

### 🐛 Describe the bug

Hello，I am customizing process group backends using cpp extensions according to PyTorch Tutorials，[Customize Process Group Backends Using Cpp Extensions — PyTorch Tutorials 2.0.1+cu117 documentation](https://pytorch.org/tutorials/intermediate/process_group_cpp_extension_tutorial.html) . But an error occurred . It seems to be type error, but I find the type is right. 
**In addition, I tried on the PyTorch Release 23.05 image and found no issues. However, when I uninstalled torch2.0.0 and reinstalled it through pip install, I was able to reproduce the type error. Does this mean that installing torch through pip install may miss some dependencies, causing type errors?** 
The specific error message is shown below:
```
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 895, in init_process_group
    default_pg = _new_process_group_helper(
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1034, in _new_process_group_helper
    backend_class = creator_fn(store, group_rank, group_size, timeout)
TypeError: (): incompatible function arguments. The following argument types are supported:
    1. (arg0: c10d::Store, arg1: int, arg2: int, arg3: datetime.timedelta) -> c10d::Backend

Invoked with: <torch.distributed.distributed_c10d.PrefixStore object at 0x7f8db2ec9230>, 1, 4, datetime.timedelta(seconds=1800)
```

### Versions

nvcr.io/nvidia/pytorch:23.05-py3

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
[onnx] aten::cumprod cannot be exported to ONNX
Labels: module: onnx, triaged, OSS contribution wanted

### 🐛 Describe the bug

aten::cumprod fails to be exported to ONNX

```
import torch
from torch import nn

class Test(nn.Module):
    def forward(self, x):
        return torch.cumprod(x, dim=-1)

if __name__ == '__main__':

    model = Test()
    torch.onnx.export(model, torch.randn(1, 1, 1024, 65), "cumprod.onnx")
```

error message:
```
torch.onnx.errors.UnsupportedOperatorError: Exporting the operator 'aten::cumprod' to ONNX opset version 14 is not supported.
```

### Versions

PyTorch version:  2.0.1+cu117
python : 3.9
Support for efficiently processing categorical distributions with varying dimensions
Labels: module: distributions, feature, triaged, needs research

### 🚀 The feature, motivation and pitch

Say I have a list of probability distribution tensors of varying size. It would be nice if `Categorical` could accept a flattened tensor `probs_flat` containing all the distributions, along with a pointer `ptr` that indicates the starting index of each distribution. Standard functions can efficiently be computed, e.g. entropy can be computed as `-segment(probs_flat.log() * probs_flat, ptr)`, where `segment` is something like `segment_add_csr` from `torch_scatter` (I'm not sure if torch provides this function out of the box).

I'm not too sure of the best way to integrate this with the existing code. Just an idea!

### Alternatives

Let `probs_list` be a list of probability distribution tensors, and `xs` be a list of samples from their respective distributions. There are currently two options to e.g. evaluate the log-probabilities of the samples:
1. (evaluate for each)
```Python
log_probs = torch.stack([
    Categorical(probs=probs).log_prob(x)
    for probs, x in zip(probs_list, xs)
])
```
2. (pad, stack, evaluate)
```Python
max_size = max(probs.numel() for probs in probs_list)
probs_padded_list = []
for probs in probs_list:
    probs_padded = probs.new_zeros(max_size)
    probs_padded[:probs.numel()] = probs
    probs_padded_list += [probs_padded]
pi = Categorical(torch.stack(probs_padded_list))
log_probs = pi.log_prob(torch.stack(xs))
```

### Additional context

This can be useful in RL, where the number of possible actions varies between environment steps, as a less wasteful alternative to action masking.

cc @fritzo @neerajprad @alicanb @nikitaved
torch.cuda.is_available() returns False on GTX 1650 with cuda 11.7 and torch==2.0.0+cpu
Labels: module: build, triaged

### 🐛 Describe the bug

i get False when i run torch.cuda.is_available() 

nvidia-smi
**+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.98                 Driver Version: 535.98       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1650      WDDM  | 00000000:01:00.0  On |                  N/A |
| 30%   39C    P0              15W /  75W |   1145MiB /  4096MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+**

i even installed 11.7 and it still shows 12.2




when i run pip show torch it gives me version 2.0.1 + cu117 which is why i installed cuda 11.7 but i previously had 12.2
**C:\Users\ambsu>pip show torch
Name: torch
Version: 2.0.1+cu117**


py -m torch.utils.collect_env
gives 
**Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==2.0.0+cpu
[pip3] torchaudio==2.0.2+cu118
[pip3] torchvision==0.15.2
[conda] Could not collect**


everything is showing different versions









### Error logs

**Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==2.0.0+cpu
[pip3] torchaudio==2.0.2+cu118
[pip3] torchvision==0.15.2
[conda] Could not collect**

nvidia-smi
**+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.98                 Driver Version: 535.98       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1650      WDDM  | 00000000:01:00.0  On |                  N/A |
| 30%   39C    P0              15W /  75W |   1145MiB /  4096MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+**

### Minified repro

import torch

print(torch.cuda.is_available()) # False

### Versions

cuda 11.7 gtx
torch 2.0.1 and 2.0.0

cc @malfet @seemethere @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
Unbox expectedml
Labels: triaged

### 🐛 Describe the bug

Stable Diffusion failed to generate an image.  In an unusual twist, the console asks me to submit a bug report to PyTorch. So I am, although I don't exactly have details.  SD has been acting up this evening, despite doing just fine for two days.  I've never seen this before.  This occurred while I was using Neurogen 1.1.

### Versions

The script pops up and disappears.  If there is an output somewhere, I can't find it.
![error with tensor](https://github.com/pytorch/pytorch/assets/81931767/9f5eab8e-d55a-4040-9692-fe35aceb73ca)
nn.ChannelShuffle1d
Labels: module: nn, triaged, needs research

### 🚀 The feature, motivation and pitch

nn.ChannelShuffle operates on 2D tensors. Either:

- Introduce nn.ChannelShuffle1d, nn.ChannelShuffle2d, etc
- Adjust nn.ChannelShuffle to support 1d as well

### Alternatives

Convert 1D -> 2D artificially before applying ChannelShuffle:

```python
channel_shuffle = nn.ChannelShuffle(4)
x = torch.arange(64).reshape(1, 8, 8)
x = channel_shuffle(x[..., None])[..., 0]
```

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
After dynamo minifier generates repros that don't entirely match what we minified over
Labels: triaged, module: dynamo

### 🐛 Describe the bug

The after dynamo minifier tests minified graphs by feeding them directly to the backend in question. However, this is not what the repro script does: the repro script runs dynamo (torch.compile) on the serialized FX graph, with the backend set. If running dynamo on a dynamo'ed FX graph is not idempotent, then you can end up with a slightly different dynamo graph, which could in turn mean you end up with a repro that doesn't actually repro.

Fortunately, for the most part, Dynamo'ing a serialized FX graph idempotent. But there's one notable case I recently ran into where it isn't: when the graph is empty! (In that case we never bother calling the graph at all.) In my case (testing) this is easy enough to work around; I'm not sure if there are other cases that actually matter in practice.

### Versions

main

cc @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy
BCELoss and BCEWithLogitsLoss differ when one of the input logits is float("inf")
Labels: module: nn, triaged, module: edge cases

### 🐛 Describe the bug

Based on the docs, it seems like `BCEWithLogitsLoss` is supposed to be the more numerically table sibling of `BCELoss`. However, when one of the input logits to `BCEWithLogitsLoss` is `float("inf")` the result is `Nan` whereas `sigmoid(inf) = 1.0`, so it should probably be well defined. When I use `BCELoss(sigmoid(x), target)` it gives the desired result.

```
x = torch.tensor([float("inf"), 1.0])
y = torch.tensor([1.0, 0.0])

loss = F.binary_cross_entropy_with_logits(x, y)
print (loss)

loss = F.binary_cross_entropy(x.sigmoid(), y)
print (loss)

print(torch.nn.BCEWithLogitsLoss()(x, y))
print(torch.nn.BCELoss()(x.sigmoid(), y))
```

Output:

```
tensor(nan)
tensor(0.6566)
tensor(nan)
tensor(0.6566)
```


### Versions

1.12.1

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
[dynamo] Diffusers - Graph break on OrderedDict 
Labels: triaged, module: dynamo

UPDATE - There are simpler repros to fix in the comments below. The one in the first post requires more interaction because of how dataclass and OrderedDict works.

~~~
import diffusers
from diffusers.utils import BaseOutput
import dataclasses
from dataclasses import dataclass
import torch

@dataclass
class Transformer2DModelOutput(BaseOutput):
    """
    Args:
        sample (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)` or `(batch size, num_vector_embeds - 1, num_latent_pixels)` if [`Transformer2DModel`] is discrete):
            Hidden states conditioned on `encoder_hidden_states` input. If discrete, returns probability distributions
            for the unnoised latent pixels.
    """

    sample: torch.FloatTensor



def fn(x):
    return Transformer2DModelOutput(sample=x)


x = torch.randn(4)
ref = fn(x)

opt_fn = torch.compile(fn, backend="eager", fullgraph=True)
res = opt_fn(x)
~~~




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy
Inductor: delete code that extracts out sizevars by inspecting tensor inputs to find a size that handled it
Labels: triaged, better-engineering, module: inductor

### 🐛 Describe the bug

This is no longer needed since we now past in all free symbols as explicit arguments

### Versions

main

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225
pdb but for dynamo (and time travel debugging)
Labels: feature, triaged, oncall: pt2

### 🐛 Describe the bug

Pitch: In normal eager PyTorch, I can breakpoint() inside Python code and then single step forward.

If I torch.compile() a module, I can no longer do this anymore. What I would like is a way to do this, and single step, but in the context of dynamo symbolic tracing.

Because dynamo symbolic tracing is non-destructive, it should also be possible to time travel debug (e.g., step backwards).

### Versions

main

cc @msaroufim @wconstab @bdhirsh @anijain2305
The operator 'aten::poisson' is not currently implemented for the MPS device
Labels: triaged, module: mps

### 🚀 The feature, motivation and pitch

I am running some MCMC sample using torch. currently, the torch distributions don't support sampling from a Poisson distribution when the intensity parameter is a tensor (say 2D). Essentially, it cannot do a vector/matrix of entries. 

I get this error: 
NotImplementedError: The operator 'aten::poisson' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764. As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op. 
WARNING: this will be slower than running natively on MPS.

### Alternatives

_No response_

### Additional context

_No response_

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
Dynamo should only unroll loops by a preset factor (unless otherwise explicitly instructed)
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

Today, dynamo is willing to unroll loops for arbitrarily long. This can lead to extremely large graphs which in turn lead to slow compile times, for example, https://github.com/pytorch/pytorch/issues/102622

Instead, we should unroll only by a pre-set factor, e.g., 5 or something. If we notice that we have looped too many times (and the loop is generating graph), we should graph break, forcing us to generate separate code for the loop body. (In the further future, we could automatically host side loop'ify the graph, so we could avoid having to do guard tests on entry.) 

### Versions

main

cc @msaroufim @wconstab @bdhirsh @anijain2305
LibTorch-Lite 1.13.0.1 Crash on iOS 12 on app startup
Labels: triaged, module: ios

### 🐛 Describe the bug

When app start on iOS 12.5.7 (latest version for iphone 5s, 6 and some iPad), the app crash with the following logs: 

```
dyld: Symbol not found: ___chkstk_darwin
  Referenced from: /var/containers/Bundle/Application/XXXX/PlantNet.app/PlantNet
  Expected in: /usr/lib/libSystem.B.dylib
 in /var/containers/Bundle/Application/XXXX/PlantNet.app/PlantNet
(lldb) 
```

The [Podspec](https://github.com/CocoaPods/Specs/blob/master/Specs/c/c/3/LibTorch-Lite/1.13.0.1/LibTorch-Lite.podspec.json) indicate iOS 12.0 

#### Impact

All iOS Users on 12.5.7 cannot use the app as the app crash on start after the update. 
Potential workaround are time consuming involving releasing a specific version for them on the App Store before publishing a new app dropping the support for iOS 12

Stacktrace as follow: 
```
# 0 __abort_with_payload
dyld`:
    0x1087f2410 <+0>:  mov    x16, #0x209
    0x1087f2414 <+4>:  svc    #0x80
->  0x1087f2418 <+8>:  b.lo   0x1087f2430               ; <+32>
    0x1087f241c <+12>: stp    x29, x30, [sp, #-0x10]!
    0x1087f2420 <+16>: mov    x29, sp
    0x1087f2424 <+20>: bl     0x1087f1850               ; cerror_nocancel
    0x1087f2428 <+24>: mov    sp, x29
    0x1087f242c <+28>: ldp    x29, x30, [sp], #0x10
    0x1087f2430 <+32>: ret    
```

```
# 5 _dyld_start
dyld`:
    0x1087b1000 <+0>:   mov    x28, sp
    0x1087b1004 <+4>:   and    sp, x28, #0xfffffffffffffff0
    0x1087b1008 <+8>:   mov    x0, #0x0
    0x1087b100c <+12>:  mov    x1, #0x0
    0x1087b1010 <+16>:  stp    x1, x0, [sp, #-0x10]!
    0x1087b1014 <+20>:  mov    x29, sp
    0x1087b1018 <+24>:  sub    sp, sp, #0x10
    0x1087b101c <+28>:  ldr    x0, [x28]
    0x1087b1020 <+32>:  ldr    x1, [x28, #0x8]
    0x1087b1024 <+36>:  add    x2, x28, #0x10
    0x1087b1028 <+40>:  adrp   x4, -1
    0x1087b102c <+44>:  add    x4, x4, #0x0
    0x1087b1030 <+48>:  adrp   x3, 90
    0x1087b1034 <+52>:  ldr    x3, [x3, #0xad8]
    0x1087b1038 <+56>:  sub    x3, x4, x3
    0x1087b103c <+60>:  mov    x5, sp
    0x1087b1040 <+64>:  bl     0x1087b1088               ; dyldbootstrap::start(macho_header const*, int, char const**, long, macho_header const*, unsigned long*)
->  0x1087b1044 <+68>:  mov    x16, x0
    0x1087b1048 <+72>:  ldr    x1, [sp]
    0x1087b104c <+76>:  cmp    x1, #0x0
    0x1087b1050 <+80>:  b.ne   0x1087b105c               ; <+92>
    0x1087b1054 <+84>:  add    sp, x28, #0x8
    0x1087b1058 <+88>:  br     x16
    0x1087b105c <+92>:  mov    x30, x1
    0x1087b1060 <+96>:  ldr    x0, [x28, #0x8]
    0x1087b1064 <+100>: add    x1, x28, #0x10
    0x1087b1068 <+104>: add    x2, x1, x0, lsl #3
    0x1087b106c <+108>: add    x2, x2, #0x8
    0x1087b1070 <+112>: mov    x3, x2
    0x1087b1074 <+116>: ldr    x4, [x3]
    0x1087b1078 <+120>: add    x3, x3, #0x8
    0x1087b107c <+124>: cmp    x4, #0x0
    0x1087b1080 <+128>: b.ne   0x1087b1074               ; <+116>
    0x1087b1084 <+132>: br     x16

```

### Versions

LibTorch-Lite 1.13.0.1
React Native 0.70.9
TypeError: (): incompatible function arguments
Labels: needs reproduction, module: cpp-extensions, triaged

### 🐛 Describe the bug

Hello，I am customizing process group backends using cpp extensions according to PyTorch Tutorials，[Customize Process Group Backends Using Cpp Extensions — PyTorch Tutorials 2.0.1+cu117 documentation](https://pytorch.org/tutorials/intermediate/process_group_cpp_extension_tutorial.html) . But an error occurred . It seems to be type error, but I find the type is right. So how can I fix it? The following is the detailed error:
```
File “/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 895, in init_process_group
default_pg = _new_process_group_helper(
File “/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 1029, in _new_process_group_helper
backend_class = creator_fn(backend_prefix_store, group_rank, group_size, timeout)
TypeError: (): incompatible function arguments. The following argument types are supported:
1. (arg0: c10d::Store, arg1: int, arg2: int, arg3: datetime.timedelta) → c10d::Backend
```

### Versions

torch.__version__: 2.0.0+cu117

cc @malfet @zou3519
Unknow error when using `make_graphed_callables`
Labels: triaged, module: cuda graphs

### 🐛 Describe the bug

When I try to run the following code, 
```python
import torch

def test_func(t1: torch.Tensor, t2: torch.Tensor):
    return t1 + t2

r1 = torch.randn([1, 4, 8, 3, 3]).to('cuda')
r2 = torch.randn([1, 4, 8, 3, 3]).to('cuda')
tmp = torch.randn([1, 4, 8, 2]).to('cuda').requires_grad_()
r2[..., 1, 1] = tmp[..., 1]

cudagraph_func = torch.cuda.make_graphed_callables(test_func, (r1, r2))
```

I got the error msg:
```
    result = Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA error: operation would make the legacy stream depend on a capturing blocking stream
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```

### Versions

PyTorch version: 2.0.0a0+1767026
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.24.1
Libc version: glibc-2.31

Python version: 3.9.16 (main, Mar  8 2023, 14:00:05)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.4.0-131-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 12.1.66
CUDA_MODULE_LOADING set to: LAZY

cc @mcarilli @ezyang
mark_dynamic may error too aggressively
Labels: triaged, oncall: pt2, module: dynamic shapes

### 🐛 Describe the bug

Suppose you have pseudocode like this:

```
B = 4
x = torch.randn(B, 16)
y = torch.randn(B, 16)
torch._dynamo.mark_dynamic(x, 0)

@torch.compile()
def f(x, y):
    return x + y

f(x, y)
```

This will fail, because the addition with y will cause x to get specialized.

One might reasonably argue that this is what you want to happen, because you asked for x to be dynamic, and it wasn't. But in non-synthetic examples, it might be quite difficult to identify all of the locations you need to mark_dynamic to ensure all of the unifications work out. In particular, I was trying to understand why we fail to maintain dynamic batch size in HuggingFace T5, and the reason is that in generation, there are a decent number of graph breaks in some of the preparatory code which allocates tensors that also have batch size (remember, eager mode does not currently propagate dynamic dim). Understanding where you have to insert `mark_dynamic` also requires you to understand where all the graph breaks are happening, not an easy feat!

It would be simple enough to delete the "require that computation involving this tensor must be dynamic" test (or perhaps, keep it but under some other name; in fact, ShapeEnv is directly factored this way). But could there be something even better we can do? One observation is we might still want mark dynamic to interact with `assume_static_by_default`; the desire is to avoid a recompilation by noting that a kernel should be compiled dynamically from the getgo.

In this case, there is something we can do. We could allocate dynamic dimensions for all shapes in the beginning, and then infer all the ones that should be dynamic (e.g., because they were marked dynamic). For all other unconstrained dimensions, we would monomorphize them (that's the assume static by default). We give up the benefits of performing dynamo tracing with static shapes (something that is quite profitable for compile time), but save a recompile. This may be a case of penny-wise pound-foolish though, depending on the relative costs of allocating lots of symbols vs eating a recompile. A hybrid approach is to optimistically trace with everything static, and only if a mark_dynamic is violated, we trigger a retrace with everything dynamic by default.

As a short term unblocker, it probably makes sense to special case T5 a little so that it goes the slow route (but then successfully compiles dynamic batch size). Maybe if we can eliminate enough graph breaks things will get better.

### Versions

main

cc @msaroufim @wconstab @bdhirsh @anijain2305
`torch.poisson(torch.tensor([torch.inf))` returns 0
Labels: module: distributions, module: error checking, triaged, module: edge cases

### 🐛 Describe the bug

For `torch.poisson`,  `torch.inf` results `0`

```python
torch.poisson(torch.tensor([torch.inf]))
```
Output:
```
tensor([0.])
```
And interestingly
```python
torch.poisson(torch.tensor([torch.inf - 1]))
```
Output:
```
tensor([9.2234e+18])
```
While for `torch.binomial`
```python
torch.binomial(torch.tensor([torch.inf]), torch.tensor([0.5]))
```
Output:
```
tensor([nan])
```
Seems that torch would like to return `nan` for `inf` ? But `binomial` is some what inconsistent with `poisson`, and I believe for `inf` lambda or rate, `0` is not a good return value, since it's valid but nearly impossible for *large* rate.
I don't know it's a bug or work as intended, so I tag bug for now.

### Versions

PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.2 (arm64)
GCC version: Could not collect
Clang version: 12.0.1
CMake version: version 3.24.1
Libc version: N/A

Python version: 3.9.12 (main, Jun  1 2022, 06:34:44)  [Clang 12.0.0 ] (64-bit runtime)
Python platform: macOS-13.2-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

cc @fritzo @neerajprad @alicanb @nikitaved @malfet
Do smarter layout decisions with concatenate.
Labels: triaged, oncall: pt2, module: inductor

### 🚀 The feature, motivation and pitch

Today, we always decide to turn the result of a concatenate into a contiguous layout (see https://github.com/pytorch/pytorch/blob/main/torch/_inductor/ir.py#L2578).

This makes sense if we're concatenating along the last dim, since all of our data is stored in contiguous tensors.

However, if we're concatenating along outer dims, and particularly if that outer dim is very small (like with stack, which lowers into a concatenate with dim size 1), then this will result in each element being stored in noncontiguous memory layouts, which may be quite unperformant to write into.

Instead of always making it contiguous, we could do some more intelligent decisions on the output layout depending on the shape of the concatenate.

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225
DISABLED test_make_fx_symbolic_exhaustive_special_bessel_j0_cpu_float32 (__main__.TestProxyTensorOpInfoCPU)
Labels: triaged, module: flaky-tests, skipped, module: ProxyTensor

Platforms: linux, mac, macos, win, windows

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_make_fx_symbolic_exhaustive_special_bessel_j0_cpu_float32&suite=TestProxyTensorOpInfoCPU) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/undefined).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 2 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_make_fx_symbolic_exhaustive_special_bessel_j0_cpu_float32`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_proxy_tensor.py`
Introduce OptimizerInfos to test_optim (pt1)
Labels: Stale, topic: not user facing

This PR does two things:
1. Introduces OptimizerInfos, BUT does not use them.
2. Imports all the optimizers at the top and avoid re-accessing later.

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #102774
Support In-place Triangular Matrix Multiplication
Labels: triaged, enhancement, module: linear algebra

### 🚀 The feature, motivation and pitch

The cuBLAS API supports Triangular Matrix Multiplication (trmm) https://docs.nvidia.com/cuda/cublas/index.html?highlight=trmm#cublasxt-t-trmm which is particularly cool because you can do `X = T X` _in-place!_.
The trmm function supports both upper and lower triangular matrices, and they can be stored in the same tensor T. (Using upper and lower half.)
This gives a nice way to do general matrix multiplication in-place using the LU-decomposition. Alternatively, if we are using an MLP we may just learn the triangular matrices L and U directly.

### Alternatives

For systems without cuda support, I believe BLAS also supports trmm.
It's always possible to just fall back to normal matrix mult and allocate a new matrix.

### Additional context

See also the discussion here about in-place matrix multiplication: https://stackoverflow.com/questions/25450809/is-there-an-algorithm-to-multiply-square-matrices-in-place

cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano
Followup on the extra graph breaks for yolov3 model caused by layout optimization
Labels: triaged, module: inductor

### 🚀 The feature, motivation and pitch

Layout optimization increase graph breaks for yolov3 from 12 to 13. But number of unique graph breaks does not change.

Run 
```
python benchmarks/dynamo/torchbench.py --backend inductor --amp --accuracy --only yolov3 --training
```
And found the number of graph breaks to be 13 in the csv file. (unique number of graph breaks to be 7)

Disable layout optimization and rerun:
```
TORCHINDUCTOR_LAYOUT_OPTIMIZATION=0 python benchmarks/dynamo/torchbench.py --backend inductor --amp --accuracy --only yolov3 --training
```
We can found the number of graph breaks to be 12 in the csv file (unique number of graph breaks still to be 7).


### Alternatives

_No response_

### Additional context

_No response_

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel
Pytorch Build images for RISCV64 Devices in the nightly builds
Labels: module: build, feature, triaged

### 🚀 The feature, motivation and pitch

Please provide Nightly Builds for the RISCV64 CPU's in a similar fashion to the other nightly builds you provide.

### Alternatives

_No response_

### Additional context

_No response_

cc @malfet @seemethere
Unlocking PyTorch's Power: README.md in Multiple Languages! 
Labels: triaged, topic: new features, topic: docs

### 📚 The doc issue

We are a diverse group of students, each with different cultural backgrounds. As newcomers to the world of PyTorch, we realized the importance of having the README documentation available in our native languages.


### Suggest a potential alternative/fix

If you find our idea interesting and relevant, we would be happy to submit a pull request with the completed translations from our forked version of the project. We have already taken the initiative to translate the documentation into different languages, including German, Portuguese, Arabic, and Turkish, to ensure that beginners from diverse backgrounds can benefit from a localized experience.

By merging our translations into the main project, we can contribute to making PyTorch more accessible and user-friendly for a wider audience. We believe that providing comprehensive documentation in multiple languages will empower individuals to grasp the concepts more efficiently and foster a sense of inclusivity within the PyTorch community.

We look forward to your response and the opportunity to collaborate in making PyTorch a globally accessible resource for aspiring learners.

We encourage individuals from other nationalities to join us in this effort, extending the benefits to their own communities and facilitating the learning experience for everyone. Let's collaborate and contribute to a more inclusive and user-friendly PyTorch community.
Building NCCL with `make -l $MAX_JOBS` slows down builds
Labels: module: build, triaged

We build NCCL with a cpu load limit (see https://github.com/pytorch/pytorch/blame/main/cmake/External/nccl.cmake#L37) to avoid OOMs, but this has some unfortunate consequences for devs using Meta-managed devservers.  Mine has lots of RAM and CPU, but is very "noisy" with background processes (I currently have a 3% cpu utilization but a load average of 137).  I think `make -l` inspects the latter number, and essentially decides to play it safe and build NCCL serially.

The end result?  My e2e clean build time is 17m50s, but if I remove `-l` it drops to 6m20s.  (This is with no particular build time optimizations either!)

I'd like to find a way to keep the OOM-avoidance properties of `-l` for users that need them, while not unnecessarily penalizing developers with big, noisy boxes (i.e. most Meta engineers :-p )

cc @malfet @seemethere @peterbell10 @huydhn since y'all have worked on this area in the past :-)
Best practices clarification for initialization strategies
Labels: module: nn, triaged, module: initialization, needs research, module: python frontend

### 📚 The doc issue

PyTorch provides a few initialization strategies. However, changing the default (Xavier), while made easier via the `nn.init` submodule, can result in code duplication. Is it worthwhile to introduce decorators to more easily define new layers / redefine existing layers?

```python
import torch.nn as nn
from typing import (TypeAlias, List, TypeVar)

TorchLayer: TypeAlias = nn.Module


def apply_init_to_layer(
    torch_layer:TorchLayer, init_func:Callable[[TorchLayer], TorchLayer],
    *args, **kwargs
):
    layer = torch_layer(*args, **kwargs)
    layer.apply(init_func)
    return layer

def apply_kaiming_init(layer:TorchLayer):
    for name, param in layer.named_parameters():
        if 'weight' in name:
            nn.init.kaiming_normal_(param.data)
        elif 'bias' in name:
            param.data.fill_(0)

def apply_xavier_init(layer:TorchLayer):
    for name, param in layer.named_parameters():
        if 'weight' in name:
            nn.init.xavier_normal_(param.data)
        elif 'bias' in name:
            param.data.fill_(0)

def kaiming_init(torch_layer:TorchLayer) -> Callable[..., TorchLayer]:
    def init_layer(*args, **kwargs) -> TorchLayer:
        layer = apply_init_to_layer(torch_layer, apply_kaiming_init, *args, **kwargs)
        return layer
    return init_layer
```

with existing layers:

```python
kLSTM = kaiming_init(nn.LSTM)
kGRU = kaiming_init(nn.GRU)
kLinear = kaiming_init(nn.Linear)
```



### Suggest a potential alternative/fix

See above

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
DISABLED test_Conv2d_dilated_cuda_tf32 (__main__.TestNN)
Labels: module: nn, module: cuda, triaged, module: flaky-tests, skipped

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_Conv2d_dilated_cuda_tf32&suite=TestNN) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/undefined).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 2 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_Conv2d_dilated_cuda_tf32`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_nn.py`

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @ptrblck
Exporting the operator \'aten::fused_moving_avg_obs_fake_quant\' to ONNX opset version 13  is not supported
Labels: module: onnx, triaged

### 🚀 The feature, motivation and pitch

May i know if there is a way to fix this or when it will be added, im trying to export quantized model but it will always fail due to that ONNX opset V13 doesnt support fused_moving_avg_obs_fake_quant

### Alternatives

_No response_

### Additional context

_No response_
Fix dynamo-related debug Python 3.11 failures
Labels: triaged, bug, release notes: dynamo

Multiple `PYTORCH_TEST_WITH_DYNAMO=1` tests raise asserts on debug Python 3.11 or otherwise fail. These should be fixed.

Script to run all tests: https://gist.github.com/williamwen42/c430d1a5667c270c209be51b24490d6e (Each individual test will stop running at the first abort/segfault. To continue, skip the failing test.)

Tests to fix:

- [ ] `python test/test_optim.py -k test_adagrad_sparse` (`Python/ceval.c:2431: _PyEval_EvalFrameDefault: Assertion 'EMPTY()' failed.`)

For reference, these tests also fail without `PYTORCH_TEST_WITH_DYNAMO=1` on 3.11:

- [ ] `python test/test_tensor_creation_ops.py -k test_non_writable_buffer_cpu_bool` (`Fatal Python error: _Py_CheckSlotResult: Slot getbuffer of type bytes succeeded with an exception set`) (debug 3.11 only) (also fails in debug 3.10)
- [ ] `python test/functorch/test_ops.py -k test_vmapvjpvjp_linalg_tensorsolve_cpu_float32` (`AssertionError: Tensor-likes are not close!`) (debug 3.11 only)
- [ ] Tests `test_bool_indices_cpu` and `test_bool_indices_cuda` in `python test/test_indexing.py` (`AssertionError: Scalars are not equal!`) (only occurs when the entire test suite is run -- running individually passes, fails locally on release 3.11, passes in CI) (https://github.com/pytorch/pytorch/issues/103355)
- [ ] `python test/test_ops_gradients.py -k test_fn_grad_linalg_det_singular_cpu_float64` (`torch.autograd.gradcheck.GradcheckError: Jacobian mismatch for output 0 with respect to input 0,`) (debug 3.11 only)
- [ ] `python test/test_sparse_csr.py -k test_invalid_input_csr_large_cpu` (`AssertionError: "32-bit integer overflow in nnz" does not match "value cannot be converted to type int32 without overflow"`) (skipped in CI)
Investigate the perf drop on timm for dynamic shape when layout optimization is enabled
Labels: triaged, topic: performance, oncall: pt2, module: inductor

### 🚀 The feature, motivation and pitch

We see 14% perf win on timm models with layout optimization. But enabling dynamic shape cause 6% perf drop instead.

Theoretically we should also see wins with dynamic shape.

I've done some early investigation on gernet_l.

Here are the perf numbers:
static shape + layout opt: 1.474x
static shape without layout opt: 1.165x

dyn shape + layout opt: 0.912x
dyn shape without layout opt: 1.102x


Dive into the dynamic shape runs further.
For dyn shape + layout opt
- Abs latency: 80.241734 ms
- Fwd wrapper: https://gist.github.com/shunting314/147285c6ba55576ad528b6bc3c02aff7
- Fwd wrapper profiling result: ​​https://gist.github.com/shunting314/dcb6d8a80739487a8a782f84a3c01976 
- Fwd wrapper total wall time 28.636 ms
- Bwd wrapper: https://gist.github.com/shunting314/6218248272f9519d497a91403b676103 
- Bwd wrapper profiling result: https://gist.github.com/shunting314/d7afef45fcfa7e5b7b776b61ed9fa7a2 
- Bwd wrapper Total wall time 49.491 ms

For dyn shape without layout opt:
- Abs latency: 66.486952 ms
- Fwd wrapper: https://gist.github.com/shunting314/f0a0c038850a94283ae3a87975e75e2b 
- Fwd wrapper profiling result: https://gist.github.com/shunting314/e649dc72df2c9a247952d91e60a8de10
- Fwd wrapper Total wall time 20.838 ms
- Bwd wrapper: https://gist.github.com/shunting314/5dbc4ae04c1b8c695d31651af6185509 
- Bwd wrapper profiling result: https://gist.github.com/shunting314/41adfa37fd1cde3600f40ffebe4532fc
- Bwd wrapper Total wall time 44.032 ms 

We can see that with layout opt, overall we slow down by ~14ms. While the fwd wrapper slows downs by ~8ms and bwd wrapper slows down by ~5.5 ms. I'll dive into the fwd wrapper as the next step.

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @jansel @Chillee @eellison 



### Alternatives

_No response_

### Additional context

_No response_