gopikrsmscs/torch-issues huggingface dataset:
Option to bypass NOLA check in torch.istft
Labels: triaged, module: fft

### 🚀 The feature, motivation and pitch

The proposal is to add an option to skip the NOLA check during `istft` computation as the NOLA check causes host-device synchronization. 

This has been discussed in other issues regarding reconstruction with certain windows when `center=False` ( #62323, #91309), but my proposal is from a performance standpoint (https://discuss.pytorch.org/t/torch-istft-nola-check-causes-synchronization-and-massive-slow-down/172430).

 I understand that NOLA check is necessary for invertibility, but if we are certain about the NOLA condition then we should ideally have the option to bypass the check (and subsequent synchronization) in the `istft` function.

### Alternatives

_No response_

### Additional context

Below is the profiler trace for `istft`. As you can see, it spends almost the entire time in synchronization. While for standalone usage it might not be much of a problem, in the case when `istft` is part of a neural network, the synchronization really slows down the computations during training. 

![image](https://user-images.githubusercontent.com/12119806/218385260-06bc264d-c78d-45ed-8e5b-774c748aea48.png)

The above trace was obtained with the following code:

```
import torch
from torch.profiler import profile, ProfilerActivity

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

dummy_x = torch.randn(256, 7*44100, device=device)
dummy_spec = torch.randn(256, 2049, 302, device=device, dtype=torch.cfloat)

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
             on_trace_ready=torch.profiler.tensorboard_trace_handler('./log'),
             schedule=torch.profiler.schedule(
                 wait=5,
                 warmup=1,
                 active=1),
             profile_memory=True,
             with_stack=True,
             record_shapes=True
             ) as prof:
    for _ in range(8):
        ispec = torch.istft(dummy_spec, n_fft=4096, hop_length=1024, length=dummy_x.shape[-1])
        prof.step()

```


cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano @peterbell10
Investigate queue disparity between `windows.4xlarge` and `linux.4xlarge`
Labels: high priority, module: ci, triaged, module: regression

### 🐛 Describe the bug

See graph below: in last 7 day there were rarely any queueing for `linux.4xlarge` instances, but `windows.4xlarge` seems hard to come by:
<img width="690" alt="image" src="https://user-images.githubusercontent.com/2453524/218336344-46c3d0a1-9757-4670-a2d6-6244b2f5b291.png">

But according to https://github.com/pytorch/test-infra/blob/90ee2adb46a542babe3f8c016b06b135f211e6b8/.github/scale-config.yml they are allocated on a very similar resources: `c5.4xlarge` vs `c5d.4xlarge`. 
Two questions:
 - Can they be move to the same resources? (in case `c5d.4xlarge` are harder to come by)
 - Does our scaling algorithm prefers Linux to Windows?(as one can also observe by observing queueing times for `linux.g5.4xlarge.nvidia.gpu` vs `windows.g5.4xlarge.nvidia.gpu`)


cc @ezyang @gchanan @zou3519 @seemethere @pytorch/pytorch-dev-infra @jeanschmidt
`where` triggers INTERNAL ASSERT FAILED when `out` is a long tensor due to mixed types
Labels: module: error checking, triaged, module: type promotion

### 🐛 Describe the bug

`where` triggers INTERNAL ASSERT FAILED when `out` is a long tensor due to mixed types

```py
import torch

a = torch.ones(3, 4)
b = torch.zeros(3, 4)
c = torch.where(a > 0, a, b, out=torch.zeros(3, 4, dtype=torch.long))
# RuntimeError: !needs_dynamic_casting<func_t>::check(iter) INTERNAL ASSERT FAILED 
# at "/opt/conda/conda-bld/pytorch_1672906354936/work/aten/src/ATen/native/cpu/Loops.h":308, 
# please report a bug to PyTorch.  
```

### Versions

```
PyTorch version: 2.0.0.dev20230105
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.9.15 (main, Nov 24 2022, 14:31:59)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 515.86.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==2.0.0.dev20230105
[pip3] torchaudio==2.0.0.dev20230105
[pip3] torchvision==0.15.0.dev20230105
[conda] blas                      1.0                         mkl
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py39h7f8727e_0
[conda] mkl_fft                   1.3.1            py39hd3c417c_0
[conda] mkl_random                1.2.2            py39h51133e4_0
[conda] numpy                     1.23.5           py39h14f4228_0
[conda] numpy-base                1.23.5           py39h31eccc5_0
[conda] pytorch                   2.0.0.dev20230105 py3.9_cuda11.7_cudnn8.5.0_0    pytorch-nightly
[conda] pytorch-cuda              11.7                 h67b0de4_2    pytorch-nightly
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] torchaudio                2.0.0.dev20230105      py39_cu117    pytorch-nightly
[conda] torchtriton               2.0.0+0d7e753227            py39    pytorch-nightly
[conda] torchvision               0.15.0.dev20230105      py39_cu117    pytorch-nightly
```

cc @nairbv @mruberry
Nan is output by GRU on mps
Labels: triaged, module: mps

### 🐛 Describe the bug

When I use the mps it turns into nan values for just a simple encoder similar to the tutorial on PyTorch.org. This was after I tried converting the tensors to float32.
```
class Encoder(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers=1):
        super(Encoder, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.embedding = nn.Embedding(input_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size, num_layers, batch_first=True)
    
    def forward(self, x, hidden=None):
        embedded = self.embedding(x)
   
        output, hidden = self.gru(embedded, hidden)
        #the output and the hidden is nan
        return output, hidden
```

I just got the input that is basically a tokenized text from the basic_english and build vocab from iterator in torch text library.  The embedded works fine but after I feed the embedding into the GRU it just turns into nan values.

### Versions

This uses torch text and torch library. 

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
Missing FX documents for some modules
Labels: module: docs, triaged

### 📚 The doc issue

It seems some submodules of torch.fx are missing from the documentation.

The rst file https://github.com/pytorch/pytorch/blob/master/docs/source/fx.rst intends to include the following modules:
.. py:module:: torch.fx.passes
.. py:module:: torch.fx.passes.infra
.. py:module:: torch.fx.passes.backends
.. py:module:: torch.fx.passes.utils
.. py:module:: torch.fx.passes.tests
.. py:module:: torch.fx.experimental
.. py:module:: torch.fx.experimental.unification
.. py:module:: torch.fx.experimental.unification.multipledispatch
.. py:module:: torch.fx.experimental.migrate_gradual_types
.. py:module:: torch.fx.passes.dialect
.. py:module:: torch.fx.passes.dialect.common

While currently the doc https://pytorch.org/docs/stable/fx.html does not contain these submodules.

### Suggest a potential alternative/fix

Maybe there are something wrong with the rst file.

cc @svekars @carljparker
dynamo: handle contiguous graph breaks 
Labels: feature, triaged, oncall: pt2, module: dynamo, module: graph breaks

### 🚀 The feature, motivation and pitch

Currently, we create a `function call to traced code` and a `resume at call` in a graph break section every time we encounter an instruction we cannot handle within a `convert_frame`.

![IDE construct a function call around our non-inlineable function](https://user-images.githubusercontent.com/9093549/218234288-bd5a86c3-f584-439f-8740-e6b8aab03a5f.png)

I believe this could result in unnecessarily slow code if one has a lot of graph breaks spaced closely together or which are contiguous.

Instead of eagerly creating a `resume at call`, we should lazily attempt to continue to trace the program. Until we hit another instruction that is explicitly a torch op, we should not create a `resume at` instruction immediately. When we do hit a torch op, we can trigger a `resume at`.

Essentially, we will avoid the situation where we have many graph breaks with trivial `output` functions.

Possibly, this could also help us move control-flow not dependent on tensors into graph breaks. We could possibly reduce the number of guards required in the partial graphs this way.

To implement this, the `InstructionTranslator` should be stateful. When in graph break mode, it should keep tracing if encountering graph breaks, but create a resume at if encountering torch ops. When in torch tracing mode, it should switch modes to graph break mode if encountering a graph break. `resume` at is already capable of recreating the stack for variables it references, so I don't think we need to worry too much.

### Alternatives

_No response_

### Additional context

This is an optimization and hence lower priority. We should only investigate further if we do in fact see contiguous graph breaks appearing in user code and they are a substantial source of slowness in execution.

It is unclear if there is even a high performance penalty. However, note that a graph break is considerably more expensive than ordinary function call, things like the `SETUP WITH` and other variables need to be setup and torn down on each graph break.

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
[RFC] Add a static_graph mode for FSDP
Labels: oncall: distributed, triaged, module: fsdp

### 🚀 The feature, motivation and pitch

In most of workloads, the graph is static, and a lot of optimizations like explicit prefetch, execution order based wrapping are based on assumption that graph is static, we can add a static_graph flag similar to vanilla DDP API, if users explicitly specify the graph is static, FSDP can turn on all the related optimizations in default. 

This flag may be helpful for FSDP + Dynamo mode as well to avoid unneeded recompilation

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Saving a `torch.nn.HuberLoss` using `torch.jit.script().save()` doesn't seem to implicitly convert from `int` type to `float` type.
Labels: oncall: jit

### 🐛 Describe the bug

Saving a `torch.nn.HuberLoss(delta=1)` crashes with a `RuntimeError` which is actually a type error, but saving a `torch.nn.HuberLoss(delta=1.1)` works fine. The problem is that Pytorch complaints that 1 is an integer and not a float. I would expect that Pytorch would know to implicitly convert argument `delta` to a float all the time. The user (programmer) shouldn't be expected to explicitly type 1.0 (as opposed to just 1) in Python 3.

To reproduce:

```
loss_module = torch.nn.HuberLoss(delta=1)
torch.jit.script(loss_module).save("Drive:\Path\To\LossModule.pt")
```

Error message:

```
RuntimeError:

huber_loss(Tensor input, Tensor target, str reduction="mean", float delta=1.) -> Tensor:
Expected a value of type 'float' for argument 'delta' but instead found type 'int'.
:
  File "C:\Users\raaaa\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\loss.py", line 988
    def forward(self, input: Tensor, target: Tensor) -> Tensor:
        return F.huber_loss(input, target, reduction=self.reduction, delta=self.delta)
               ~~~~~~~~~~~~ <--- HERE
```

I'm guessing this is a bug because I can't imagine this is the desired behaviour?

### Versions

Collecting environment information...
PyTorch version: 1.13.0+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Pro
GCC version: (x86_64-posix-seh, Built by strawberryperl.com project) 8.3.0
Clang version: Could not collect
CMake version: version 3.20.2
Libc version: N/A

Python version: 3.10.1 (tags/v3.10.1:2cd268a, Dec  6 2021, 19:10:37) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22621-SP0
Is CUDA available: True
CUDA runtime version: 11.7.64
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3080
Nvidia driver version: 528.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
`permute` for named tensors
Labels: triaged, module: named tensor

### 🚀 The feature, motivation and pitch

I called `tensor.permute(...)` and got:

> RuntimeError: aten::permute is not yet supported with named tensors. Please drop names via `tensor = tensor.rename(None)`, call the op with an unnamed tensor, and set names on the result of the operation.

I really would have expected that named tensors would work for such a fundamental low-level function, esp since named tensors have been around for many years now. Are they actually used? (#60832)

### Alternatives

_No response_

### Additional context

_No response_

cc @zou3519
Abort Caused by Virtual Function
Labels: module: build, triaged, module: regression

### 🐛 Describe the bug

`Abort` happens when PyTorch releasing its resources (i.e., when Python terminates) in

```cpp
 struct TORCH_API OperatorKernel : public c10::intrusive_ptr_target {
-  // Abort triggered.
-  ~OperatorKernel() override = default;
+  // Old thing works well.
+  virtual ~OperatorKernel() = default;
 };
```

After reverting `f172feae0d6e6e510d2133ed10dd76dbcaf0f0fe` (#93069), it works again. My build command is `DEBUG=1 VERBOSE=1 USE_KINETO=0 TORCH_CUDA_ARCH_LIST="7.0;7.2" PATH=/cuda-11.8/lib64:/cuda-11.8/include:/cuda-11.8/bin:$PATH CUDACXX=/cuda-11.8/bin/nvcc ONNX_NAMESPACE=onnx1 BUILD_SHARED_LIBS=1 BUILD_CAFFE2=0 BUILD_CAFFE2_OPS=0 USE_GLOO=1 USE_NCCL=0 USE_NUMPY=1 USE_OBSERVERS=1 USE_OPENMP=1 USE_DISTRIBUTED=1 USE_MPI=1 BUILD_PYTHON=1 USE_MKLDNN=0 USE_CUDA=1 BUILD_TEST=1 USE_FBGEMM=1 USE_NNPACK=1 USE_QNNPACK=0 USE_XNNPACK=1 python3 setup.py develop 2>&1 | tee build.log`.

### Versions

Latest master branch.

cc @malfet @seemethere
DISABLED test_pickle_nn_RNN_eval_mode_cuda_float64 (__main__.TestModuleCUDA)
Labels: module: rnn, triaged

Platforms: linux
This test was disabled because it is failing on master ([recent examples](http://torch-ci.com/failure/test_pickle_nn_RNN_eval_mode_cuda_float64%2CTestModuleCUDA)).

cc @zou3519
[mypy] skipping mypy for a few torch/fx and torch/_subclass files
Labels: module: lint, triaged

### 🐛 Describe the bug

In PR https://github.com/pytorch/pytorch/pull/94173, the below files were failing on mypy.  The PR doesn't change these files but probably some import causes mypy to be run on these files and they fail.

Since it is not a regression, the PR now excludes those files from mypy checks. This issue is to track the same.

Files:
```
torch/fx/proxy.py
torch/fx/passes/shape_prop.py
torch/fx/node.py
torch/fx/experimental/symbolic_shapes.py
torch/fx/experimental/proxy_tensor.py
torch/_subclasses/fake_utils.py
torch/_subclasses/fake_tensor.py
```

### Versions

https://github.com/pytorch/pytorch/pull/94173
pybind11 SymNode binding is a footgun py::cast
Labels: triaged, module: pybind

### 🐛 Describe the bug

Say you have a SymNode and you want to convert it into a PyObject. You might try `py::cast` it. But that will give you a `_C.SymNode`; if it was a Python SymNode you wanted it to unwrap directly. Big footgun.

### Versions

master
Document and promise reproducibility torch.randn / torch.rand / torch.randint family behavior on CPU devices
Labels: feature, triaged, module: random

### 🚀 The feature, motivation and pitch

In PyTorch's documentation: https://pytorch.org/docs/stable/notes/randomness.html#reproducibility The reproduciblity of RNG is not guaranteed cross different releases / commits / platforms. While it is difficult to guarantee reproducibility with exotic hardware or GPU, it is beneficial, and practically unchanged for CPU on PyTorch end.

This pitch suggests we revisit some of our CPU implementations, and further promise stability / reproducibility for CPU-based RNG. In the context of where I am coming from (generative AI), what comes to known as "the seed" helps many creators to verify images generated by other creators and to build new work upon that. The noise tensor init from that seed is often small in size (4x64x64) thus performance is not a concern.

#### Counterargument

This pitch will force us to fix on using MT19937 family of RNGs as the starting point. While it is robust, there may be future, better, faster RNGs that better suit.

The community already moved on to use GPU-based RNG, which makes this pitch moot. There is no stability / reproducibility whatsoever with GPU RNGs, making this suggestion a fool.

#### Questions

Upon further investigate current PyTorch implementation, there are some questions on whether the current implementation on CPU is optimal. For example, when number of elements smaller than 16, we currently sample from double precision and then cast back to float when fill in `torch.randn([15], dtype=torch.float)`: https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cpu/DistributionTemplates.h#L192 When there are more than 16 elements, we use float throughout.

### Alternatives

There are a few alternatives:

#### Declare a particular RNG mode that is guaranteed reproducibility cross releases / commits / platforms.

This helps us to continue iterate on main RNG implementation while let whoever wants stability to opt-in. It does incur the cost of maintaining another implementation eventually.

#### Don't guarantee any stability / reproducibility.

Continue doing this will not break things. But the stability / reproducibility is practically guaranteed due to very little change in CPU RNG implementation. This may risk a future when we actually break RNG reproducibility (because we can), there are incompatibility concerns when upgrade.

I won't discuss the situation where we guarantee stability / reproducibility cross hardware, as that may not be practical at all.

### Additional context

_No response_

cc @pbelevich
`get_debug_state` a script function causes INTERNAL ASSERT FAILED
Labels: oncall: jit, triaged

### 🐛 Describe the bug

`get_debug_state` a script function causes INTERNAL ASSERT FAILED

```py
import torch

input = torch.randn(1, 2, 3)

def func(input):
    trace = torch.jit.trace(lambda x: x * x, [input])
    script_fn = torch.jit.script(trace)
    script_fn.get_debug_state()

func(input)

# RuntimeError: optimized_plan_ INTERNAL ASSERT FAILED 
# at "/opt/conda/conda-bld/pytorch_1672906354936/work/torch/csrc/jit/runtime/profiling_graph_executor_impl.cpp":697, 
# please report a bug to PyTorch.
```

### Versions

```
PyTorch version: 2.0.0.dev20230105
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.9.15 (main, Nov 24 2022, 14:31:59)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 515.86.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==2.0.0.dev20230105
[pip3] torchaudio==2.0.0.dev20230105
[pip3] torchvision==0.15.0.dev20230105
[conda] blas                      1.0                         mkl
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py39h7f8727e_0
[conda] mkl_fft                   1.3.1            py39hd3c417c_0
[conda] mkl_random                1.2.2            py39h51133e4_0
[conda] numpy                     1.23.5           py39h14f4228_0
[conda] numpy-base                1.23.5           py39h31eccc5_0
[conda] pytorch                   2.0.0.dev20230105 py3.9_cuda11.7_cudnn8.5.0_0    pytorch-nightly
[conda] pytorch-cuda              11.7                 h67b0de4_2    pytorch-nightly
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] torchaudio                2.0.0.dev20230105      py39_cu117    pytorch-nightly
[conda] torchtriton               2.0.0+0d7e753227            py39    pytorch-nightly
[conda] torchvision               0.15.0.dev20230105      py39_cu117    pytorch-nightly
```

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Exporting the operator 'aten::_transformer_encoder_layer_fwd' to ONNX opset version 13 is not supported
Labels: module: onnx, low priority, triaged, onnx-needs-info

### 🐛 Describe the bug

I just wanted to export to onnx torch.nn.TransformerEncoder, and got this type of error
raise errors.UnsupportedOperatorError(
torch.onnx.errors.UnsupportedOperatorError: Exporting the operator 'aten::_transformer_encoder_layer_fwd' to ONNX opset version 13 is not supported.)

### Versions

numpy==1.24.1
pytorch-lightning==1.9.0
torch==1.13.1
torchaudio==0.13.1
torchdata==0.5.1
torchmetrics==0.11.1
torchvision==0.14.1
[RFC]FSDP API should make limit_all_gathers and forward_prefetch both default to be True
Labels: triaged, module: fsdp

### 🚀 The feature, motivation and pitch

limit_all_gathers=True can avoid over-prefetch when CPU thread is fast; 
forward_prefetch=True can help more prefetch when CPU thread is slow;

so basically we can always explicitly prefetch but with rate limiter;

We probably need to make number of all_gathers be the same for forward_prefetch and limit_all_gathers code paths.

Have both to be True in default should work fine no matter CPU thread is fast or slow, so that users do not tune these by themselves. We can do some experiments to confirm this.

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @rohan-varma @awgu
jacfwd and jacrev are fundamentally broken for complex inputs
Labels: module: autograd, triaged, module: complex, complex_autograd, module: functorch

### 🐛 Describe the bug

Follow up of https://github.com/pytorch/pytorch/issues/90499

Consider a map `f : C -> C`. `f` is called holomorphic if it's complex differentiable. By the Cauchy-Riemann equations), one shows that this is equivalent to it being real differentiable as functions `f : R^2 -> R^2` and their `2 x 2` Jacobian being representable using one complex number. Now, there are functions that are not holomorphic, the simplest of them being `x.conj()`. These are functions whose Jacobian *cannot* be represented using complex numbers. For example the Jacobian of `x.conj()` is given by the matrix `[[1, 0], [0, 1]]`. The Jacobian vector product at a vector `v \in C` is given by `v.conj()`. There is no complex number `z` such that `z*v = v.conj()`.

All this says that `jacfwd` and `jacrev` should *always* return a real Jacobian, as there is no easy test for when a function is holomorphic. More on this a possible APIs moving forward at the end though.

A few examples that are broken ATM.
```python
>>> x = torch.tensor(0.5+1.7j, dtype=torch.complex64)
>>> jacfwd(torch.conj)(x)  # should return tensor([[ 1,  0], [ 0, -1]])
tensor(1.-0.j)
>>> jacrev(torch.conj)(x)  # should return tensor([[ 1,  0], [ 0, -1]])
tensor(1.-0.j)
>>> jacfwd(torch.abs)(x)  # should return  tensor([[0.2822, 0.9594]])
tensor(0.2822)
>>> jacrev(torch.abs)(x)   # almost! Should return tensor([[0.2822], [0.9594]])
tensor(0.2822+0.9594j)
>>> jacfwd(torch.Tensor.cfloat)(torch.ones(()))  # should return  tensor([[1, 0]])
tensor(1.+0.j)
>>> jacrev(torch.Tensor.cfloat)(torch.ones(()))   # should return  tensor([[1], [0]])
tensor(1.)
```

Note that there is no non-constant holomorphic function from `R -> C` or `C -> R` so none of the functions above are holomorphic and PyTorch should never return complex "Jacobians".

There is no easy way to test whether a function is holomorphic, but we do know that a composition of holomorphic functions is holomorphic. As such, a possible API for this would be to have a kwarg `holomorphic: bool = False`  that if set to `True` it tries to return a complex Jacobian if possible. This would be implemented by having a list of functions which are holomorphic, and making sure these are the only ones called in the computaiton of `jacfwd` / `jacrev`.

### Versions

master

cc @ezyang @gchanan @zou3519 @albanD @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @anjali411 @dylanbespalko @mruberry @Chillee @samdow @soumith @kshitij12345 @janeyx99
`func.jacrev()` should be implemented as `func.jacfwd().mT.contiguous()`
Labels: triaged, module: complex, module: functorch

### 🐛 Describe the bug

Forward AD is (should be?) faster to execute theoretically than backward AD, because it does not need to create a graph, save intermediate tensors, etc. Furthermore its formulas are simpler than those for the backward, so they should be faster for that reason as well.

We would also not have issues like https://github.com/pytorch/pytorch/issues/90499. 

### Versions

master

cc @ezyang @anjali411 @dylanbespalko @mruberry @Lezcano @nikitaved @zou3519 @Chillee @samdow @soumith @kshitij12345 @janeyx99
Inconsistent results when using torch.Tensor.bernoulli with float instead of Tensor probabilities
Labels: module: distributions, triaged, module: random, module: determinism

### 🐛 Describe the bug

When using `torch.Tensor.bernoulli` with a float value for `p` the results does not match the results when doing bernoulli manually (see case B) or when using a Tensor storing the probabilities.

I created 4 tests:
- B: manual bernoulli
- C: the test that fails, using `p` as float
- D: using a Tensor for probabilities
- E: same, with inplace bernoulli

```python
import torch

A = torch.zeros(15)
p = 0.75

torch.manual_seed(314159)
R = torch.rand_like(A)
B = (R < p).to(torch.float)
print('B:', B)

torch.manual_seed(314159)
C = A.bernoulli(p)
print('C:', C)

torch.manual_seed(314159)
p_ = torch.ones_like(A) * p
D = p_.bernoulli()
print('D:', D)

torch.manual_seed(314159)
E = A.detach().clone().bernoulli_(p_)
print('E:', E)

print()
print('Summary')
print('B == C', (B == C).all())
print('B == D', (B == D).all())
print('B == E', (B == E).all())

```

Output:
```
B: tensor([1., 0., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
C: tensor([1., 0., 0., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
D: tensor([1., 0., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
E: tensor([1., 0., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

Summary
B == C tensor(False)
B == D tensor(True)
B == E tensor(True)
```
As you can see, the 3rd value in case C is different from the other cases. You can also increase the size of A, same result. If you look at `R`, the 3rd value is not even close to 0.75, so it cannot be a numerical problem.


### Versions

Collecting environment information...
PyTorch version: 1.13.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (GCC) 10.3.0
Clang version: Could not collect
CMake version: version 3.24.2
Libc version: glibc-2.17

Python version: 3.7.12 (default, Feb  6 2022, 20:29:18)  [GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] (64-bit runtime)
Python platform: Linux-3.10.0-1160.76.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
Is CUDA available: False
CUDA runtime version: 11.4.120
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] pytorch-lightning==1.6.4
[pip3] torch==1.13.1
[pip3] torchmetrics==0.9.1
[pip3] torchvision==0.14.1
[conda] Could not collect

cc @fritzo @neerajprad @alicanb @nikitaved @pbelevich @mruberry @kurtamohler
[fx] const_fold.split_const_subgraphs leads to UserWarning
Labels: triaged, module: fx

### 🐛 Describe the bug

```python
import functorch
import torch
from torch.fx.experimental import const_fold

from functools import partial

torch.manual_seed(0)

def fn(x, y):
    z = x + torch.ones_like(x)
    z1 = z.sin().cos().exp().log()
    z2 = z1 * y
    z3 = x + 2 * y
    return z2, z3

x = torch.randn(3, 1)
y = torch.randn(3, 1)

par_fn = partial(fn, x)
graph = functorch.make_fx(par_fn)(y)

mod_folded: const_fold.FoldedGraphModule = const_fold.split_const_subgraphs(graph)
```

Output
```
torch/fx/experimental/const_fold.py:250: UserWarning: Attempted to insert a get_attr Node with no underlying reference in the owning GraphModule! Call GraphModule.add_submodule to add the necessary submodule, GraphModule.add_parameter to add the necessary Parameter, or nn.Module.register_buffer to add the necessary buffer
  new_node = root_const_gm.graph.get_attr(in_node.target)
```

### Versions

master

cc @ezyang @SherlockNoMad @soumith @EikanWang @jgong5 @wenzhe-nrv
QAT + torch.autocast does not work with default settings, missing fused fake_quant support for half
Labels: oncall: quantization, low priority, triaged

### 🐛 Describe the bug

QAT + torch.autocast should be composable.  It currently doesn't work with default settings of QAT:

```
import torch
import torch.nn as nn
from torch.ao.quantization.quantize_fx import prepare_fx
from troch.ao.quantization import get_default_qat_qconfig_mapping                                                   
                                                                                   
m = nn.Sequential(nn.Linear(1,  1)).cuda()                                         
data = torch.randn(1, 1).cuda()                                                    
                                                                                   
# note: setting version to 0, which disables fused fake_quant, works without issues
qconfig_mapping = get_default_qat_qconfig_mapping('fbgemm', version=1)             
mp = quantize_fx.prepare_fx(m, qconfig_mapping, (data,))                           
with torch.autocast('cuda'):                                                       
    res = mp(data)                                                                 
res.sum().backward()                                                               
```

The script above fails with this:

```
Traceback (most recent call last):
  File "/data/users/vasiliy/pytorch/../tmp/test.py", line 23, in <module>
    res = mp(data)
  File "/data/users/vasiliy/pytorch/torch/fx/graph_module.py", line 660, in call_wrapped
    return self._wrapped_call(self, *args, **kwargs)
  File "/data/users/vasiliy/pytorch/torch/fx/graph_module.py", line 279, in __call__
    raise e
  File "/data/users/vasiliy/pytorch/torch/fx/graph_module.py", line 269, in __call__
    return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
  File "/data/users/vasiliy/pytorch/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "<eval_with_key>.2", line 8, in forward
  File "/data/users/vasiliy/pytorch/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/users/vasiliy/pytorch/torch/ao/quantization/fake_quantize.py", line 342, in forw
ard
    return torch.fused_moving_avg_obs_fake_quant(
RuntimeError: expected scalar type Float but found Half
```

Note that using unfused fake_quants works correctly (this can be configured by using `version=0` in `get_default_qat_qconfig_mapping`.  It looks like the cuda kernel for the fused fake quant + observer (`FusedObsFakeQuant.cu`) does not support `torch.half` yet.

### Versions

master

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @jgong5 @Xia-Weiwen @leslie-fang-intel
RFC: Enabling AVX512 dispatch for compute-intensive ATen ops
Labels: module: performance, module: cpu, triaged, module: intel

# 🚀 The feature, motivation and pitch

## Summary
On some more recent x86-64 architectures,  AVX512 performs better than AVX2 on compute-bound workloads, and AVX512 instructions do not cause as much throttling as they did on older CPU architectures. Eager mode performance can thus be improved on such machines.

Targeting for PyTorch 2.1, we propose enabling the dispatch of AVX512 ATen kernels if they'd be _expected_ to perform better than their AVX2 counterparts. The default ATen CPU capability would thus be AVX512.

## Approach

### Naïve solution
First, we would extend & enhance the coverage of individual ATen ops in [OpBench](https://github.com/pytorch/pytorch/tree/master/benchmarks/operator_benchmark), which would offer us more insights into ATen kernel performance characteristics by varying certain factors such as vectorization ISA, thread-pool size, memory layout, dtype, etc. OpBench already benchmarks for various input-sizes. #104655 is an example PR towards this endeavor.

[We are trying to enable AVX512 dispatch for kernels that _always_ perform well with AVX512](https://github.com/pytorch/pytorch/pull/104165). This solution is easier to implement, as it entails only enabling AVX512 dispatch for compute-bound ATen ops, and disabling AVX512 dispatch for memory-bound ATen ops. We would not compile AVX512 kernels of memory-bound ATen ops to reduce the binary-size. This approach is quite restrictive & would disable AVX512 dispatch for most ATen ops. This set of kernels is quite small, as we are aiming for high precision & low recall. 

### Can we do better?
Rather than disable AVX512 dispatch for all kernels that perform poorly with AVX512 in certain cases, we can enable AVX512 dispatch for the cases in which AVX512 performance would be better. We would have to analyze oodles of data gleaned with OpBench (n factors such as CPU generation, dtype, input size, number of threads, etc) to drive this task.

The second solution requires more fine-grained analysis of ATen ops' characteristics, so we are starting with the first one, and plan to gradually move towards the second one.

### Tasks that will be completed soon (have open PRs)
- [ ] Extend OpBench coverage - #104655
- [ ] Enable AVX512 dispatch of those AVX512 ATen kernels that would always perform better than their AVX2 counterparts. - #104165

### Tasks under investigation
- [ ] A new design PoC for AVX-n kernel dispatch.
- [ ] Adoption of new design.

We welcome your comments & are open to changing our approach after discussion. Thanks!

cc @ngimel @jgong5 @mingfeima @XiaobingSuper @ashokei @jingxu10 @frank-wei @malfet 


### Alternatives

_No response_

### Additional context

Currently, the default ATen CPU capability is AVX2 but users can use the environment variable `ATEN_CPU_CAPABILITY=avx512` to use AVX512 ATen kernels
What type of attributes does symbolic function support?
Labels: triaged

### 📚 The doc issue

I read the doc of [symbolic](https://pytorch.org/docs/stable/onnx.html#static-symbolic-method) but no related introduction.

I read the source code and get supported [suffix](https://github.com/pytorch/pytorch/blob/master/torch/onnx/_patch_torch.py#L15), but I still don't know what types each letter represents.

### Suggest a potential alternative/fix

By the way, I want to ask whether symbolic support boolean attributes and integer array attributes?
unsqueeze a single dimension multiple times
Labels: feature, triaged, module: viewing and reshaping

### 🚀 The feature, motivation and pitch

Originally discussed in https://github.com/pytorch/pytorch/issues/30702#issuecomment-570678356. One usecase is that it could be used for (implementation details of) broadcasting right by inserting unitary dimensions `lambda x, y: y.unsqueeze(dim = -1, repeat = x.dim() - y.dim())`

In the wild would be useful e.g. here: https://github.com/pytorch/pytorch/pull/94227#pullrequestreview-1286167590

It already exists in C++: https://github.com/pytorch/pytorch/blob/68f378210666c079326139b6557e8a601fc89ebf/tools/autograd/templates/Functions.cpp#L173

### Alternatives

_No response_

### Additional context

_No response_
Addition of hybrid CSR tensors produces incorrect and invalid CSR tensor
Labels: module: sparse, triaged, module: correctness (silent), bug


## Issue description

As in the title

## Code example

```python
>>> x=torch.ones((2, 2, 2)).to_sparse(layout=torch.sparse_csr, dense_dim=1)
>>> x
tensor(crow_indices=tensor([0, 2, 4]),
       col_indices=tensor([0, 1, 0, 1]),
       values=tensor([[1., 1.],
                      [1., 1.],
                      [1., 1.],
                      [1., 1.]]), size=(2, 2, 2), nnz=4,
       layout=torch.sparse_csr)
>>> x + x
tensor(crow_indices=tensor([0, 2, 4]),
       col_indices=tensor([0, 1, 0, 1]),
       values=tensor([2., 2., 2., 2.]), size=(2, 2, 2), nnz=4,
       layout=torch.sparse_csr)
```

The expected result is
```python
>>> x + x
tensor(crow_indices=tensor([0, 2, 4]),
       col_indices=tensor([0, 1, 0, 1]),
       values=tensor([[2., 2.],
                      [2., 2.],
                      [2., 2.],
                      [2., 2.]]), size=(2, 2, 2), nnz=4,
       layout=torch.sparse_csr)
```

## System Info

- PyTorch: master


cc @alexsamardzic @nikitaved @cpuhrsch @amjames @bhosmer
Addition of CSC/BSR/BSC tensors raises RuntimeError exceptions
Labels: module: sparse, triaged

## Issue description

As in the title.

## Code example

```python
>>> x=torch.ones((2, 2)).to_sparse_csc()
>>> x + x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: col_indices expected sparse row compressed tensor layout but got SparseCsc
>>> x=torch.ones((2, 2)).to_sparse_bsr((1, 1))
>>> x + x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: empty_sparse_compressed expected sparse compressed (non-block) tensor layout but got SparseBsr
>>> x=torch.ones((2, 2)).to_sparse_bsc((1, 1))
>>> x + x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: empty_sparse_compressed expected sparse compressed (non-block) tensor layout but got SparseBsc
```

The addition of CSR tensors works as expected:
```python
>>> x=torch.ones((2, 2)).to_sparse_csr()
>>> x + x
tensor(crow_indices=tensor([0, 2, 4]),
       col_indices=tensor([0, 1, 0, 1]),
       values=tensor([2., 2., 2., 2.]), size=(2, 2), nnz=4,
       layout=torch.sparse_csr)
```

## System Info

- PyTorch: master

cc @alexsamardzic @nikitaved @cpuhrsch @amjames @bhosmer
Addition of batch CSR tensors produces incorrect and invalid CSR tensor
Labels: module: sparse, triaged, module: correctness (silent), bug

## Issue description

As in the title.

## Code example

```python
>>> x=torch.ones((2, 2, 3)).to_sparse_csr()
>>> x
tensor(crow_indices=tensor([[0, 3, 6],
                            [0, 3, 6]]),
       col_indices=tensor([[0, 1, 2, 0, 1, 2],
                           [0, 1, 2, 0, 1, 2]]),
       values=tensor([[1., 1., 1., 1., 1., 1.],
                      [1., 1., 1., 1., 1., 1.]]), size=(2, 2, 3), nnz=6,
       layout=torch.sparse_csr)
>>> y=x.add(x)  # or x + x
>>> y
tensor(crow_indices=tensor([[0, 3, 6],
                            [0, 3, 6]]),
       col_indices=tensor([0, 1, 2, 0, 1, 2]),
       values=tensor([2., 2., 2., 2., 2., 2.]), size=(2, 2, 3), nnz=6,
       layout=torch.sparse_csr)
>>> torch._validate_sparse_csr_tensor_args(y.crow_indices(), y.col_indices(), y.values(), y.shape)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: crow_indices and col_indices dimensionalities must be equal but got 2 and 1, respectively
```

The expected result is
```python
>>> y
tensor(crow_indices=tensor([[0, 3, 6],
                            [0, 3, 6]]),
       col_indices=tensor([[0, 1, 2, 0, 1, 2],
                           [0, 1, 2, 0, 1, 2]]),
       values=tensor([[2., 2., 2., 2., 2., 2.],
                      [2., 2., 2., 2., 2., 2.]]), size=(2, 2, 3), nnz=6,
       layout=torch.sparse_csr)
```

In place addition produces garbage as well:
```python
>>> x.add_(x)
tensor(crow_indices=tensor([[0, 3, 6],
                            [0, 3, 6]]),
       col_indices=tensor([0, 1, 2, 0, 1, 2]),
       values=tensor([2., 2., 2., 2., 2., 2.]), size=(2, 2, 3), nnz=6,
       layout=torch.sparse_csr)
```

## System Info

- PyTorch: master


cc @alexsamardzic @nikitaved @cpuhrsch @amjames @bhosmer
[pt2] The min and max parameters of torch.clamp do not support numpy format
Labels: triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

```python
import torch
import numpy as np


def fn(input):
    max_ratio = np.abs(np.log(4))
    dwh = input.clamp(min=-max_ratio, max=max_ratio)
    return dwh

x = torch.rand([1]).cuda()

ret_eager = fn(x)
print('==== Eager mode OK! ====')

compiled = torch.compile(fn, backend='eager')
ret_compiled = compiled(x)
print('==== torchcomp mode OK! ====')
```

```shell
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument min in method wrapper_CUDA_clamp_Tensor)
```


### Error logs

_No response_

### Minified repro

_No response_

### Versions

Pytorch version: 2.0.0.dev20230131+cu116

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
Faster `pad_sequence` and `tensor_split` function with CUDA kernel, are they possible?
Labels: module: rnn, triaged, oncall: pt2, module: dynamic shapes

### 🚀 The feature, motivation and pitch

I was using `torch.nn.utils.rnn.pad_sequence()` to pad a list of variable length tensors. However, I find the padding operation is not so fast. Although I found that this operation has already been moved to C++ side for shorter run time. It is still slow compared to other operations with CUDA kernel support.

(I briefly profiled the computation to see this. Data size about 4000 1D tensors after spliting, maximum vector length about 200 float64 numbers. Running time without profiler 0.05s.
![image](https://user-images.githubusercontent.com/16524611/216874366-710a703b-e0ae-477b-bb4c-c0b2b80996b5.png)

Is it possible to get a CUDA kernel support version of these operation? I found a possible implementation [here](https://github.com/frostblue-wukong/Paddle-Lite/commit/126691f01f96aa0dbe34417165cb0cb09b7e557a), but I am not handy in CUDA kernel. Could you add a feature for this?

### Alternatives

I have tried pad_sequence and tensor_split with GPU tensors and CPU tensors, and I found that CPU tensors have a shorter runing time. So I guess that C++ side padding and splitting does not have a CUDA kernel yet?

### Additional context

_No response_

cc @zou3519 @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
DISABLED test_index_select_scalar (__main__.TestNLLLoss)
Labels: triaged, module: flaky-tests, skipped, module: mps

Platforms: mac, macos

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/failure/test_index_select_scalar) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/11122655837).

Over the past 72 hours, it has flakily failed in 2 workflow(s).

**Debugging instructions (after clicking on the recent samples link):**
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Grep for `test_index_select_scalar`


Test file path: `test_mps.py`


cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev

```
2023-02-05T21:48:19.5597380Z FAIL [0.002s]: test_index_select_scalar (__main__.TestNLLLoss)
2023-02-05T21:48:19.5597530Z ----------------------------------------------------------------------
2023-02-05T21:48:19.5597590Z Traceback (most recent call last):
2023-02-05T21:48:19.5597760Z   File "/Users/ec2-user/runner/_work/pytorch/pytorch/test/test_mps.py", line 5128, in test_index_select_scalar
2023-02-05T21:48:19.5597800Z     helper(22, 0, [])
2023-02-05T21:48:19.5597950Z   File "/Users/ec2-user/runner/_work/pytorch/pytorch/test/test_mps.py", line 5125, in helper
2023-02-05T21:48:19.5598020Z     self.assertEqual(idx_result, idx_result_cpu)
2023-02-05T21:48:19.5598240Z   File "/Users/ec2-user/runner/_work/_temp/conda_environment_4098394330/lib/python3.9/site-packages/torch/testing/_internal/common_utils.py", line 2926, in assertEqual
2023-02-05T21:48:19.5598290Z     assert_equal(
2023-02-05T21:48:19.5598490Z   File "/Users/ec2-user/runner/_work/_temp/conda_environment_4098394330/lib/python3.9/site-packages/torch/testing/_comparison.py", line 1244, in assert_equal
2023-02-05T21:48:19.5598550Z     raise error_metas[0].to_error(msg)
2023-02-05T21:48:19.5598610Z AssertionError: Scalars are not close!
2023-02-05T21:48:19.5598620Z 
2023-02-05T21:48:19.5598710Z Absolute difference: 0.5 (up to 1e-05 allowed)
2023-02-05T21:48:19.5598810Z Relative difference: 1.0 (up to 1.3e-06 allowed)
```
Asking for a LAZYMODULEMIXIN warning
Labels: module: nn, module: molly-guard, triaged, module: lazy

Hi, I spent several days trying to understand why a gcn model (from pytorch geometric) produced different results on different runs, despite the manual seed setting.

The answer was due to the way lazy modules in PyTorch initialize parameters differently than other modules in a same ML model. (as described here: https://pytorch.org/docs/stable/generated/torch.nn.modules.lazy.LazyModuleMixin.html ). To mitigate this unpleasant situation, could you add a warning about reproducibility of experiments when using a lazy module? Thank you in advance

cc @albanD @mruberry @jbschlosser @walterddr @saketh-are