gopikrsmscs/torch-issues huggingface dataset:
[FSDP] Progress of ParamExecOrderWrapPolicy
Labels: in progress, triaged, module: fsdp

## Description

This issue is used to track the development progress of ParamExecOrderWrapPolicy in PyTorch distributed FSDP, initial PR: #79238.

## Remaining TODOs:

- [ ] Starting from the second iteration, remove all internal FSDP wraps from the root FSDP module. In addition, register all removed FSDP wraps as attributes of the root FSDP. These FSDP wraps will be used to schedule parameter communication and resharding.
- [ ] Implement a helper function that could merge a list of FSDP wraps into one. This allows one to group multiple FSDP wraps and perform communication/resharding together. This allows neighboring modules based on the execution order be wrapped together in the ParamExecOrderWrapPolicy, which is a feather currently not supported in recursive wrapping and has the potential to enhance the performance.
- [ ] Based on FSDP._fsdp_params_exec_order, patch the forward() function (or use forward hook) of each module.

cc @zhaojuanmao @mrshenli @rohan-varma @awgu
Missing the time unit in duration time of DDP logging
Labels: triaged, module: ddp

### 🐛 Describe the bug

The interface _get_ddp_logging_data() misses labeling the time unit of the duration time, such as forward_compute_time, backward_compute_time, backward_comm_time, and backward_compute_comm_overlap_time. The current time unit is ns.

To make the DDP logging data more readable, here are the recommended names according to their actual meanings:
- "forward_compute_time" should be "forward_device_duration_ns"
- "forward_compute_time_start" should be "forward_host_start"
- "backward_compute_time" should be "backward_device_duration_ns"
- "backward_compute_time_start" should be "backward_host_start"
- "backward_compute_time_end" should be "backward_host_end"
- "backward_comm_time" should be "backward_comm_device_duration_ns"
- "backward_comm_time_start" should be "backward_comm_host_start"
- "backward_comm_time_end" should be "backward_comm_host_end"
- "backward_compute_comm_overlap_time" should be "backward_compute_comm_overlap_device_duration_ns"

### Versions

since pytorch 1.9
[FSDP] Verify that FSDP-managed parameters are the same across ranks
Labels: triaged, better-engineering, module: fsdp

Similar to https://github.com/pytorch/pytorch/issues/68803, we can add a check that the FDSP-managed parameters (i.e. all parameters minus the ignored parameters on that rank, which may be different across ranks) are the same across ranks in debug mode.


cc @zhaojuanmao @mrshenli @rohan-varma
[LTC] Introduce a `MetricsReport` python binding and allow backend to add their report as string
Labels: triaged, lazy

### 🚀 The feature, motivation and pitch

`Metrics` and `Counters` classes were originally introduced in https://github.com/pytorch/xla/blob/master/third_party/xla_client/metrics.h and then got upstream to LTC in https://github.com/pytorch/pytorch/blob/master/torch/csrc/lazy/core/metrics.cpp.

As part of the LTC migration, pytorch/xla wants to adapt the upstream `Metrics` class but we run into an issue. Currently pytorch/xla `Metrics` class is part of the `xla_client` which will be built with tensorflow. `xla_client` can not take any dependency to pytorch. `ComputationClinet` in the `xla_client` also needs to use `Metrics` to record some debugging information. I think it will be very hard to get rid of the pytorch/xla version of the `Metrics`.

Given that `xla_client` complications, I think we can still keep 2 `metrics` classes. When `MetricReport`(which we will need to add a binding on the LTC side) is being called in the LTC, LTC will collects all of its counters and metrics. After that is done, LTC can call a backendInterface to get the backend specified `MetricReport` as string and append that to its report.

I think it is actually a good idea since user can easily tell which metrics/counters are backend specified and where to look.

To summary I think we should
1. Add a `MetricsReport` python binding in LTC
2. Add a `GetBackendMetricReport` method to the `BackendInterface`
3. Append `GetBackendMetricReport` at the end of the LTC `MetricsReport`

@Krovatkin @wconstab 

### Alternatives

have two API one only collect metrics reports for LTC and another one only for xla

### Additional context

_No response_
__torch__dispatch does not return new output in inplace function
Labels: triaged, module: __torch_dispatch__

### 🐛 Describe the bug

Even though we are returning a `ModeTensor` in `x.add_(1)` below, the output is still the input `torch.rand([4])` which is not a `ModeTensor`.

```
import torch
from torch.utils._python_dispatch import TorchDispatchMode

class ModeTensor(torch.Tensor):
    def __new__(cls, elem, mode):
        r = torch.Tensor._make_subclass(cls, elem, elem.requires_grad)
        r.elem = elem
        r.mode = mode
        return r

    def __torch_dispatch(self, func, types, args=(), kwargs=None):
        with self.mode:
            return func(*args, **kwargs)

class Mode(TorchDispatchMode):
    def __torch_dispatch__(self, func, types, args=(), kwargs=None):
        def unwrap(e):
            if isinstance(e, ModeTensor):
                return e.elem
            else:
                return e

        def wrap(t):
            if isinstance(t, torch.Tensor):
                return ModeTensor(t, self)
            else:
                return t

        return wrap(func(*tuple(unwrap(a) for a in args), **kwargs))

x = torch.rand([4])
with Mode():
    out_func = x.add(1)
    out_inplace = x.add_(1)

print(type(out_func), out_inplace)
# <class '__main__.ModeTensor'> tensor([1.2925, 1.5918, 1.2033, 1.9141])
```

### Versions

master

cc @Chillee @ezyang @zou3519 @albanD @samdow
test_ops.py extremely slow on cuda11.3
Labels: triaged

Since the merge of #78304, our Linux 11.3 CUDA tests have increased in test time by 3 hours. @0x00b1's PR introduces several OpInfos that are culprit.

![image](https://user-images.githubusercontent.com/31798555/173619363-3ddd8433-30f9-4c16-b8da-5e8440e0011d.png)
Before the change, our total test time was averaging around 7.5 hrs. After, it was consistently 10.5hrs.
Display a "reference" link for ops that points to primTorch implementations
Labels: module: docs, triaged, better-engineering, module: primTorch

Users have often requested a more readable way to understand PyTorch's operators. See https://github.com/pytorch/pytorch/pull/79413#issuecomment-1153988754 for a recent reiteration of this suggestion. 

cc @svekars @holly1238 @ezyang @mruberry @ngimel
DISABLED test_checkpoint_wrapper_parity (__main__.CheckpointWrapperTest)
Labels: triaged, module: flaky-tests, skipped, module: fsdp

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_checkpoint_wrapper_parity&suite=CheckpointWrapperTest&file=distributed/fsdp/test_checkpoint_wrapper.py) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/6872974643).

Over the past 3 hours, it has been determined flaky in 13 workflow(s) with 13 red and 13 green.

cc @zhaojuanmao @mrshenli @rohan-varma
DISABLED test_caching_pinned_memory (__main__.TestCuda)
Labels: module: cuda, triaged, module: flaky-tests, skipped

Platforms: linux, rocm

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_caching_pinned_memory&suite=TestCuda&file=test_cuda.py) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/6872881686).

Over the past 3 hours, it has been determined flaky in 1 workflow(s) with 1 red and 1 green.

cc @ngimel
Implement NestedTensor size function
Labels: module: nestedtensor, oncall: transformer/mha

### 🚀 The feature, motivation and pitch

NestedTensor does not implement a size method. NestedTensor has been integrated into nn.Transformer, which has revealed a couple of use cases where the sizes of intermediate tensors in the model are needed.

The three main cases are:
- PyTorch profiler fails on nn.Transformer because it attempts to find the size of all intermediate tensors, which includes NestedTensors in the transformer. 
- Autograd with NestedTensor fails because it requires the size of the tensor
- Torchscript with NestedTensor fails because torch.jit.script(model) attempts to find the size of intermediate tensors for performance optimizations.

### Alternatives

Two main ways to fix:
1. In all code that tries to find size of intermediate tensors, check for NestedTensor, and if it's a NestedTensor, do not ask for size. This seems feasible for the above use cases (profiler: has a WIP patch to check for NestedTensor, autograd: WIP in https://github.com/pytorch/pytorch/issues/79039, torchscript: could probably make a torchscript-specific patch). But I imagine that there are many such use cases and it would be hard to cover them all.  
2. Implement NestedTensor size method in a way that does not break existing use cases. 

### Additional context

_No response_

cc @cpuhrsch @jbschlosser @bhosmer @erichan1
[META] Sign up to discuss significantly modifying CI
Labels: module: ci, triaged

Our CI currently tests tens of thousands of tests across many different platforms and compilers. As PyTorch grows and different modules would like to add to our CI, we should meet and discuss the added value of increased testing vs our constraints of CI capacity and time to signal (TTS). We should also discuss any changes that may largely affect CI.

Please sign up for a slot (Tuesdays 4:05-5:00pm ET) below! Please add a topic and an RFC/document that should be prepared ahead of time so we can spend more time in discussion. The PyTorch Dev Infra team will be de-facto attendees. **Please include emails so the invites could be sent out accordingly.**

### 6/14/22 [EXAMPLE]
**Topic**: Let's Add More Fun(c) to CI
**Presenter(s)**: Firstname Lastname (presenter@email.com)
**RFC/Document**: https://github.com/pytorch/pytorch/issues/78082 is a good example
**Invitees/Attendees**: Team MemberA (teammateA@email.com), Team MemberB (teammateB@email.com), Module ExpertA (expertA@email.com), etc..

### 6/21/22
**Topic**: 
**Presenter(s)**: 
**RFC/Document**: 
**Invitees**: 

### 7/12/22
**Topic**:
**Presenter(s)**: 
**RFC/Document**: 
**Invitees**: 

### 7/26/22
**Topic**:
**Presenter(s)**: 
**RFC/Document**: 
**Invitees**:


cc @seemethere @malfet @pytorch/pytorch-dev-infra
Ensure the guards in distributed_c10d.py wrappers get executed in the replay of the graph
Labels: oncall: distributed, triaged

Ensure the guards in distributed_c10d.py wrappers get executed or reassured in the replay of the captured graph in any graph mode. A follow up on #76722.

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
Add autograd support for dispatch passable c10d ops
Labels: oncall: distributed, triaged

Add autograd support for dispatch passable c10d ops as a follow up on #76722. It can probably reuse the implementation here: https://github.com/pytorch/pytorch/blob/master/torch/distributed/nn/functional.py.

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
Iteration # 1-offset in DDP logging
Labels: oncall: distributed, triaged, module: ddp

### 🐛 Describe the bug

When using the _get_ddp_logging_data() to collect logging data, the iteration# has 1-offset. Starting from the second iteration, DDP logging data are always 1 iteration offset. 

This table shows the results when comparing the DDP logging data with the pytorch profiler traces. e.g., the iteration 2 of DDP logging data is 20.5 ms, which is very close to the iteration 1 of profiler traces that is 20.6 ms. 

Iteration \ Time (ms) | Forward_compute_time (DDP) | Forward time in profiling
-- | -- | --
1 | 1953.336832 | 20.661
2 | 20.523072 | 22.753
3 | 23.968768 | 28.798
4 | 29.75024 | 18.042
5 | 18.861792 | 40.775
6 | 41.697664 |  

</div></b>

Here is the sample code of Resnet18:

      import os
      import socket
      import sys
      import logging
      import time
      from collections import OrderedDict
      import torch
      import torch.distributed as dist
      import torch.nn as nn
      from torch.autograd.profiler import record_function
      import torchvision
      from torchvision import models, transforms
  
      def run_worker(rank, world_size):

        torch.cuda.set_device(rank)
        dist.init_process_group("nccl", rank=rank, world_size=world_size)
  
        model = models.resnet18(pretrained=False).cuda()
        ddp_model = nn.parallel.DistributedDataParallel(
            module=model,
            device_ids=[rank],
        )
        # Define loss function, input, labels, and optimizer.
        loss_fn = nn.MSELoss()
        input = torch.randn(10, 3, 224, 224).cuda()
        labels = torch.randn(10, 1000).cuda().to(rank)
        optimizer = torch.optim.SGD(ddp_model.parameters(), lr=0.001)
        
        with torch.profiler.profile(
            activities=[torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA],
            on_trace_ready=torch.profiler.tensorboard_trace_handler("/tmp/traces"),
            record_shapes=True,
        ) as prof:
            for i in range(0, 10):
                with record_function(f"#Step {i}"):
                    optimizer.zero_grad()
                    with record_function("#forward"):
                        output = ddp_model(input)
                    loss = loss_fn(output, labels).half().cuda()
                    with record_function("#backward"):
                        loss.backward()
                    optimizer.step()
                    ddp_logging = ddp_model._get_ddp_logging_data()
                    print(ddp_logging)
        # Cleanup.
        dist.destroy_process_group()
      
      if __name__ == "__main__":
        os.environ["MASTER_ADDR"] = "localhost"
        os.environ["MASTER_PORT"] = "12345"
        world_size = torch.cuda.device_count()
        mp.spawn(run_worker, args=(world_size,), nprocs=world_size, join=True)`

### Versions

since pytorch 1.9

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
[BE][ZeRO] Enable multigpu unit tests
Labels: oncall: distributed, triaged, better-engineering

Following https://github.com/pytorch/pytorch/pull/77947, we should enable `ZeroRedundancyOptimizer` multigpu unit tests.

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
[LTC] Make `torch::lazy::BackendImplInterface::ExecuteComputation` takes `ComputationPtr` instead of `Computation`
Labels: triaged, lazy

### 🚀 The feature, motivation and pitch

PyTorch/XLA is adapting to upstream LTC. In https://github.com/pytorch/xla/pull/3627 I am trying to implement the reaming of the backend interface. The issue I run into right now is `ExecuteComputation` currently takes `Computation`. The call is coming from 

https://github.com/pytorch/pytorch/blob/30fb2c4abaaaa966999eab11674f25b18460e609/torch/csrc/lazy/core/lazy_graph_executor.cpp#L964-L967

The issue here is that PyTorch/XLA inherits Computation and defined `torch_xla::Computation` in 
https://github.com/pytorch/xla/blob/a5d83f412ca603cb6aa16cc86c77fde8f05aabbd/torch_xla/csrc/computation.h#L34

The request is for `ExecuteComputation` to take `ComputationPtr`(which is what `async->cached_computation->computation` directly).

### Alternatives

I don't see any alternative.

### Additional context

_No response_
Use c10d broadcast_object in Zero
Labels: oncall: distributed, module: bootcamp, good first issue, triaged, better-engineering, pt_distributed_rampup

### 🚀 The feature, motivation and pitch

The current implementation of Zero Redundancy optimizer has its [own implementation](https://github.com/pytorch/pytorch/blob/master/torch/distributed/optim/zero_redundancy_optimizer.py#L73) of object broadcasting.

We should replace it with c10d [broadcast_object_list](https://pytorch.org/docs/stable/distributed.html#torch.distributed.broadcast_object_list).


To verify the result, run the tests in `test/distributed/optim/test_zero_redundancy_optimizer.py`.

### Alternatives

_No response_

### Additional context

_No response_

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
Guards for a linked list will be `O(n^2)`
Labels: module: internals, triaged, enhancement, oncall: pt2

Suppose the user creates a linked list like data structure.  The guards might be something like:
```
head.next.val == 1 and
head.next.next.val == 2 and
head.next.next.next.val == 3 and
head.next.next.next.next.val == 4 and
head.next.next.next.next.next.val == 5 and
...
```

Since each line in the guards starts from `f_locals`, accessing the nth item will be `O(n)` and the total guards will be `O(n^2)`.  Something similar might also happen for deep `nn.Module` hierarchies.

We should rewrite how do codegen for guards to include [common subexpression elimination](https://en.wikipedia.org/wiki/Common_subexpression_elimination), which will fix this issue.  We also consider implementing guards with C++ as they are a potentially hot part of the code.



cc @ezyang @bhosmer @smessmer @ljk53 @bdhirsh @soumith @msaroufim @wconstab @ngimel
Add doc formatting check to lintrunner
Labels: module: lint, triaged, better-engineering

From @mruberry: Docs are difficult to build locally (our documentation for how to do so is often out of date) and formatting errors in CI appear without line numbers (in at least some cases). Some recent example errors/warnings:

```
/opt/conda/lib/python3.7/site-packages/torch/__init__.py:docstring of torch.set_float32_matmul_precision:22: ERROR: Unexpected indentation.
/opt/conda/lib/python3.7/site-packages/torch/__init__.py:docstring of torch.set_float32_matmul_precision:23: WARNING: Block quote ends without a blank line; unexpected unindent.
```

If lintrunner could detect doc formatting issues it would making adding/updating docs much easier.
Conda enviroment
Labels: triaged

### 🐛 Describe the bug

Traceback (most recent call last):
  File "/data1/yvr/.conda/envs/opt/bin/opt-baselines", line 33, in <module>
    sys.exit(load_entry_point('metaseq==0.0.1', 'console_scripts', 'opt-baselines')())
  File "/data1/yvr/.conda/envs/opt/lib/python3.7/site-packages/metaseq-0.0.1-py3.7-linux-x86_64.egg/metaseq/launcher/opt_baselines.py", line 314, in cli_main
    get_grid, postprocess_hyperparams, add_extra_options_func=add_extra_options_func
  File "/data1/yvr/.conda/envs/opt/lib/python3.7/site-packages/metaseq-0.0.1-py3.7-linux-x86_64.egg/metaseq/launcher/sweep.py", line 378, in main
    backend_main(get_grid, postprocess_hyperparams, args)
  File "/data1/yvr/.conda/envs/opt/lib/python3.7/site-packages/metaseq-0.0.1-py3.7-linux-x86_64.egg/metaseq/launcher/slurm.py", line 34, in main
    grid = get_grid(args)
  File "/data1/yvr/.conda/envs/opt/lib/python3.7/site-packages/metaseq-0.0.1-py3.7-linux-x86_64.egg/metaseq/launcher/opt_baselines.py", line 78, in get_grid
    raise RuntimeError("Where are you running this?! Check DATA_LOCATIONS.")
RuntimeError: Where are you running this?! Check DATA_LOCATIONS.

### Versions

After I conda environment following the setup.md,I plan to follow the train.md,it comes to the bug?
SymInt equality tests are unsound
Labels: triaged

### 🐛 Describe the bug

Current implementation is  

```
  bool operator==(const SymInt& p2) const {
    return data_ == p2.data_;
  }

  bool operator!=(const SymInt& p2) const {
    return data_ != p2.data_;
  }

```

This is unsound. Two sym ints can be referencing different symbolic variables but after guarding on their concrete values we may discover they are equal and == should return true. These need to be virtualized like the rest of the operators on SymInt.

@Krovatkin 

### Versions

master
caffe2_nvrtc is produced even when it won't be used
Labels: module: build, triaged, module: selective build

### 🐛 Describe the bug

The library is produced whenever CUDA is enabled. However, in the code, lazyNVRTC is called instead of loading this library. The only situation it appears to be used in is for ROCm.

Perhaps the library should only be produced for ROCm builds?

### Versions

Has been like this for a long time. Affects any modern version.

cc @malfet @seemethere @dhruvbird @ljk53
torch failure to open libcuda.so.1 on macOS
Labels: triaged, module: macos

### 🐛 Describe the bug

the torch had a hardcode of cuda lib soname, it does not match the macOS version, it is libcuda.dylib

I just try to run an example on [stable-baselines3](https://github.com/DLR-RM/stable-baselines3) ,  it raise exception

<pre>
Traceback (most recent call last):
  File "/Users/xlla/test/test-stable-baseline3.py", line 11, in <module>
    model.learn(total_timesteps=10_000)
  File "/usr/local/lib/python3.9/site-packages/stable_baselines3/ppo/ppo.py", line 304, in learn
    return super(PPO, self).learn(
  File "/usr/local/lib/python3.9/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 270, in learn
    self.train()
  File "/usr/local/lib/python3.9/site-packages/stable_baselines3/ppo/ppo.py", line 264, in train
    loss.backward()
  File "/usr/local/lib/python3.9/site-packages/torch/_tensor.py", line 363, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/usr/local/lib/python3.9/site-packages/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Error in dlopen: dlopen(libcuda.so.1, 6): image not found
</pre>

I search this string and found it at 
`aten/src/ATen/cuda/detail/LazyNVRTC.cpp` and `third_party/tensorpipe/tensorpipe/common/cuda_lib.h`



### Versions

PyTorch version: 1.11.0a0+git967238d
Is debug build: False
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: macOS 10.13.6 (x86_64)
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.23.2
Libc version: N/A

Python version: 3.9.13 (main, Jun  3 2022, 08:10:53)  [Clang 10.0.1 (clang-1001.0.46.4)] (64-bit runtime)
Python platform: macOS-10.13.6-x86_64-i386-64bit
Is CUDA available: True
CUDA runtime version: 10.1.243
GPU models and configuration: GeForce GTX 1060 6GB
Nvidia driver version: 1.1.0
cuDNN version: Probably one of the following:
/usr/local/cuda/lib/libcudnn.7.dylib
/usr/local/cuda/lib/libcudnn_static.a
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy==0.910
[pip3] mypy-extensions==0.4.3
[pip3] mypy-protobuf==3.2.0
[pip3] numpy==1.21.5
[pip3] pytorch-lightning==1.6.4
[pip3] torch==1.11.0a0+git967238d
[pip3] torchmetrics==0.9.1
[pip3] torchtext==0.14.0a0+e2fa8d8
[pip3] torchvision==0.14.0a0+5f6e22d
[conda] Could not collect


cc @malfet @albanD
TorchScript bidirectional lnlstm from example doesn't work
Labels: oncall: jit

I tried to run a bidirectional LNLSTM from example benchmarks/fastrnns/custom_lstms.py with code:
```python
def test_script_stacked_bidir_lnlstm(seq_len, batch, input_size, hidden_size,
		num_layers):
	inp = torch.randn(seq_len, batch, input_size)
	states = [LSTMState(torch.randn(batch, hidden_size),
			torch.randn(batch, hidden_size))
		for _ in range(num_layers)]
	rnn = script_lnlstm(
			input_size, hidden_size, num_layers, bidirectional=True)

	# just a smoke test
	out, out_state = rnn(inp, states)

	return out, out_state
```

It caused such error:
```
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "/home/xun/projs/ML/ML_alls/custom_lstms.py", line 202, in forward
		outputs = torch.jit.annotate(List[Tensor], [])
		for i in range(len(inputs)):
			out, state = self.cell(inputs[i], state)
                ~~~~~~~~~ <--- HERE
			outputs += [out]
		return torch.stack(outputs), state
  File "/home/xun/projs/ML/ML_alls/custom_lstms.py", line 176, in forward
		hx, cx = state
		igates = self.layernorm_i(torch.mm(input, self.weight_ih.t()))
		hgates = self.layernorm_h(torch.mm(hx, self.weight_hh.t()))
                            ~~~~~~~~ <--- HERE
		gates = igates + hgates
		ingate, forgetgate, cellgate, outgate = gates.chunk(4, 1)
RuntimeError: self must be a matrix
```
[build] No documented way to install C++ binaries for pure-python development of pytorch
Labels: oncall: binaries, module: build, triaged

## Issue description

There is currently no method documented to facilitate pure python development of PyTorch. This results in unnecessary compilation and building for those who just want to work on the Python part of PyTorch.

Even after disabling almost everything as documented in the "Install from Source" portion in README, the compilation can still take a long time. This feels quite unnecessary to just work on the Python portion of pytorch.

Current alternatives include manually symlinking files you are working on into site-packages.

## Code example

Run `DEBUG=1 USE_DISTRIBUTED=0 USE_MKLDNN=0 USE_CUDA=0 BUILD_TEST=0 USE_FBGEMM=0 USE_NNPACK=0 USE_QNNPACK=0 USE_XNNPACK=0 python setup.py develop` (documented in the documentation)



cc @ezyang @seemethere @malfet
[bazel] build spams warnings
Labels: triaged, module: bazel

### 🐛 Describe the bug

Minor annoyance about bazel build:
it doesn't use the same COPTs as the CMake build for the warning ignores, which leads to a lot of spam during the build.
It would be useful to do something like "build_variables.bzl" but for the COPTs.

### Versions

master 164029f783ba52d206862925e9341e6b851179ff
Adam not optimally implemented: unnecessary torch.div
Labels: module: performance, module: optimizer, triaged, actionable

### 🚀 The feature, motivation and pitch

Adam is implemented in https://github.com/pytorch/pytorch/blob/master/torch/optim/adam.py#L259 using the following effective formula (terminology from the [paper](https://arxiv.org/abs/1412.6980) and the code with light abbreviations): 
```
theta = lr / bias1 * m / (sqrt(v) / bias2 + eps)
```
This formulation requires an additional division of the full-sized tensor `sqrt(v)` by `bias2`. Presumably it was done this way to have `eps` not be implicitly affected by `bias2`. However, we can still achieve this without the costly `sqrt(v)/bias2` op - multiply & divide `eps` by `bias2` and then factor out `1/bias2` in the denominator and you get: 
```
theta = lr * bias2 / bias1 * m / (sqrt(v) + bias2 * eps)
```
which should be incrementally faster to compute. 

Note: the current formulation permeates the Adam and Adam-esque implementations, so any update should presumably be propagated across them. 




### Alternatives

N/a

### Additional context

I've confirmed that the current formulation does indeed induce an extra `div` op in the Adam step: 

![image](https://user-images.githubusercontent.com/90846/173192526-34c07c06-e6f9-4d49-8d56-2dc92bdc5466.png)

This was done on pytorch v1.10, and the code has been reorganized since so the precise file & line reference is no longer accurate, but the relevant piece of code hasn't actually changed, it's still there on https://github.com/pytorch/pytorch/blob/master/torch/optim/adam.py#L259 as of this writing. 

cc @VitalyFedyunin @ngimel @vincentqb @jbschlosser @albanD
[bazel] ability to run gpu tests on gpu machines in RBE
Labels: triaged, module: bazel

### 🚀 The feature, motivation and pitch

As a user who builds pytorch using bazel and Remote Build Execution, I need a way to tell the remote build to use GPU machines for tests that require GPU. In order to achieve this, it's beneficial to consolidate all tests usage through the common macros definition.

### Alternatives

Build and test everything on GPU machines. Very expensive.

### Additional context

This is quality of life improvement for bazel builds.
PyTorch get positive log_prob of a multivariate normal distribution 
Labels: triaged

### 🐛 Describe the bug

PyTorch get positive log_prob of a multivariate normal distribution 

version is 1.11.0
here is the code:
```
import torch
m = torch.Tensor([0.3512,  1.3149, -0.8541,  0.5619]).to(torch.double)
cov = torch.Tensor([[0.2304, 0.0000, 0.0000, 0.0000],
                    [0.0000, 0.1919, 0.0000, 0.0000],
                    [0.0000, 0.0000, 0.0785, 0.0000],
                    [0.0000, 0.0000, 0.0000, 0.1838]]).to(torch.double)

a = torch.Tensor([0.3278,  1.3005, -0.8631,  0.5571]).to(torch.double)
print(torch.distributions.MultivariateNormal(m, cov).log_prob(a))
```
And I get result 0.0006

### Versions

version is 1.11.0
here is the code:
```
import torch
m = torch.Tensor([0.3512,  1.3149, -0.8541,  0.5619]).to(torch.double)
cov = torch.Tensor([[0.2304, 0.0000, 0.0000, 0.0000],
                    [0.0000, 0.1919, 0.0000, 0.0000],
                    [0.0000, 0.0000, 0.0785, 0.0000],
                    [0.0000, 0.0000, 0.0000, 0.1838]]).to(torch.double)

a = torch.Tensor([0.3278,  1.3005, -0.8631,  0.5571]).to(torch.double)
print(torch.distributions.MultivariateNormal(m, cov).log_prob(a))
```
And I get result 0.0006
Conda install from pytorch-nightly channel does not install the expected version on macOS
Labels: oncall: binaries, triaged, module: macos

### 🐛 Describe the bug

A similar error occurred before https://github.com/pytorch/pytorch/issues/33103
Another related issue about pytorch nightly on macOS https://github.com/pytorch/pytorch/issues/78681

* Platform: macOS 12.4, Apple M1 Max
* Conde version 4.13
* Conda config: channel_priority: disabled

1. When installing pytorch only the expected nightly build version (1.13) is installed 

```
conda install -c pytorch-nightly pytorch
```

2. When installing pytorch together with torchvision, ```conda-forge``` channel is used and the stable versions are installed, which do not have Apple M chip support

```
conda install -c pytorch-nightly pytorch torchvision
```

3. When installing pytorch first then torchvision, pytorch version is forced to fall back to 1.11

```
The following packages will be SUPERSEDED by a higher-priority channel:

  pytorch            pytorch-nightly::pytorch-1.13.0.dev20~ --> conda-forge::pytorch-1.11.0-cpu_py39h03f923b_1
```

### Versions

```
Collecting environment information...
PyTorch version: 1.13.0.dev20220610
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.4 (arm64)
GCC version: Could not collect
Clang version: 13.1.6 (clang-1316.0.21.2.5)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 17:00:33)  [Clang 13.0.1 ] (64-bit runtime)
Python platform: macOS-12.4-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.4
[pip3] torch==1.13.0.dev20220610
[conda] numpy                     1.22.4           py39h7df2422_0    conda-forge
[conda] pytorch                   1.13.0.dev20220610         py3.9_0    pytorch-nightly
```

cc @ezyang @seemethere @malfet @albanD
Batches are being duplicated from go http call
Labels: triaged

### 🐛 Describe the bug

Hi there,

I'm currently testing a  sentence bert (MNLI) model hosted on torchserve GPU.  Here is the relevant encoding call.

```python
        embedding_list = self.model.encode(queries, batch_size=min(len(queries), 512)) 
```

I'm testing two entrypoints to understand latency of this model. 

This first, is in python with a simple requests call

```python

for i in range(1000):
  requests.post('http://localhost:8989/predictions/bert_soft_label_one_to_many_query_scores_new', headers=headers, json=json_blob_10_6500_chars)
  time.sleep(.1) 
```

For the python call, with 10 encodings of ~6500 chars each (total 65k), I see a prediction latency of ~90-120ms. Great (screenshot below)! I also notice similar responses with 100 encoding per call.
<img width="1201" alt="image" src="https://user-images.githubusercontent.com/8621935/173171373-38ccfb07-e137-4b36-8090-c71c8b4db652.png">


However, I'm also testing latency from a golang http calls with the following function call. This code is encoding up to 100 objects per request with ~140  chars each (~14k total), I'm seeing much slower latency but also it looks like the requests are being duplicated? Notice the double `batches` line.  

<img width="1335" alt="image" src="https://user-images.githubusercontent.com/8621935/173171482-ad472166-bcb2-4632-8479-16c14ed227ab.png">


```golang

func (c TorchservedClient) NewGetOneToManyQueryScores(ctx context.Context, otmr *OneToManyQueryScoreRequest) (*OneToManyQueryScoreResponse, error) {
	if otmr == nil {
		return nil, fmt.Errorf("request is empty")
	}
	otmrJson, err := json.Marshal(otmr)
	if err != nil {
		return nil, err
	}
	// This endpoint corresponds to the model name in the .mar file. The base url passed in should look like `http://torchserved-0.torchserved:8080/predictions ` and we add in the model name to this url.
	req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.makeURL("bert_soft_label_one_to_many_query_scores_new"), bytes.NewBuffer(otmrJson))
	if err != nil {
		return nil, fmt.Errorf("Request construction failed: %w", err)
	}

	req.Header.Set("Content-Type", "application/json")
	resp, err := c.client.Do(req)
	if err != nil {
		return nil, fmt.Errorf("POST %s failed: %w", req.URL, err)
	}

	defer webutil.Discard(resp)
	if resp.StatusCode != http.StatusOK {
		b, _ := ioutil.ReadAll(resp.Body)
		return nil, fmt.Errorf("POST %s returned %v %v", req.URL, resp.StatusCode, string(b))
	}

	var otmResp OneToManyQueryScoreResponse
	if err := json.NewDecoder(resp.Body).Decode(&otmResp); err != nil {
		return nil, fmt.Errorf("failed to parse response for POST %s: %w", req.URL, err)
	}

	return &otmResp, nil
}

```

Any advice for what may be going on here?

### Versions

Versions of relevant libraries:
[pip3] mypy==0.942
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.2
[pip3] torch==1.10.0
[pip3] torchserve==0.5.3
[pip3] torchvision==0.11.1
[ONNX] Internal assert error during export
Labels: oncall: jit, module: onnx, onnx-triaged

### 🐛Bug
When exporting pytorch script model to onnx, I get an error: ` input_values.size() == param_count_list.size()INTERNAL ASSERT FAILED at "../torch/csrc/jit/python/script_init.cpp":493, please report a bug to PyTorch.  `
```
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals

import torch
import torch.nn as nn
import torch.nn.functional as F
import re
import os
import unicodedata
import numpy as np

device = torch.device("cpu")

class EncoderRNN(nn.Module):
    def __init__(self, hidden_size, embedding, n_layers=1, dropout=0):
        super(EncoderRNN, self).__init__()
        self.n_layers = n_layers
        self.hidden_size = hidden_size
        self.embedding = embedding
        self.gru = nn.GRU(hidden_size, hidden_size, n_layers,
                          dropout=(0 if n_layers == 1 else dropout), bidirectional=True)

    def forward(self, input_seq, input_lengths, hidden=None):
        embedded = self.embedding(input_seq)
        packed = torch.nn.utils.rnn.pack_padded_sequence(embedded, input_lengths)
        outputs, hidden = self.gru(packed, hidden)
        outputs, _ = torch.nn.utils.rnn.pad_packed_sequence(outputs)
        outputs = outputs[:, :, :self.hidden_size] + outputs[:, : ,self.hidden_size:]
        return outputs, hidden
        
hidden_size = 500
encoder_n_layers = 2
decoder_n_layers = 2
dropout = 0.1
voc_num_words = 7826
embedding = nn.Embedding(voc_num_words , hidden_size)
encoder = EncoderRNN(hidden_size, embedding, encoder_n_layers, dropout)

test_seq = torch.LongTensor(10, 1).random_(0, voc_num_words ).to(device)
test_seq_length = torch.LongTensor([test_seq.size()[0]]).to(device)
script_encoder = torch.jit.script(encoder)
torch.onnx.export(script_encoder, (test_seq, test_seq_length),"model8.onnx") 
```
### Expected behavior
Successful model export to ONNX.
### Environment

PyTorch version: 1.11.0+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.22.4
Libc version: glibc-2.26

Python version: 3.7.13 (default, Apr 24 2022, 01:04:09)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: True
 
cc @svenstaro @eklitzke @dreiss
`prepare_qat_fx` docstring doesn't run
Labels: oncall: quantization, triaged, module: fx

This code example is in the docstring for `torch.ao.quantization.quantize_fx_prepare_qat_fx`:

https://github.com/pytorch/pytorch/blob/35eda5f95956c3631e5684d519213944ab66b012/torch/ao/quantization/quantize_fx.py#L495-L509

There are two issues:
(1) The import `from torch.ao.quantization import prepare_fx` doesn't actually work. The path is wrong.
(2) This example doesn't even use the function it is describing, but instead uses `prepare_fx`, which is for PTQ, not QAT

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel @ezyang @SherlockNoMad @EikanWang @wenzhe-nrv
allowed_functions_module_string_ignorelist doesn't work very well
Labels: triaged, oncall: pt2, module: dynamo

I noticed that adding torch.utils._pytree to the ignore list wasn't enough to induce dynamo into inlining it; I had to add it to the explicitly excluded functions list.

My hypothesis for why this is happening is that sometimes PyTorch modules accidentally reexport identifiers from other modules. These then get picked up and shoved into the allowlist.

To be honest, it's not clear to me why we ignorelist rather than explicitly allowlist what we put into graphs.

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
[NVFuser] bad performance on pyhpc_isoneutral_mixing
Labels: triaged, module: nvfuser

### 🐛 Describe the bug

-9% on pyhpc_isoneutral_mixing compared to nnc.

Repro: 
```
[ENV_VAR] python run.py -m jit -d cuda -t eval pyhpc_isoneutral_mixing
```
where ENV_VAR is either:
* PYTORCH_JIT_USE_NNC_NOT_NVFUSER=1
* PYTORCH_JIT_ENABLE_NVFUSER=1

Alternatively, run https://gist.github.com/davidberard98/53b277b57ddc8d35340fecd279a9688d to run the individual fusion groups used in this model. Run `python pyhpc_isoneutral_mixing_ir.py --id x y z ... ` with a list of graph indices. The graph indices that are performing < -10% worse than nnc are: `8 12 13 15 19 20 29`

### Versions

viable/strict, torchbench=main, A100

cc @jjsjann123
[BE] Generalize recursive wrapping utility
Labels: oncall: distributed, better-engineering, module: fsdp

### 🚀 The feature, motivation and pitch

The wrap utilities in `torch.distributed.fsdp.wrap` are not actually FSDP specific as they take a configurable `wrapper_cls` argument. 

As seen in https://github.com/pytorch/pytorch/pull/78704/files, it can be leveraged for activation checkpointing as well. Might be useful to generalize this utility and offer it more widely.

### Alternatives

_No response_

### Additional context

_No response_

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
[NVFuser] bad performance on mobilenet_v2 and mobilenet_v3_large
Labels: triaged, module: nvfuser

### 🐛 Describe the bug

-17% on mobilenet_v2 and -9.68% on mobilenet_v3_large.

From torchbench, run:
```
[ENV_FLAG] python run.py -m jit -d cuda -t eval [MODEL_NAME]
```

* replace ENV_FLAG with either `PYTORCH_JIT_ENABLE_NVFUSER=1` (for nvfuser) or `PYTORCH_JIT_USE_NNC_NOT_NVFUSER=1` (for nnc)
* replace MODEL_NAME with either `mobilenet_v2` or `mobilenet_v3_large`

Interestingly, if you use the log_extract tool it shows good perf for all of the graphs. Repro:
```
$ PYTORCH_JIT_LOG_LEVEL=">>log_extract" PYTORCH_JIT_ENABLE_NVFUSER=1 python [torchbench]/run.py -m jit -d cuda -t eval > logs.txt 2>&1
$ python [pytorch]/scripts/jit/log_extract.py logs.txt --nnc-static --nnc-dynamic --nvfuser
```

^ all the individual fusion groups show improvement, but the model as a whole gets worse.

Not quite sure why, but here are some possibilities:
* Could be some correctness issue, e.g. returning the wrong memory format from some fusion group (which would impact the overall model performance but possibly not the individual fusion group performance)
* Could be some sort of overhead (but why would that only happen in log_extract and not in the torchbench results?)
* Could be bad benchmarking either in log_extract or in torchbench.

### Versions

viable/strict, torchbench on main, A100. Built with DEBUG=0.

cc @jjsjann123
[NVFuser] bad performance on pyhpc_equation_of_state
Labels: triaged, module: nvfuser

### 🐛 Describe the bug

pyhpc_equation_of_state is showing ~40% decrease in performance compared to NNC. Note that in this model, the entire model is fused (as far as I can tell...)

Model: https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/pyhpc_equation_of_state/eos_pytorch.py
Alternate repro: 
```
python [pytorch dir]/scripts/jit/log_extract.py ir.txt --nnc-dynamic --nvfuser
```

download ir.txt from here: https://gist.github.com/davidberard98/5c89b07e101a332653a16da6da0d431d

### Versions

A100, viable/strict (`bfaa187fb0`), torchbench main branch (`9bb0bd7066`)

cc @jjsjann123
[ONNX] Enable more operators to support data propagation
Labels: module: onnx, triaged, onnx-triaged

### 🚀 The feature, motivation and pitch

Continue the work https://github.com/pytorch/pytorch/pull/75307. That PR enables Shape and Gather op to use ONNX's data propagation. There are other supported operators from ONNX ([the list](https://github.com/pytorch/pytorch/pull/75307)). To cover more use cases for shape inference in PyTorch-ONNX exporter, we should enable more operators in PyTorch-ONNX exporter to support data propagation.

### Alternatives

_No response_

### Additional context

_No response_
out-of-place functional optimizers: functional optimizers may not be composite compliant
Labels: module: optimizer, triaged, needs research, module: __torch_dispatch__, tensor subclass

## The Problem

Claim: today, the functional optimizers API is not "Composite Compliant". In other words, the implementation uses in-place operations that do not interact well with Tensor Subclasses.

For example, consider the implementation of [sgd](https://github.com/pytorch/pytorch/blob/e99c8ec3c267228c9ff9fefbfb0235dd41698948/torch/optim/sgd.py#L166-L179)

```
def sgd(params, grads, momentum_buffers: "aka, the state"):
  pass
```
Let's assume that some of the momentum buffers are Tensor subclasses, but none of params and grads are Tensor subclasses. Then we would run into a situation where we are performing `Tensor.mul_(TensorSubclass)` ([see here](https://github.com/pytorch/pytorch/blob/e99c8ec3c267228c9ff9fefbfb0235dd41698948/torch/optim/sgd.py#L233))

The "Tensor-Subclass-ness" of the buffers would be lost.

## Concrete examples

- Maybe the user has (params, grad), but want to test the effect of a different set of momentum_buffers. So they'll create a BatchedTensor for the momentum_buffers, and given (params, grad, momentum_buffers), what the user really wants is to obtain a BatchedTensor that represents the updates to apply to the params. Then, the user will manually apply the updates to the params out-of-place (instead of in-place!) to avoid an error
```
params = Tensor(...)
grads = Tensor(...)
momentum_buffers = BatchedTensor(...)
updates = sgd(params, grads, momentum_buffers)

# The following operation doesn't work, because params is a Tensor but momentum_buffers is a BatchedTensor
params.add_(updates)

# So the user's workaround, and our recommended approach, is the following:
new_params = params + updates
```

## Pitch

- Introduce a version of functional optimizers that just returns the (Tensor) updates to be applied to the parameters. This gives our users more flexiblity (they can perform the in-place themselves, or decide to do an out-of-place).
- Fix the internals of functional optimizers to ensure that if any of the inputs are Tensor subclasses, the output (the updates) propagate the Tensor-Subclass-ness.




cc @vincentqb @jbschlosser @albanD @Chillee @ezyang @zou3519 @samdow
[feature request] Support dataclass derivations of nn.Module
Labels: module: nn, triaged

### 🚀 The feature, motivation and pitch

Discussed in:
- https://github.com/pytorch/pytorch/issues/72901#issuecomment-1042408766
- https://github.com/pytorch/pytorch/issues/72901#issuecomment-1046011310
- https://github.com/pytorch/pytorch/issues/72901#issuecomment-1046014180

Usecase: eliminating boiler-plate module constructor such as in https://huggingface.co/blog/annotated-diffusion#position-embeddings . This happens a lot in simple composed modules. They could benefit from auto-generated constructors.

### Alternatives

Could nn.Module-derived classes support auto-generated constructors (that would probably mean adjusting the default nn.Module constructor to allow `**kwargs`) without marking the class as dataclass (that allow setting fields by passing constructor kwargs)? Would IDE's support hints of declared fields? (akin to dataclasses?) (afaik one can now declare and type fields in the class definition)

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345
[bug] fill_, masked_fill_ : fill ops allow lossy downcasting of fill value
Labels: module: bc-breaking, triaged, topic: bc breaking, module: primTorch

### 🐛 Describe the bug

```python
>>> import torch
>>> t = torch.ones(3, dtype=torch.int32)
>>> t.fill_(3.14)
tensor([3, 3, 3], dtype=torch.int32)
>>> t.masked_fill_(t > 0, 3.14)
tensor([3, 3, 3], dtype=torch.int32)
# binary ops and other ops correctly error out
>>> t.add_(3.14)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: result type Float can't be cast to the desired output type Int
```

cc: @mruberry 


### Versions

master

cc @ezyang @gchanan @mruberry @Lezcano @peterbell10 @ngimel