gopikrsmscs/torch-issues huggingface dataset:
KeyError `shape,stack,cos` on pennylane quantum circuit
Labels: triaged, oncall: pt2

@anijain2305 would you rather I create an "interesting model" tracker or should I keep creating unique issues for each kind of model

## Repro

`python -m pip install pennylane`

```python
import torch
import pennylane as qml
import torchdynamo

dev = qml.device('default.qubit', wires=2)

@qml.qnode(dev, interface='torch')
def circuit4(phi, theta):
    qml.RX(phi[0], wires=0)
    qml.RZ(phi[1], wires=1)
    qml.CNOT(wires=[0, 1])
    qml.RX(theta, wires=0)
    return qml.expval(qml.PauliZ(0))

def cost(phi, theta):
    return torch.abs(circuit4(phi, theta) - 0.5)**2

phi = torch.tensor([0.011, 0.012], requires_grad=True)
theta = torch.tensor(0.05, requires_grad=True)

opt = torch.optim.Adam([phi, theta], lr = 0.1)

steps = 200

def closure():
    opt.zero_grad()
    loss = cost(phi, theta)
    loss.backward()
    return loss

with torchdynamo.optimize("eager"):
  for i in range(steps):
        opt.step(closure)
```

## Logs
https://gist.github.com/msaroufim/ce9ec004536e762fb5c94eb3ab2670f1


cc @ezyang @soumith @wconstab @ngimel @bdhirsh
Replace decompositions in torchinductor/decompositions with refs/decomps in pytorch proper
Labels: triaged, oncall: pt2, module: inductor

We should audit existing decompositions and replace them with refs/decomps where appropriate, e.g. `clamp` decomposition currently incorrectly type promotes and doesn't handle non-number boundaries (see also pytorch/pytorch#93784). 

cc @ezyang @soumith @msaroufim @wconstab @bdhirsh @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire
DDP + FSDP: Investigate behavior for nn.Module APIs
Labels: high priority, triage review, oncall: distributed, triaged, better-engineering, module: ddp, module: fsdp

### 🚀 The feature, motivation and pitch

Since DDP and FSDP are class wrappers, some nn.Module APIs may additional prefixing or otherwise not work in an equivalent fashion to local nn.Module instances due to the wrapping nature. This can cause some pain points in composing the modules together, checkpointing with `state_dict`, or any code that relies on module / parameter names / APIs such as `named_parameters`. This issue is to track what we can do for DDP / FSDP for each API. I've attempted to document current status below:

-- DDP --
- state_dict / load_state_dict: Adds `module` prefix, but this can be worked around and stripped out with a flag or env variable (needs a PR to enable). 
- named_parameters / named_buffers - Adds `module` prefix as well
- named_modules / modules() - includes DDP wrapped module as well, wrapped module is prefixed with "module"
- children() - returns wrapped module and its children, whereas we might just want to return the wrapped module's children if we want equivalence to local

-- FSDP --
- state_dict / load_state_dict: There are various ways to checkpoint, but `full_state_dict` and `sharded_state_dict` should checkpoint with original parameter names
- named_parameters / named_buffers - In `summon_full_params` context, equivalent to local module checkpoint. Otherwise, `flat_param` type info is returned. @awgu is working on exposing original parameters which may help alleviate this issue
- named_modules / modules() - same as DDP, includes wrapped module as well, and all nested FSDP wrappers.
- children() - returns wrapped module and its children, same as DDP.

Also, we should check what these APIs do when DDP / FSDP is not the top-level module and just a non-root in the module hierarchy. It is not guaranteed that these APIs would remain the same.

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @gchanan @zou3519 @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
checkpoint function is not jit compatible
Labels: oncall: jit

When I use the checkpoint function here: https://pytorch.org/docs/stable/checkpoint.html with jit in scripting mode, I got following error:

torch.jit.frontend.NotSupportedError: Compiled functions can't take variable number of arguments or use keyword-only arguments with defaults:
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/checkpoint.py", line 145
def checkpoint(function, *args, **kwargs):
                                 ~~~~~~~ <--- HERE
    r"""Checkpoint a model or part of the model
    raise NotSupportedError(ctx_range, _vararg_kwarg_err)

I rewrote the checkpoint function and removed *args and **kwargs, then I got another issue here:

Python builtin <built-in method apply of FunctionMeta object at 0x55a65def18e0> is currently not supported in Torchscript:
  File "/opt/ml/code/asr_kwoon/models/checkpoint.py", line 255
    
    #if use_reentrant:
    return CheckpointFunction.apply(function, preserve, output, pe, mask)
           ~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE

Seems like I need to rewrite all functions used by checkpoint to make it jit compatible. Is there any solution for this issue?
dataparallel function doesn't work
Labels: triaged, module: data parallel

### 🐛 Describe the bug

issue: 
When I use torch.nn.parallel to distribute data onto multiple gpus, I found the Dataparallel obj `filter_model` only return output of one gpu.

```
def train_model():
    bert_model = BertModel.from_pretrained(args.bert_model)
    rank_model = BertForSearch(bert_model)

    filter_model = FilterModel(bert_model,args)
    model_state_dict = torch.load("./anchors/models/filter_model_ngram4/filter_model.bin",map_location=device)
    filter_model.load_my_state_dict({k.replace('module.', ''):v for k, v in model_state_dict.items()})


    if args.gpu!="-1":
        rank_model = rank_model.to(device)
        rank_model = torch.nn.DataParallel(rank_model)

        filter_model = filter_model.to(device)
        filter_model = torch.nn.DataParallel(filter_model)

    fit(filter_model,rank_model, train_data, dev_data)
```
 
Then I checked the source code and found the problem. After I modify the code below in torch.nn.parallel.scatter_gather. this problem will be solved.
![32624cf7e3f77e4f729e185262be2344](https://user-images.githubusercontent.com/52754849/170940922-d5ae4c5c-c4d3-4acf-a2f5-3e616b63f157.jpg)

But actually I don't know why the code before doesn't work. Could somebody explain why?  Thanks!!

### Versions
torch 1.9.0+cu102
gpu: two Tesla v100


cc @ezyang
torch.Tag doesn't have accurate mypy info
Labels: module: typing, triaged

### 🐛 Describe the bug

```
  Error (MYPY) [attr-defined]
    Module has no attribute "Tag"

        151  |        r = func_overload.decompose(*args, **kwargs)
        152  |        if r is not NotImplemented:
        153  |            return r
    >>> 154  |    if torch.Tag.data_dependent_output in func_overload.tags:
        155  |        # Check if all of the Tensor inputs are constants
        156  |        all_constant = True
        157  |
```

### Versions

master

cc @ezyang @malfet @rgommers @xuzhao9 @gramster
TorchVision testing in CI + test_fx
Labels: triaged, module: fx, fx

Recently we had the TorchVision hash updates fail due to failing tests in test_fx related to TorchVision.  Even though they have been fixed, perhaps we should review how TorchVision is tested in core.  TorchVision has its own tests that ensure that pytorch core doesn't break TorchVision, and the tests in test_fx seem to make invalid assumptions about TorchVision models.  Should we instead just be running TorchVision tests in core?  Is it necessary for test_fx to trace the TorchVision models?

tagging @datumbox and fx b/c they probably have more info

cc @ezyang @SherlockNoMad
Improvements to ProcessGroupGloo monitored_barrier
Labels: high priority, triage review, oncall: distributed, triaged, module: c10d

### 🚀 The feature, motivation and pitch

`monitored_barrier` is a well adopted API that seeks to perform a barrier that also reports the faulty rank(s) which did not make it into the barrier. However, it has some usability issues at the moment:

1) Sometimes, we see an error where rank x, x != 0 reports that it recevied errors from rank 0. But rank 0 reports that it received errors from x, so there is a circular blame issue. We should identify when this could happen 
2) In debug mode, we should add comprehensive logging around which ranks successfully acked and which ones did not. For example, when rank x acks to rank 0, under torch_distributed_debug=DETAIL, we should log this event.
3) ability to Surface these errors to external error logging databases.
4) Add a string API into `monitored_barrier` that allows users to pass in details about what is going on in the application, PG gloo can attach this string to the error msg for better debugability. 

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @gchanan @zou3519 @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
cannot import name 'ProcessGroup' from 'torch.distributed' 
Labels: oncall: distributed, triaged, module: macos

### 🐛 Describe the bug

Can't import 'ProcessGroup' from 'torch.distributed' 

`from torch.distributed import ProcessGroup`

Generates:

`Traceback (most recent call last):
  File "/Users/darrenthorpe/PycharmProjects/tableDetection/scripts/bug_report.py", line 1, in <module>
    from torch.distributed import ProcessGroup
ImportError: cannot import name 'ProcessGroup' from 'torch.distributed' (/Users/darrenthorpe/miniforge3/envs/torch-bug/lib/python3.9/site-packages/torch/distributed/__init__.py)
`

This is doing my head in..... feel like there's something trivial and silly going on.

Note I'm on MacOS with mps 12.5 Monterey

### Versions

(torch-bug) darrenthorpe@Darrens-MacBook-Air torch-bug % python collect_env.py

Collecting environment information...
PyTorch version: 1.12.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.5 (arm64)
GCC version: Could not collect
Clang version: 13.1.6 (clang-1316.0.21.2.5)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 17:00:33)  [Clang 13.0.1 ] (64-bit runtime)
Python platform: macOS-12.5-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.1
[pip3] torch==1.12.1
[pip3] torchaudio==0.12.1
[pip3] torchvision==0.12.0a0
[conda] numpy                     1.23.1           py39h7df2422_0    conda-forge
[conda] pytorch                   1.12.1          cpu_py39h0768760_0    conda-forge
[conda] torchaudio                0.12.1                 py39_cpu    pytorch
[conda] torchvision               0.12.0          cpu_py39h384c48c_1    conda-forge


cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501 @malfet @albanD
Emulating FP64 and increased precisions on Apple silicon
Labels: feature, triaged, needs research, module: mps

Spinoff of https://github.com/pytorch/pytorch/issues/77764#issuecomment-1211503363 as requested by @albanD:

---

This might not be 100% centered around PyTorch, but I would think it's still a worthy discussion.

I was contemplating a future project to emulate FP64 precision on Apple GPUs, possibly using FP64 dynamic range with FP32 precision just like Nvidia's TF32/TF19 does (to get reasonable performance). This would be a 35-bit IEEE format. I found [SoftFloat](https://github.com/ucb-bar/berkeley-softfloat-3) which should provide a good basis. I could utilize high-throughput 16-bit integers to process exponent bits, leaving FP32 ALUs for mantissa (can both integer and floating-point ALUs be utilized simultaneously?). Also, a fully IEEE compliant (but slow) genuine FP64 format would be developed. I did some back of the envelope calculations based on SoftFloat's fully IEEE-compliant FP64 emulation, and they were depressing. ~24 GFLOPS on an M1 Max GPU.

Through personal experience with (failed attempts at GPU-accelerating via Metal) physics simulations and linear algebra solvers, it's the dynamic range that matters most, not the precision. Thus the 35-bit compromise format could be very useful.

I also discussed FP64 emulation at https://github.com/DTolm/VkFFT/issues/61 for more context.

cc @kulinseth @albanD
PyYAML not listed as a dependency
Labels: oncall: releng, triaged

### 🐛 Describe the bug

`torchgen` is now packaged with the rest of `torch`. Seeing as `torchgen` requires the `yaml` module, it seems like a bug that its not listed as a install requirement
https://github.com/pytorch/pytorch/blob/beceb8b92f030b19a0ea48ab9ae0def8f554e7f3/setup.py#L440-L443

In fact, there is a check in the `setup.py` that will raise an error if `yaml` is not installed
https://github.com/pytorch/pytorch/blob/beceb8b92f030b19a0ea48ab9ae0def8f554e7f3/setup.py#L399

So, why is it not listed as a install requirement in the first place? Is there anything preventing `yaml` from being added as a dependency?

CC: @silvasean

### Versions

N/A
Using Pytorch and Mapbox in the same project
Labels: oncall: mobile

Hello,

I'm trying to use Mapbox and Pytorch together in the same Android project but I receive this compilation error:
2 files found with path 'lib/arm64-v8a/libc++_shared.so' from inputs:
 - /Users/eyalamir/.gradle/caches/transforms-3/39574a221961eb18fa9deadbee69fea4/transformed/common-22.1.0/jni/arm64-v8a/libc++_shared.so
 - /Users/eyalamir/.gradle/caches/transforms-3/c73fb4c1d8f48b7cbb206e310613488d/transformed/pytorch_android_lite-1.10.0/jni/arm64-v8a/libc++_shared.so
If you are using jniLibs and CMake IMPORTED targets, see
https://developer.android.com/r/tools/jniLibs-vs-imported-targets

If I only use one of the libraries, the project compiles correctly.
I also reproduced this issue with a new Android Project.

I'm using these versions:
Mapbox - com.mapbox.maps:android:10.7.0
Pytorch - org.pytorch:pytorch_android_lite:1.10.0

The issue is also reproduced with the full PyTorch library instead of the lite version.

Can you please instruct me on how to solve this issue?
During DDP training timm densenet121, mobilenetv2(v3) models do not save state_dict correctly.
Labels: oncall: distributed, triaged, module: ddp

### 🐛 Describe the bug

During DDP training timm densenet121, mobilenetv2(v3) models do not save state_dict correctly.
 After load the stat dicts, the model do not act well.  However, when using torch.save save the model, it works well.


### Versions

both 1.11, 1.12 and 1.12.1

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501 @ezyang
torch.nn.Upsample's error message is inconsistent with the documentation
Labels: module: docs, module: nn, triaged, actionable

### 🐛 Describe the bug

Problem 1:
Parameter 'scale_factor' shoule be float or tuple[float] on documentation. However, when I set list to 'scale_factor', the error message shows that 'scale_factor' must be a string or a number which cannot match the documentation.
```
import torch
results={}
arg_1 = [1]
arg_2 = "nearest"
arg_class = torch.nn.Upsample(scale_factor=arg_1,mode=arg_2,)
arg_3 = torch.rand([1, 1, 2, 2], dtype=torch.float32)
results['res'] = arg_class(*arg_3)
#TypeError: float() argument must be a string or a number, not 'list'
```

Problem 2:
It's written on documentation that if 'scale_factor' is tuple, it has to match input size. That is to say, if input is a 2D tensor, then 'scale_factor' should be a float number or a tuple of length 2. I find that if input is a 2D tensor, 'scale_factor' can be a tuple of length 1. It's inconsistent with the documentation.
```
import torch
results={}
arg_1 = (33)
arg_2 = "nearest"
arg_class = torch.nn.Upsample(scale_factor=arg_1,mode=arg_2,)
arg_3 = torch.rand([1, 1, 2, 2], dtype=torch.float32)
results['res'] = arg_class(*arg_3)
```

### Versions

pytorch: 1.8.1
python: 3.8.3
os: win11

cc @svekars @holly1238 @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
torch.nn.TripletMarginLoss margin can be less than 0
Labels: module: nn, triaged

### 🐛 Describe the bug

Parameter _margin_ of torch.nn.TripletMarginLoss is a value greater than 0 on documentation. However, when its value is less than 0, this API also works.
```
import torch
results={}
arg_1 = -7.0
arg_2 = 2
arg_class = torch.nn.TripletMarginLoss(margin=arg_1,p=arg_2)
arg_3_0 = torch.rand([100, 128], dtype=torch.float32)
arg_3_1 = torch.rand([100, 128], dtype=torch.float32)
arg_3_2 = torch.rand([100, 128], dtype=torch.float32)
arg_3 = [arg_3_0,arg_3_1,arg_3_2]
results['res'] = arg_class(*arg_3)
```

### Versions

pytorch: 1.8.1
python: 3.8.3
os: win11

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
The type of parameter 'p' in torch.nn.TripletMarginLoss wrong
Labels: module: nn, triaged

### 📚 The doc issue

Parameter 'p' of torch.nn.TripletMarginLoss is int value on the documentation parameter part. However, in source code, p's type is float. By running code, it can be proved that parameter 'p' supports float value.

```
import torch
results={}
arg_class = torch.nn.TripletMarginLoss(margin=1.0,p=24.5)
arg_3_0 = torch.rand([100, 128], dtype=torch.float32)
arg_3_1 = torch.rand([100, 128], dtype=torch.float32)
arg_3_2 = torch.rand([100, 128], dtype=torch.float32)
arg_3 = [arg_3_0,arg_3_1,arg_3_2,]
results['res'] = arg_class(*arg_3)
```
Above code works.

### Suggest a potential alternative/fix

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
torch.nn.ReplicationPad{1|2}d supports more input dimension than are written on documentation
Labels: module: docs, module: nn, triaged

### 📚 The doc issue

torch.nn.ReplicationPad1d supports (C,W) or (N, C, W) input, and needs a int or 2-tuple padding on the documentation. That is to say, torch.nn.ReplicationPad1d only supports 2D or 3D tensor.

torch.nn.ReplicationPad2d supports (C,H,W) or (N, C, H,W) input, and needs a int or 4-tuple padding on the documentation. That is to say, torch.nn.ReplicationPad2d only supports 3D or 4D tensor.

After running code, I find that torch.nn.ReplicationPad1d also supports 4D tensor with 4-tuple padding, and torch.nn.ReplicationPad2d also supports 5D tensor with 6-tuple padding.

```
import torch
results={}
arg_1 = [3,0,2,1]
arg_class = torch.nn.ReplicationPad1d(arg_1,)
arg_2 = torch.rand([25, 2, 46, 1], dtype=torch.float32)
results['res'] = arg_class(arg_2)
```

```
import torch
results={}
arg_1 = [2,2,2,2,2,2]
arg_class = torch.nn.ReplicationPad2d(arg_1,)
arg_2 = torch.rand([4, 1, 1, 3, 3], dtype=torch.float32)
results['res'] = arg_class(arg_2)
```
Above code runs well.

### Suggest a potential alternative/fix

_No response_

cc @svekars @holly1238 @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
torch.nn.PixelShuffle error message wrong
Labels: module: nn, triaged

### 🐛 Describe the bug

There is something wrong in torch.nn.PixelShuffle's error message. It seems that the result is overflow.

```
import torch
results={}
arg_class = torch.nn.PixelShuffle(10000000000)
arg_2 = torch.rand([16, 256, 72, 72], dtype=torch.float32)
results['res'] = arg_class(arg_2)
```
Above code outputs: RuntimeError: pixel_shuffle expects its input's 'channel' dimension to be divisible by the square of upscale_factor, but input.size(-3)=256 is not divisible by 7766279631452241920.

But actually the error message should be RuntimeError: pixel_shuffle expects its input's 'channel' dimension to be divisible by the square of upscale_factor, but input.size(-3)=256 is not divisible by 100000000000000000000.

### Versions

pytorch: 1.8.1
python: 3.8.3
os: win11

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
torch.nn.MaxUnpool2d get negative size tensor
Labels: module: nn, triaged

### 🐛 Describe the bug

When the kernel_size is negative, torch.nn.MaxUnpool2d will output negative size tensor.
In torch.nn.MaxUnpool3d, if the kernel_size is <=0, the program will die, ctrl+c cannot quit and have to force to kill the program.
```
import torch
results={}
arg_1 = -100
arg_2 = False
arg_class = torch.nn.MaxUnpool2d(arg_1,stride=arg_2,)
arg_3_0 = torch.rand([1, 1, 2, 2], dtype=torch.float32)
arg_3_1 = torch.randint(-1,64,[1, 1, 2, 2], dtype=torch.int64)
arg_3 = [arg_3_0,arg_3_1,]
results['res'] = arg_class(*arg_3)
print(results['res'].shape)
#torch.Size([1, 1, -100, -100])
```

```
import torch
pool = torch.nn.MaxPool3d(3, stride=2, return_indices=True)
unpool = torch.nn.MaxUnpool3d(-3, stride=2)
output, indices = pool(torch.randn(20, 16, 51, 33, 15))
unpooled_output = unpool(output, indices)
print(unpooled_output.size())
#program die
```

### Versions

pytorch: 1.8.1
python: 3.8.3
os: win11

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
torch.nn.InstanceNorm{1|2|3}d doesn't verify the value type of parameter num_features
Labels: module: nn, triaged

### 🐛 Describe the bug

Parameter 'num_features' is the number of features or channels C of the input. However, I found that num_features can be set to negative integral / string / list and other type value. torch.nn.InstanceNorm1d doesn't verify whether the value of num_features and input channels are equal. 
```
import torch
results={}
arg_1 = 'max'
arg_2 = False
arg_class = torch.nn.InstanceNorm1d(arg_1,affine=arg_2,)
arg_3 = torch.rand([20, 100, 40], dtype=torch.float32)
results['res'] = arg_class(arg_3)
```
Above code works.

### Versions

pytorch: 1.8.1
python: 3.8.3
os: win11

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
[VDD] unique_by_key from _embedding_bag_dense_backward  isn't blocklisted by CUDA graphs
Labels: triaged, module: cuda graphs, oncall: pt2, module: dynamo

repro:

```
jf get --update D38599092
CUDA_LAUNCH_BLOCKING=1 buck2 run @mode/opt -c python.package_style=inplace //hpc/torchrec/models/feed/benchmark:vdd_benchmark -- --iters 31 --compile True --cudagraphs True --pad_seq_embs=true --dynamo True
```

Fails with

```
RuntimeError: unique_by_key: failed to synchronize: cudaErrorStreamCaptureUnsupported: operation not permitted when stream is capturing
```

Relevant backtrace:

```
Traceback (most recent call last):
  File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/b01f384851ab2430/hpc/torchrec/models/feed/benchmark/__vdd_benchmark__/vdd_benchmark#link-tree/torchinductor/compile_fx.py", line 195, in cudagraphify
    static_outputs = model(*static_inputs)
  File "/tmp/torchinductor_ezyang/gj/cgjht4biqqcp6eqw6noomuiivkokp3rk5czqlfqt3ttsy27p2xlj.py", line 3408, in call
    buf304 = torch.ops.aten._embedding_bag_dense_backward.default(buf302, buf303, getitem_80, getitem_81, getitem_82, 100, False, 0, None)
```

cc @mcarilli @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
Enable freezing parts of the model in Fully Sharded Data Parallel
Labels: oncall: distributed, triaged, module: fsdp

### 🚀 The feature, motivation and pitch

Due to how `FlattenParamsWrapper` is used in the current FSDP implementation, there doesn't seem to be a straightforward way to shard a parameter which doesn't need to be optimized. 

The particular use case I'm aiming at is where one shards the whole model's parameters to save memory, while only computing gradients for a small subset of parameters. 

### Alternatives

_No response_

### Additional context

Related to https://github.com/pytorch/pytorch/issues/76501

@awgu

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501 @ezyang
Check support of FSDP + set_materialize_grads(False)
Labels: oncall: distributed, triaged, module: fsdp

### 🐛 Describe the bug

We should investigate whether FSDP will work well or any assumptions break with custom functions and setting `ctx.set_materialize_grad(False)` for undefined / None gradients: https://pytorch.org/docs/stable/generated/torch.autograd.function.FunctionCtx.set_materialize_grads.html#torch.autograd.function.FunctionCtx.set_materialize_grads.

In particular, some assumptions around here might break: https://github.com/pytorch/pytorch/blob/f534b2c627da65bbee7ccc8f7e054da0ba48eb79/torch/distributed/fsdp/fully_sharded_data_parallel.py#L2884

### Versions

main

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501 @ezyang
module 'torch.distributed' has no attribute 'pipeline' - macOS, PyTorch 1.12.1
Labels: oncall: distributed, triaged, pipeline parallelism, release notes: distributed (pipeline)

### 🐛 Describe the bug

On macOS 12.5, I installed PyTorch 1.12.1 using Miniconda.

The following referring to class `Pipe` raised an exception `AttributeError: module 'torch.distributed' has no attribute 'pipeline`.

```
import torch
model = torch.distributed.pipeline.sync.Pipe(model, chunks=8)
```

However, the following works

```
from torch.distributed.pipeline.sync import Pipe
model = Pipe(model, chunks=8)
```

And the following works too.

```
from torch.distributed.pipeline.sync import Pipe
model = torch.distributed.pipeline.sync.Pipe(model, chunks=8)
```


### Versions

macOS 12.5

Python 3.10.5 | packaged by conda-forge

PyTorch 1.21.1 installed using Miniconda

```
>>> torch.__version__
'1.12.1'
```

It seems that PyTorch Distributed is enabled.

```
>>> import torch
>>> torch.distributed.is_available()
True
```



cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
torch.nn.GRU runs long time, when num_layers is large
Labels: module: nn, triaged, module: edge cases

### 🐛 Describe the bug

When num_layers is 100000, torch.nn.GRU runs more than 5 minutes. Program hangs and doesn't return result.
```
import torch
results={}
arg_1 = 10
arg_2 = 20
arg_3 = 100000
arg_class = torch.nn.GRU(arg_1,arg_2,arg_3,)
```

### Versions

pytorch: 1.8.1
python: 3.8.3
os: win11

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
torch.nn.functional.softplus / torch.nn.Softplus parameter beta can be set to zero
Labels: module: nn, triaged

### 🐛 Describe the bug

According to the documentation, Softplus(x)=(1/β)∗log(1+exp(β∗x)). Beta is  the denominator. However, I found that beta can be assigned to zero. Is this reasonable?
```
import torch
results={}
arg_1 = torch.rand([], dtype=torch.float32)
results['res'] = torch.nn.functional.softplus(arg_1,beta=0,threshold=20)
```
```
import torch
results={}
arg_class = torch.nn.Softplus(beta=0)
arg_1 = torch.rand([2], dtype=torch.float32)
results['res'] = arg_class(arg_1)
```
Above code runs.

### Versions

pytorch: 1.8.1
python: 3.8.3
os: win11

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
deepcopy of LazyLinear fails
Labels: module: nn, triaged, actionable

### 🐛 Describe the bug

When running 
```
l_linear = LazyLinear(10)
deepcopy(l_linear)
```
the following error is triggered
```
Exception has occurred: TypeError
empty() received an invalid combination of arguments - got (int, dtype=NoneType, device=bool), but expected one of:
 * (tuple of ints size, *, tuple of names names, torch.memory_format memory_format, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
 * (tuple of SymInts size, *, torch.memory_format memory_format, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
 * (tuple of ints size, *, torch.memory_format memory_format, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
  File "/Users/ndufour/Documents/pytorch/rl/debug_deepcopy.py", line 9, in <module>
    deepcopy(l_linear)
```
This prevents making deepcopy of models before doing forward on one of them

### Versions

Collecting environment information...
PyTorch version: 1.13.0.dev20220725
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.5 (arm64)
GCC version: Could not collect
Clang version: 13.1.6 (clang-1316.0.21.2.5)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.12 (main, Jun  1 2022, 06:34:44)  [Clang 12.0.0 ] (64-bit runtime)
Python platform: macOS-12.5-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] functorch==0.3.0a0+516a8cd
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.1
[pip3] torch==1.13.0.dev20220725
[pip3] torchrl==0.0.1a0+008bebd
[pip3] torchvision==0.14.0.dev20220725
[conda] functorch                 0.3.0a0+516a8cd          pypi_0    pypi
[conda] numpy                     1.23.1           py39h42add53_0  
[conda] numpy-base                1.23.1           py39hadd41eb_0  
[conda] pytorch                   1.13.0.dev20220725         py3.9_0    pytorch-nightly
[conda] torchrl                   0.0.1a0+008bebd           dev_0    <develop>
[conda] torchvision               0.14.0.dev20220725        py39_cpu    pytorch-nightly

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
torch.nn.functional.log_softmax  parameter '_stacklevel' undocumented
Labels: module: nn, triaged, actionable

### 📚 The doc issue

The documentation of the very popular torch.nn.functional.log_softmax doesn't explain what _stacklevel is, why it's set to 3 and what to do with it.

### Suggest a potential alternative/fix

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
Optimize for mobile metal model
Labels: oncall: mobile

### 🐛 Describe the bug

Traceback (most recent call last):
  File "metal_model.py", line 15, in <module>
    script_model_metal = optimize_for_mobile(script_model, backend='metal')
  File "/anaconda3/envs/metal/lib/python3.8/site-packages/torch/utils/mobile_optimizer.py", line 69, in optimize_for_mobile
    optimized_cpp_module = torch._C._jit_pass_metal_optimize_for_mobile(script_module._c, preserved_methods_str)
RuntimeError: 0INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1623448216815/work/torch/csrc/jit/ir/alias_analysis.cpp":532, please report a bug to PyTorch. We don't have an op for metal_prepack::conv2d_prepack but it isn't a special case.  Argument types: Tensor, Tensor?, int[], int[], int[], int, NoneType, NoneType, 


### Versions

Collecting environment information...
PyTorch version: 1.10.1
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.24.0
Libc version: glibc-2.31

Python version: 3.8.13 (default, Mar 28 2022, 11:38:47)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-43-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1060 6GB
Nvidia driver version: 515.65.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] torch==1.10.1
[pip3] torchaudio==0.10.1
[pip3] torchvision==0.11.2
[conda] blas                      1.0                         mkl    conda-forge
[conda] cudatoolkit               11.3.1               h2bc3f7f_2    anaconda
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640    anaconda
[conda] mkl-service               2.4.0            py38h7f8727e_0    anaconda
[conda] mkl_fft                   1.3.1            py38hd3c417c_0    anaconda
[conda] mkl_random                1.2.2            py38h51133e4_0    anaconda
[conda] numpy                     1.22.3           py38he7a7128_0    anaconda
[conda] numpy-base                1.22.3           py38hf524024_0    anaconda
[conda] pytorch                   1.10.1          py3.8_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.10.1               py38_cu113    pytorch
[conda] torchvision               0.11.2               py38_cu113    pytorch
Expand Learning rate scheduling to any optimization hyperparameter
Labels: feature, module: optimizer, triaged, needs design, module: LrScheduler

### 🚀 The feature, motivation and pitch

I'm developing a new optimizer using the PyTorch optimizer framework. It does not depend on a learning rate, but on a KL-divergence. I would like to schedule other optimization hyperparameters (given in the optimizer config) with the PyTorch scheduler. Right now, the key `lr` is hardcoded inside the schedulers. 

I would like to have an option where you can specify the key/name of the scheduled variable when creating a scheduler. This would also unify other schedulers (for example weight decay). 

### Alternatives

I considered renaming my optimization hyperparameter `kl_div` to `lr` and use the learning rate scheduler. However, since it is not a learning rate this is not correct and may cause confusion on the user.

### Additional context

_No response_

cc @vincentqb @jbschlosser @albanD
torch.nn.Hardtanh allows min_val > max_val
Labels: module: nn, triaged

### 🐛 Describe the bug

torch.nn.Hardtanh allows min_val greater than max_val. It doesn't throw exception, and the document has no 'add note' for case min_val > max_val.
```
import torch
results={}
arg_1 = torch.rand([80, 192, 9, 9], dtype=torch.float32)
arg_2 = 6.0
arg_3 = 0.0
arg_4 = True
results['res'] = torch.nn.functional.hardtanh(arg_1,arg_2,arg_3,arg_4,)
```

Above code works.

### Versions

pytorch: 1.8.1
python: 3.8.3
os: win11

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
When padding is big int, torch.nn.functional.fold runs too long and can't return result
Labels: module: nn, triaged

### 🐛 Describe the bug

When I run the code, there is no error information reports. After 5 mins running, there is no response and I can't stop the cmd, I have to kill the cmd.

```
import torch
results={}
arg_1_tensor = torch.rand([1, 12, 12], dtype=torch.float32)
arg_1 = arg_1_tensor.clone()
arg_2 = [4,5,]
arg_3 = [2,2,]
arg_4 = 1
arg_5 = 36028797018963968
arg_6 = 1
results['res'] = torch.nn.functional.fold(arg_1,arg_2,arg_3,arg_4,arg_5,arg_6,)
```

### Versions

pytorch: 1.8.1
python: 3.8.3
os: win11

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
Make FSDP easier to debug when erroring in backward pass
Labels: high priority, triage review, oncall: distributed, triaged, module: fsdp

### 🚀 The feature, motivation and pitch

Recently a lot of FSDP enablement efforts have been hard to debug when there's an error in backward pass, because we just get the error "autograd returned null without setting an error" without too much additional detail.

To fix this, we should comb through the code that runs in bwd and ensure we at least use `p_assert` everywhere, and maybe consider more solutions such as wrapping all code that can throw with try/except, adding additional details in the except and throwing a better error.

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @gchanan @zou3519 @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
bf16 strided tensor wrong calculation
Labels: high priority, triaged, module: bfloat16, module: correctness (silent), module: reductions, module: intel

### 🐛 Describe the bug

Issue when transpose is taken on bfloat16 tensor and passed to sum. the output is not correct. Example has been described below:
If you run same code in float32 then it works fine.
```
import torch
x = torch.ones([10, 13, 3, 3], dtype=torch.bfloat16)
x_trans = x.transpose(2, 3)
x_sum = torch.sum(x_trans, (0, 1, 2))
print(x_sum)
```

```
output: tensor([432., 432., 432.], dtype=torch.bfloat16)
but except output is tensor([390., 390., 390.])
```

### Versions

Collecting environment information...
PyTorch version: 1.12.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Jun 22 2022, 20:18:18)  [GCC 9.4.0] (64-bit runtime)
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.3
[pip3] pytorch-lightning==1.6.5
[pip3] torch==1.12.0
[pip3] torch-tb-profiler==0.4.0
[pip3] torchmetrics==0.9.3
[pip3] torchvision==0.13.0
[conda] mkl                       2022.0.2                 pypi_0    pypi
[conda] mkl-include               2022.0.2                 pypi_0    pypi
[conda] numpy                     1.22.3                   pypi_0    pypi
[conda] pytorch-lightning         1.6.5                    pypi_0    pypi
[conda] torch                     1.12.0          pypi_0    pypi
[conda] torch-tb-profiler         0.4.0                    pypi_0    pypi
[conda] torchmetrics              0.9.3                    pypi_0    pypi
[conda] torchvision               0.13.0          pypi_0    pypi

cc @ezyang @gchanan @zou3519 @VitalyFedyunin @frank-wei
[MPS] Bug on training CNN+LSTM
Labels: triaged, module: mps

### 🐛 Describe the bug

Following training on M1MAX GPU

when I training a CNN+LSTM model on Pytorch v1.12.1, it goes with this error 
loc("total derivative last state"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/20d6c351-ee94-11ec-bcaf-7247572f23b4/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":219:0)): error: input types 'tensor<1x82x64xf32>' and 'tensor<1x32x64xf32>' are not broadcast compatible
LLVM ERROR: Failed to infer result type(s).

this does not happened on previous Pytorch V11.2.0, I guess something wrong with new LSTM result matrix transformation?

### Versions

Collecting environment information...
PyTorch version: 1.12.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.5 (arm64)
GCC version: Could not collect
Clang version: 13.1.6 (clang-1316.0.21.2.5)
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.5 | packaged by conda-forge | (main, Jun 14 2022, 07:07:06) [Clang 13.0.1 ] (64-bit runtime)
Python platform: macOS-12.5-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.1
[pip3] torch==1.12.1
[pip3] torchaudio==0.12.1
[pip3] torchinfo==1.7.0
[pip3] torchvision==0.13.1
[conda] numpy                     1.23.1          py310h220015d_0  
[conda] numpy-base                1.23.1          py310h742c864_0  
[conda] pytorch                   1.12.1                 py3.10_0    pytorch
[conda] torchaudio                0.12.1                py310_cpu    pytorch
[conda] torchinfo                 1.7.0              pyhd8ed1ab_0    conda-forge
[conda] torchvision               0.13.1                py310_cpu    pytorch

cc @kulinseth @albanD
torch.nn.functional.avg_pool{1|2|3}d error message does not match what is described in the documentation
Labels: module: docs, module: nn, triaged

### 📚 The doc issue

Parameter 'kernel_size' and 'stride' of torch.nn.functional.avg_pool{1|2|3}d can be a single number or a tuple. However, I found that error message only mentioned tuple of ints which means parameter 'kernel_size' and 'stride' can be only int number or tuple of ints.

```
import torch
results={}
arg_1 = torch.rand([1, 1, 7], dtype=torch.float32)
arg_2 = 8.0
arg_3 = 2
arg_4 = 0
arg_5 = True
arg_6 = True
results['res'] = torch.nn.functional.avg_pool1d(arg_1,arg_2,arg_3,arg_4,arg_5,arg_6,)
#TypeError: avg_pool1d(): argument 'kernel_size' (position 2) must be tuple of ints, not float
```

```
import torch
results={}
arg_1 = torch.rand([16, 528, 16, 16], dtype=torch.float32)
arg_2 = 32.0
arg_3 = 13.0
arg_4 = 0
arg_5 = False
arg_6 = True
arg_7 = None
results['res'] = torch.nn.functional.avg_pool2d(arg_1,arg_2,arg_3,arg_4,arg_5,arg_6,arg_7,)
#TypeError: avg_pool2d(): argument 'stride' (position 3) must be tuple of ints, not float
```

```
import torch
results={}
arg_1 = torch.rand([20, 16, 50, 44, 31], dtype=torch.float32)
arg_2_0 = 3.0
arg_2_1 = 2
arg_2_2 = 2
arg_2 = [3.0,2,2]
arg_3_0 = 2
arg_3_1 = 1
arg_3_2 = 2
arg_3 = [2,1,2]
arg_4 = 0
arg_5 = False
arg_6 = True
arg_7 = None
results['res'] = torch.nn.functional.avg_pool3d(arg_1,arg_2,arg_3,arg_4,arg_5,arg_6,arg_7,)
#TypeError: avg_pool3d(): argument 'kernel_size' must be tuple of ints, but found element of type float at pos 1
```

### Suggest a potential alternative/fix

It would be great if the doc could be written as follows:

kernel_size – size of the pooling region. Can be a int number or a tuple (kT, kH, kW).
stride – stride of the pooling operation. Can be a int number or a tuple (sT, sH, sW).

Or modify the error message so that it matches the document description.

cc @svekars @holly1238 @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
One dlpack to rule them all
Labels: triaged, better-engineering, module: dlpack

### 🐛 Describe the bug

One here https://github.com/pytorch/pytorch/blob/master/caffe2/python/dlpack.h
And another there https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/dlpack.h
Should we make one reference to another one?

### Versions

1.12/CI
[FSDP] `test_summon_single_param()` is misleading
Labels: triaged, module: fsdp

Code pointer:
https://github.com/pytorch/pytorch/blob/6c60a656b02fbd09661b282bec53940b184db3ca/test/distributed/fsdp/test_fsdp_summon_full_params.py#L269-L288

---

WLOG suppose we have a world size of 2.
- The original parameter (i.e. the linear's weight) is like `[[X]]` where `X` is some initial value and the shape is `[1, 1]`.
- The unpadded unsharded flattened parameter is like `[X]` where the shape is `[1]`.
- The padded unsharded flattened parameter is like `[X, 0]` where the shape is `[2]`. (For larger world sizes, we pad more like `[X, 0, 0, 0]` for world size of 4.)
- After we run on both ranks
    ```
    with torch.no_grad():
        p[0] = self.rank + 2
    ```
    rank 0's local shard is `[2]`, and rank 1's local shard is `[3]`.
- When we run on both ranks
    ```
    with model.summon_full_params(model, writeback=True):
        with torch.no_grad():
           p.copy_(torch.zeros_like(p))
    ```
    `p` is actually the unpadded unsharded flattened parameter on both rank 0 and rank 1. ([code0](https://github.com/pytorch/pytorch/blob/6c60a656b02fbd09661b282bec53940b184db3ca/torch/distributed/fsdp/fully_sharded_data_parallel.py#L2548), [code1](https://github.com/pytorch/pytorch/blob/6c60a656b02fbd09661b282bec53940b184db3ca/torch/distributed/fsdp/fully_sharded_data_parallel.py#L3348)) In other words, if you print `p` in the `summon_full_params()` context, you see `[2]` for both rank 0 and rank 1. This means that the `copy_()` writes to the `0`th element for both ranks. There is not an attempt to zero the `1`st element of the padded unsharded flattened parameter.

We see that this test actually tests whether changes to the padding made before `summon_full_params()` persist after `summon_full_params()`. We could change the `p.copy_(torch.zeros_like(p))` to only run on rank 0, and the test would work just the same.

Personally, I am not sure if FSDP should make any guarantees on persisting writes to the padding. It does not seem like a real use case. 

cc @zhaojuanmao @mrshenli @rohan-varma @ezyang
FSDP crash if no parameters are used in fwd pass
Labels: high priority, triage review, oncall: distributed, triaged, module: fsdp

### 🐛 Describe the bug

The following test raises an issue where an assert is triggered if FSDP has params that require grad, but they are not used in an iteration:

```
@skip_if_lt_x_gpu(2)
    def test_fsdp_namedtuple(self):

        class MyModule(nn.Module):
            def __init__(self):
                super().__init__()
                self.lin = nn.Linear(1, 1)

            def forward(self, x):
                return x

        m = MyModule().cuda()
        m = FSDP(m)
        t = torch.ones(1, device="cuda", requires_grad=True)

        MyOutputType = namedtuple(
            "MyOutputType",
            ["a", "b", "c", "d"],
            defaults=(t, t, t, t)
        )

        inp = MyOutputType()
        out = m(inp)
        print(out)
        res = torch.cat([e for e in out]).sum()
        res.backward()
```

This triggers the assert here: https://github.com/pytorch/pytorch/blob/cd5efc6f082c81fd40712127638931b9e2e5ee69/torch/distributed/fsdp/fully_sharded_data_parallel.py#L3084, presumably because the FSDP managed param requires grad, but does not get its gradient computed, so it never entered post backward.

In practice, this came up when wrapping FLAVA encoders separately, but encoders take turns being used across iterations.

### Versions

main

cc @ezyang @gchanan @zou3519 @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
Direct use of torchdynamo.optimizations.analysis fails if you pass in None as an input
Labels: triaged, enhancement, oncall: pt2, module: dynamo

Example

```
diff --git a/fbcode/pytorch/torchdynamo/torchdynamo/optimizations/analysis.py b/fbcode/pytorch/torchdynamo/torchdynamo/optimizations/analysis.py
--- a/fbcode/pytorch/torchdynamo/torchdynamo/optimizations/analysis.py
+++ b/fbcode/pytorch/torchdynamo/torchdynamo/optimizations/analysis.py
@@ -38,8 +38,8 @@
 
     def placeholder(self, target, args, kwargs):
         value = super().placeholder(target, args, kwargs)
-        assert isinstance(value, torch.Tensor)
-        self.input_alias_groups.add(self.tensor_alias_group(value))
+        if isinstance(value, torch.Tensor):
+            self.input_alias_groups.add(self.tensor_alias_group(value))
         return value
 
     def run_node(self, n: torch.fx.Node):
diff --git a/fbcode/torchrec/sparse/jagged_tensor.py b/fbcode/torchrec/sparse/jagged_tensor.py
```

Dynamo will ensure that a None input can never occur but this is not the case for direct use of `compile_fx` api in inductor, for example

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh @mlazos @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
[CI] Create periodic fuzzy testing for PyTorch build flags
Labels: module: ci, triaged

We should have CI to test flag compatibility for PyTorch. 

Proposal: add a periodic job that randomly pick flags to enable + tries building.

The key blocker here is that we would need to make sure we have an owner to forward fix when the flags are incompatible.

cc @seemethere @malfet @pytorch/pytorch-dev-infra