gopikrsmscs/torch-issues huggingface dataset:
Memory access fault with AMD Rocm
Labels: needs reproduction, module: rocm, triaged

### 🐛 Describe the bug

When using Pytorch with Rocm, trying to train or infer with an upscaling model, I get this error:
```
Memory access fault by GPU node-1 (Agent handle: 0x55eb9b596570) on address 0x7f66960b2000. Reason: Page not present or supervisor privilege.
Abandon (core dumped)
```


### Versions

PyTorch: 2.2.0.dev20230920+rocm5.6

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
cm3leon_generate failing compilation
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

```python
import torch
import importlib 
import sys

sys.path.append("benchmark")
c = "torchbenchmark.models.cm3leon_generate"
module = importlib.import_module(c)

device = "cuda"
benchmark_cls = getattr(module, "Model", None)
benchmark = benchmark_cls(test="eval", device = device)

model, example = benchmark.get_module()

eager = model(*example)

print("eager works")

print("be patient compilation takes a while on this model")
compiled = torch.compile(model)
compiled_out = compiled(*example)
```

### Error logs

Surprisingly this model works both in eager and compiler but is showing up as failing in pt2 dashboard something else is going on

### Minified repro

_No response_

### Versions

n/a

cc @ezyang @wconstab @bdhirsh @anijain2305
[torch] Defer resolution of allowed/disallowed decorators
Labels: fb-exported, topic: not user facing, module: dynamo, ciflow/inductor

Summary:
To prevent triggering circular imports while decorating functions using
FunctionIdSet, we defer the resolution of the lazy initialization until the
time the object is used.

Test Plan: Existing tests

Differential Revision: D49350894




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Decompose to native_dropout in eval mode as well
Labels: ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #109914

Summary: https://github.com/pytorch/pytorch/pull/106274 added a
decomp for dropout such that for train mode we get
`aten.native_dropout`, while for eval mode we get `aten.clone`.
This commit makes it such that we get `aten.native_dropout` for
both train and eval modes. For eval mode, we let the cloning
happen in the aten op itself.

The main motivation behind this change is QAT, which needs to
swap between `aten.native_dropout(train=True)` and
`aten.native_dropout(train=False)` in the graph. This was
previously difficult to do since there was no dropout op to
match and replace in eval mode.

Test Plan: python test/test_ops.py

Reviewers: SherlockNoMad, bdhirsh

Subscribers: SherlockNoMad, bdhirsh, supriyar
[inductor] Avoid bool being upcast to int
Labels: open source, ciflow/trunk, topic: not user facing, module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #109913
* #110388

Currently the inductor code for `x.any(-1)` does a this strange dance:
```python
tmp0 = tl.load(in_ptr0 + (r1 + (128*x0)), rmask & xmask)
tmp1 = tmp0.to(tl.int64)
tmp2 = (tmp1 != 0)
```

This happens because `register_lowering` is doing type promotion with the
dimension argument, and so promotes to `int64` which we then cast back to bool.
A better fix would be to fix `register_lowering` but for now I just remove
the unnecessary type promotion from `aten.any`.

In the current code we also see:
```python
     tmp5 = tl.where(rmask & xmask, tmp3, 0)
```
which promotes the boolean value to int since `0` is an int32 in triton.
This fixes it to generate a boolean constant instead.

Finally there is also a triton bug where the `tl.load` itself upcasts to
`tl.int8`. I fix this by adding an explicit cast to `tl.int1`. The final
kernel code looks like:

```python
tmp0 = tl.load(in_ptr0 + (r1 + (128*x0)), rmask & xmask).to(tl.int1)
tmp1 = tl.broadcast_to(tmp0, [XBLOCK, RBLOCK])
tmp3 = tl.full([1, 1], 0, tl.int1)
tmp4 = tl.where(rmask & xmask, tmp1, tmp3)
tmp5 = triton_helpers.any(tmp4, 1)[:, None]

```
Dynamo error for autograd function
Labels: module: autograd, triaged, oncall: pt2, module: dynamo

# Summary
Encountering Dynamo error when attempting to compile an autograd function that returns a dtype.

### Repro
``` Python
import torch

class dtype_test(torch.autograd.Function):
    @staticmethod
    def forward(
        ctx,
        tensor: torch.Tensor,
        dtype=torch.dtype,
    ):
        orig_precision = tensor.dtype
        ctx.orig_precision = orig_precision
        return tensor.to(dtype), dtype
    @staticmethod
    def backward(ctx, dOut):
        return dOut.to(ctx.orig_precision), None, None,

def main():
    x = torch.randn(16, 16, device="cpu", dtype=torch.float32)
    out = dtype_test.apply(x, torch.bfloat16)
    def test_func(x, dtype):
        return dtype_test.apply(x, dtype)
    compiled_func = torch.compile(test_func, fullgraph=True)
    y = compiled_func(x, torch.bfloat16)

if __name__ == "__main__":
    main()
 ```
 
 ### Output
 ``` Shell
  File "/home/drisspg/miniconda3/envs/nightly/lib/python3.10/site-packages/torch/_dynamo/variables/base.py", line 329, in call_method
    raise unimplemented(f"call_method {self} {name} {args} {kwargs}")
  File "/home/drisspg/miniconda3/envs/nightly/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 176, in unimplemented
    raise Unsupported(msg)
torch._dynamo.exc.Unsupported: call_method TupleVariable() to [ConstantVariable(dtype)] {}

from user code:
   File "/home/drisspg/meta/float8_experimental/../scripts/compile/autograd_dtype_float.py", line 21, in test_func
    return dtype_test.apply(x, dtype)
  File "/home/drisspg/meta/float8_experimental/../scripts/compile/autograd_dtype_float.py", line 12, in forward
    return tensor.to(dtype), dtype
  ```

#### Note
This is a proxy issue I am creating for https://github.com/pytorch-labs/float8_experimental/issues/108


 

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Add `endpoint` argument in `linspace` to match numpy behavior
Labels: triaged, open source

Fixes #70919

Basically I copied the line from `numpy`
https://github.com/numpy/numpy/blob/d35cd07ea997f033b2d89d349734c61f5de54b0d/numpy/core/function_base.py#L125

There is also a slight change in @lezcano 's implementation using `torch.where` to improve performance by 10% when the `steps` is an odd number.

I updated the docs as well and I am not sure if this is needed. @mruberry
DISABLED test_tags_function (__main__.ActivationCheckpointingViaTagsTests)
Labels: triaged, module: flaky-tests, skipped, module: dynamo

Platforms: linux, rocm

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_tags_function&suite=ActivationCheckpointingViaTagsTests) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/17044930976).

Over the past 3 hours, it has been determined flaky in 4 workflow(s) with 12 failures and 4 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_tags_function`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `dynamo/test_activation_checkpointing.py`

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
moco: torch._dynamo.exc.Unsupported: hasattr: TensorVariable()
Labels: triaged, oncall: pt2

Repro:
```
python benchmarks/dynamo/torchbench.py --bfloat16 --accuracy --inference --device cuda  --export-aot-inductor  --only moco
```


cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
Revert D49433268: Multisect successfully blamed "D49433268: [pytorch][PR] [Inductor] Extend Pattern Matcher to Match Equivalent Function Invocation" for test or build failures
Labels: fb-exported, module: inductor, ciflow/inductor

Summary:
This diff is reverting D49433268
D49433268: [pytorch][PR] [Inductor] Extend Pattern Matcher to Match Equivalent Function Invocation by yanboliang has been identified to be causing the following test or build failures:

Tests affected:
- [caffe2/torch/fb/model_transform/experimental/benchmark/test/aotinductor:test_inductor_benchmark - test_inductor_benchmark_cmf30x (caffe2.torch.fb.model_transform.experimental.benchmark.test.aotinductor.test_inductor_benchmark.InductorBenchmark)](https://www.internalfb.com/intern/test/562950063524812/)

Here's the Multisect link:
https://www.internalfb.com/multisect/3104208
Here are the tasks that are relevant to this breakage:

We're generating a revert to back out the changes in this diff, please note the backout may land if someone accepts it.

If you believe this diff has been generated in error you may Commandeer and Abandon it.

Test Plan: NA

Reviewed By: hl475

Differential Revision: D49536556




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
Support torch.export.export through torch.onnx.dynamo_export
Labels: module: onnx, triaged, enhancement, onnx-triaged, release notes: onnx

### 🐛 Describe the bug

Currently, `torch.onnx.dynamo_export` leverages `torch._dynamo.export` to obtain a `GraphModule` graph. By leveraging `torch.export`, we may have a more stable and complete `GraphModule` graph, including:

* Functionalization.
* Lowering from torch ops to aten ops (aten ops are a lot more normalized than torch ops)
* Decomposition of Aten ops to a smaller opset (core aten ops)

@SherlockNoMad  FYI

### Versions

pytorch main
```[tasklist]
### Tasks
- [ ] https://github.com/microsoft/onnxscript/issues/1077
- [ ] https://github.com/pytorch/pytorch/issues/110100
```
Fix CPU bitwise shifts for out-of-limit values in VSX-vec
Labels: module: cpu, triaged, open source

Similar to #96659 this implements the conditionals handling the out-of-limit values in the shift amounts (rhs) for the vectorized VSX code using the same logic as the scalar code.

Fixes #109777

@quickwritereader @cdeepali Can you double-check?

A quick test from my side looks good


cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
Move at::{Refcounted,}MapAllocator to c10
Labels: open source, Merged, Reverted, ciflow/trunk, topic: not user facing, ciflow/periodic

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #109881

`libshm.so` depends on the torch library exclusively for `at::RefcountedMapAllocator`,
 so it makes sense to move it to c10 along with the other memory allocators.

This means `libshm.so` only depends on `c10` and we don't need to relink
`libshm.so` for every ATen change.
[FSDP ]How to convert sharded_state_dict files into full_state_dict offline without distributed process
Labels: oncall: distributed, triaged, module: fsdp, module: distributed_checkpoint

### 🚀 The feature, motivation and pitch

Currently, if I use FSDP with 128 gpus and save checkpoints with sharded_state_dict to avoid gathering the full_state_dict on rank0 for saving, there is no way to obtain the full_state_dict ckpt offline. 

The only way to obtain full_state_dict is to launch the exact 128GPU distributed process with FSDP to load that sharded_state_dict model, then switch to full_state_dict config and save the ckpt to files, which is originally problem we wanted to avoid.

I cannot read the sharded_state_dict file (with `torch.load()`) individually either, except if I launch a 128gpu distributed process to read it. The file contain `ShardedTensor` which requires the same world_size=128 to load.

I would like to have an offline script to read each sharded file and write iterative to a pytorch_model_0.bin, pytorch_model_1.bin, pytorch_model_2.bin...

And then we can load the model with `AutoModelForCausalLM.from_pretrained(...)` by loading each `.bin`

Thanks!

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin
Allow try except check for numpy bfloat16 representation
Labels: triaged, module: numpy

### 🚀 The feature, motivation and pitch

if you run this code :
```
import ml_dtypes
import torch
torch.tensor([1,2,3],dtype=torch.bfloat16).numpy()
```
It will throw this error :
```TypeError: Got unsupported ScalarType BFloat16```
even when numpy will support `bfloat16` in this case. If you guys could add a try except check then it would be cool.

### Alternatives

_No response_

### Additional context

_No response_

cc @mruberry @rgommers
DISABLED test_tags_dropout (__main__.ActivationCheckpointingViaTagsTests)
Labels: triaged, module: flaky-tests, skipped, module: dynamo

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_tags_dropout&suite=ActivationCheckpointingViaTagsTests) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/17031141549).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 6 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_tags_dropout`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `dynamo/test_activation_checkpointing.py`

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Wrongly returns nan for vectorized complex numbers division on PPC/ZArch
Labels: triaged, module: POWER

### 🐛 Describe the bug

Copied from #92043 which still applies for vectorized PPC and likely to ZArch while it was fixed for x86 by #93277

Pytorch wrongly returns nan for complex numbers division where the results are still within the range of the datatype.
For example: (0.0 + 0.0j) / (1e-36 + 0.0j) in complex64. In this case, both operands and the results (which is 0) are within the defined range of complex64. However, it returns (nan + nanj).

Here's the reproducing code:
```
a = torch.tensor([0.0 + 0.0j]*8)
b = torch.tensor([1e-36 + 0.0j]*8)
print(a / b)  # it's (nan + nanj),...
```

Or just run `test_complex_div_underflow_overflow`.

The reason is that [`abs_2_`](https://github.com/pytorch/pytorch/blob/d7c05bb2e8de24386664c01e887357ff50a09842/aten/src/ATen/cpu/vec/vec256/vsx/vec256_complex_double_vsx.h#L434) already overflows in the tested range (e.g. `finfo.min / 2` in `test_complex_div_underflow_overflow`) and the `elwise_mult(vr)` does too.

### Versions

PyTorch 2.1.0
DISABLED test_tags_decomps (__main__.ActivationCheckpointingViaTagsTests)
Labels: triaged, module: flaky-tests, skipped, module: dynamo

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_tags_decomps&suite=ActivationCheckpointingViaTagsTests) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/17027062892).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 6 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_tags_decomps`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `dynamo/test_activation_checkpointing.py`

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
DISABLED test_symints_location (__main__.ActivationCheckpointingViaTagsTests)
Labels: module: rocm, triaged, module: flaky-tests, skipped, module: unknown, oncall: pt2

Platforms: linux, rocm

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_symints_location&suite=ActivationCheckpointingViaTagsTests) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/17017086249).

Over the past 3 hours, it has been determined flaky in 11 workflow(s) with 33 failures and 11 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_symints_location`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `dynamo/test_activation_checkpointing.py`

ResponseTimeoutError: Response timeout for 5000ms, GET https://raw.githubusercontent.com/pytorch/pytorch/main/test/dynamo/test_activation_checkpointing.py -1 (connected: true, keepalive socket: false, socketHandledRequests: 1, socketHandledResponses: 0)
headers: {}

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
Fix S367052 to unblock ICVR MC3
Labels: fb-exported, Merged, ciflow/trunk, release notes: fx

Summary: Somehow "getitem" started to get Tensor starting from ads_ranking:996 and broke SDD pipelining FX-transformer. We need to skip the Tensor node in annotation.

Test Plan:
N4326037

# Before
 {F1099052907}

# With this diff

 {F1099052270}

Differential Revision: D49528046
torch._export has no logging
Labels: module: logging, triaged, oncall: pt2, module: export

### 🐛 Describe the bug

torch._export has a fairly sophisticated pass infrastructure. But there is no logging.

```
$ git grep log torch/_export/
torch/_export/serde/serialize.py:import logging
torch/_export/serde/serialize.py:log = logging.getLogger(__name__)
torch/_export/serde/serialize.py:                log.warning(f"Symbol {k} did not appear in the graph that was deserialized")  # noqa: G004
torch/_export/serde/serialize.py:        The logic of this method:
torch/_export/serde/serialize.py:            log.warning("Compiler doesn't have a version table for op namespace: {ns}. ", extra={"ns": namespace})
torch/_export/serde/upgrade.py:import logging
torch/_export/serde/upgrade.py:log = logging.getLogger(__name__)
torch/_export/serde/upgrade.py:                log.warning("Missing an upgrader to upgrade to version {ver}.", extra={"ver": ver})
```

At minimum, I would expect there to be a way to log out each IR after each pass. Probably more things to log too.

cc @msaroufim @wconstab @bdhirsh @anijain2305 @gmagogsfm @tugsbayasgalan 

### Versions

main
Implement Copy-on-write (COW) tensors
Labels: module: internals, triaged, module: viewing and reshaping

### 🚀 The feature, motivation and pitch

Copy-on-write (COW) tensor is a new kind of tensor that copies its underlying data to a new location if an operation tries to mutate the data.

COW tensors were proposed in [this document](https://docs.google.com/document/d/1kWIRUeixlgnOk1eXJSsbxOAKUKQFpmz0yzwLZUjWotg/edit?usp=sharing), which provides the full context of the idea and the problems it solves. Briefly, here's my summary of the problems, as far as I understand them at the moment:

* Currently, there are operators like `torch.reshape` which conditionally return either a copy or a view of a tensor, depending on striding. This semantic difference is an issue for the compiler, and it would be better to change these operators to always semantically return a copy. However, even if we semantically do a copy, we can use COW tensors to avoid actually doing the copy until it's absolutely necessary.

* When a user saves a tensor for backward and then an inplace operator mutates a view of that same tensor, the saved tensor no longer has the value that the user wanted to have saved. People have learned to avoid this by simply avoiding inplace operators in this case, but it would be better if users could use inplace operators without shooting themselves in the foot like this. If the saved tensor is a COW tensor, then "views" of it will actually just be copies, and the saved tensor's data will not change if the copies are mutated

Some machinery already exists in [`c10/core/impl/cow/`](https://github.com/pytorch/pytorch/tree/c789ed6e62cdee9917ae08a432ab9b6be8d59e64/c10/core/impl/cow)



### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @bhosmer @smessmer @ljk53 @bdhirsh
DISABLED test_kwargs (__main__.ActivationCheckpointingViaTagsTests)
Labels: triaged, module: flaky-tests, skipped, module: dynamo

Platforms: linux, rocm, slow

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_kwargs&suite=ActivationCheckpointingViaTagsTests) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/17013126575).

Over the past 3 hours, it has been determined flaky in 8 workflow(s) with 24 failures and 8 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_kwargs`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `dynamo/test_activation_checkpointing.py`

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
ValueError: only one element tensors can be converted to Python scalars
Labels: needs reproduction, triaged, module: regression

### 🐛 Describe the bug

This used to work. Now it doesn't.

```python
import torch
from matplotlib.pyplot import contourf, axis

mesh = torch.arange(-1, 1.1, 0.1)
x, y = torch.meshgrid(mesh, mesh, indexing='xy')
z = torch.randn_like(x)

# Breaking line
contourf(x, y, z)
axis('square')
```

`math.isfinite(val)` says `ValueError: only one element tensors can be converted to Python scalars`

To fix it, one can simply run

```python
contourf(x.numpy(), y.numpy(), z)
axis('square')
```

### Versions

Latest.
[Inductor CUTLASS backend] Epilogue fusion codegen prototype
Labels: topic: not user facing, module: inductor, ciflow/inductor

Draft pull request to discuss the changes while development is still in progress
@aakhundov @ipiszy 

This is unfinished, please don't review yet. 

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @muchulee8 @aakhundov @ColinPeppler
Bits types cannot be used under deterministic mode
Labels: triaged, module: determinism

#104995 made it impossible to use bits types under deterministic mode:
```
import torch
torch.set_deterministic_debug_mode("warn")
x=torch.empty(4, dtype=torch.bits8)
```
produces `RuntimeError: "fill_empty_deterministic_" not implemented for 'Bits8'`
While I'm sympathetic to the goals of #104995, I think that a blanket approach forcing filling all empty tensors is too strict. In many cases, determinism mode is set in production runs (because if only deterministic ops are used it imposes very little overhead and provides nice guarantees), and with valid code the empty calls are not a problem. The approach taken in #104995 requires paying filling penalty unconditionally, and also requires implementing fill op for all bits types which pytorch currently doesn't do. 
Deterministic empty tensors is a valuable option to have, but in my opinion should not be mixed with existing deterministic mode. 

cc @mruberry @kurtamohler
Surround num-destroyed-communicators with spaces
Labels: fb-exported, release notes: distributed (c10d)

Summary:
As title says

Lines look like this
{F1095151529}

Created from CodeHub with https://fburl.com/edit-in-codehub

Test Plan:
NOP change

waitforsandcastle

Sandcastle run

Reviewed By: shnavid

Differential Revision: D49378701
[inductor] Add lowering for aten.take
Labels: open source, ciflow/trunk, module: inductor, ciflow/inductor

Adds lowering for `aten.take`.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @lezcano
`torch.embedding`, `weight[indices]`, `torch.index_select` returns random data with indices on meta device
Labels: module: bc-breaking, triaged, module: embedding, ezyang's list, module: meta tensors, topic: bc breaking

### 🐛 Describe the bug

```python
import torch

weight = torch.ones(2,3)  # CPU
indices = torch.zeros(4,5, device="meta", dtype=torch.int32)
y1 = torch.embedding(weight, indices)
y1a = torch.embedding(weight, indices)
y2 = weight[indices]
y2a = weight[indices]
y3 = torch.index_select(weight, dim=0, index=indices.flatten()).reshape(indices.shape + weight.shape[1:])
y3a = torch.index_select(weight, dim=0, index=indices.flatten()).reshape(indices.shape + weight.shape[1:])
```
Now `y1`, `y1a`, `y2`, `y2a`, `y3`, `y3a` contain random data. And even all those `y*` are totally different, so it's non-deterministic.

Actually, I wondered also, why does this not throw an exception, such that it only works when both inputs are on the same device?


Or maybe `torch.embedding` actually supports mixing devices, e.g. `indices` on CPU?

(Slightly off-topic question: What is really the difference between `torch.index_select(weight, dim=0, index=indices.flatten())` + `reshape`, `weight[indices]` and `torch.embedding(weight, indices)`? I don't really know when to use what.)

I'm not really sure what output to expect. Maybe also some tensor on the "meta" device?

### Versions

Collecting environment information...
PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.6.3 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.202)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:24:02)  [Clang 11.1.0 ] (64-bit runtime)
Python platform: macOS-12.6.3-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] evotorch==0.1.0
[pip3] lovely-numpy==0.2.8
[pip3] mypy==0.991
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.24.2
[pip3] torch==2.0.1
[pip3] torch-tb-profiler==0.4.1
[pip3] torchaudio==2.0.2
[pip3] torchdata==0.6.1
[pip3] torchvision==0.15.2
[conda] Could not collect


cc @ezyang @gchanan @eellison @bdhirsh
Wrong vector shift results on PowerPC
Labels: triaged, module: vectorization, module: POWER

### 🐛 Describe the bug

This is basically https://github.com/pytorch/pytorch/issues/70904 which is still present for PPC after https://github.com/pytorch/pytorch/pull/98511 added vector operations.

Basically: Shifting a vector with out-of-bounds values results in undefined behavior and sporadic wrong results.
E.g. test_non_contig_bitwise_right_shift_cpu_int32 fails because of right shifts with too-large values and test_shift_limits_cpu_int16 fails due to left shifts with negative values. The latter is very much understandable given that #98511 uses a `reinterpret_cast` to the unsigned vector type

Reproduced with `python run_test.py -i test_binary_ufuncs` which fails

- test_contig_vs_every_other_bitwise_right_shift_cpu_int16
- test_contig_vs_every_other_bitwise_right_shift_cpu_int32
- test_contig_vs_every_other_bitwise_right_shift_cpu_int64
- test_non_contig_bitwise_right_shift_cpu_int16
- test_non_contig_bitwise_right_shift_cpu_int32
- test_non_contig_bitwise_right_shift_cpu_int64
- test_shift_limits_cpu_int16
- test_shift_limits_cpu_int32
- test_shift_limits_cpu_int64

### Versions

Using PyTorch 2.0.1 with the above PRs applied as patches.
DTensor: summon full tensor API?
Labels: oncall: distributed

### 🚀 The feature, motivation and pitch

Wondering if there's an API or plans to support materializing the full DTensor from their shards. I couldn't quite find anything in https://github.com/pytorch/pytorch/blob/main/torch/distributed/_tensor/api.py, but might be missing something. 

This could be useful for debugging / iterating on issues related to DTensor. Currently, the following seems like a workaround:

```
# redistribute to single world size
one_ws = DeviceMesh(dist.get_rank())
redist = dtensor.redistribute(one_ws)
whole_tensor = redist.to_local()
```

but not sure.

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu
[Quantization] Add "quantization_tag" as metadata to fx proxy
Labels: release notes: quantization

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #109749

Summary:
In order to make sure that quantization_tag is preserved through second
stage export, this PR adds it as a special metadata that should be
preserved.

Since quantization in export path will work on top of pre dispatch
graph, subsequent post dispatch op decomposition, will decompose ops
that quant workflow tagged. In order to make sure that the patterns
identified by quantizer, remains identifiable, even after decompositions
are applied, we must preserve "quantization_tag".

This enables backend delegates, that quantized a model for specific
backend, to be able to identify "quantized" patterns.

Test Plan:
metadata porting tests

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: [D49481562](https://our.internmc.facebook.com/intern/diff/D49481562)
[RFC][TorchElastic] topology info in training apps/ranks
Labels: oncall: distributed, triaged, module: elastic

### 🚀 The feature, motivation and pitch

RFC PR: https://github.com/pytorch/rfcs/pull/57 ([preview]( https://github.com/pytorch/rfcs/pull/57/files?short_path=c8cb2fb#diff-c8cb2fba3757888ba87a91b1f9c5d39c5e19b64b6289994d0e6d2e699aac625e)) 

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @dzhulgakov @kiukchung @d4l3k @wconstab 

### Alternatives

_No response_

### Additional context

_No response_
[caffe2/torch] Package Importer with compatibility for Lazy Imports
Labels: fb-exported, release notes: package/deploy

Summary:
This adds Lazy Imports compatibility to the Package Importer. Adds support for "lazy side effects".

With lazy imported modules, those don't add attributes to children submodules to parents. Lazy Imports adds "lazy attributes". These of course need to be added somehow. Differently from vanilla CPython, where it simply adds attributes using `setattr()`, Cinder with Lazy Imports has two APIs for doing this that need to be called during the import process:

+ `_imp._maybe_set_parent_attribute()`
+ `_imp._set_lazy_attributes()`

This diff adds support to using these APIs while importing modules in PyTorch package importer.

Test Plan:
This should be a noop if the library is not using Cinder and lazy imports.

---

Existing tests

Differential Revision: D46404107
[C10D] Report detected failures when emitting collective end events.
Labels: open source, release notes: distributed (c10d)

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #109739
* #111239
* #111246

We leverage the fact that errors get tracked in Work to report them.

The error message by itself is not crazy useful and we might need to
include some form of error code in the mix (nccl failure, timeout, etc).
[dynamo] lift the constraint that cannot make_fx a dynamo compiled function
Labels: ciflow/trunk, module: dynamo, ciflow/inductor, release notes: dynamo

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #109734
* #109916

Since we've added support for FakeTensor and SymBool inputs in dynamo, this PR removes the constraint that cannot make_fx a dynamo compiled function.

Checking the commit history, the is_fx_tracing flag in dynamo [code here](https://github.com/pytorch/pytorch/blob/main/torch/_dynamo/eval_frame.py#L382C1-L383C1) seems to be mainly used to provide a nice error message for symbolic_trace. This PR changes how to detect we're during symbolic_trace by looking at is_fx_tracing flag and checking any arguments are proxies. This allows make_fx to be able to call into an optimized function.

**Test Plan:**
See added test.

 

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Histogram Fixes for QAT
Labels: fb-exported, release notes: quantization, release notes: AO frontend

Summary: Histogram observer needs to avoid in-place detach , followed by re-shape tensors for per-channel quant

Test Plan:
Before:
```
File "/tmp/jetter.vxtgekjr/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/tmp/jetter.vxtgekjr/torch/ao/quantization/observer.py", line 1195, in forward
    self.min_val.detach_().resize_(combined_min.shape)
RuntimeError: Can't detach views in-place. Use detach() instead. If you are using DistributedDataParallel (DDP) for training, and gradient_as_bucket_view is set as True, gradients are views of DDP buckets, and hence detach_() cannot be called on these gradients. To fix this error, please refer to the Optimizer.zero_grad() function in torch/optim/optimizer.py as the solution.
```
After:
no issue.

Differential Revision: D49466364
Profiler should implicitly synchronize gpu devices 
Labels: oncall: profiler

### 🚀 The feature, motivation and pitch

There are currently some unit tests that are flakey due to a race condition when using the profiler.  The pattern typically looks like this:

>             with torch.profiler.profile() as p:
>                 actual = self.func(*inputs, **kwargs)
>             keys = tuple([e.key for e in p.key_averages()])
>             mta_called = any("multi_tensor_apply_kernel" in k for k in keys)

This intent here is to call a function and see if a particular kernel was run.  However it is starting the profiler, dispatching the kernel, then immediately stopping the profiler.  There is no guarantee the kernel executed before the profiler was stopped.  This is a pure race.

The proper fix is to add a 'torch.cuda.synchronize()' inside the context block.  However, it will be an ongoing struggle to identify every unit test that makes this error (including the new ones).  Would it be appropriate to have the context manager add an implicit device synchronize in __exit__?

Generally, a profiler should not be changing the behavior of what it is being profiled.  Would this case be a worthy exception, under principle of least surprise, to benefit casual users?

cc @robieta @chaekit @aaronenyeshi @nbcsm @guotuofeng @guyang3532 @gaoteng-git @tiffzhaofb @dzhulgakov @davidberard98 @jithunnair-amd @pruthvistony 

### Alternatives

_No response_

### Additional context

This seems to be causing random failures for any of the `is_fastpath` tests e.g. `TestForeachCUDA::test_lerp_is_fastpath_True__foreach_lerp_cuda_float32`
Make standard container classes satisfy container Protocols.
Labels: module: nn, triaged, needs research

### 🚀 The feature, motivation and pitch

The container classes in [`torch/nn/modules/container`](https://github.com/pytorch/pytorch/blob/b30ee35a6f141d3247a24fd09f96ea50a7e2b3c7/torch/nn/modules/container.py) satisfy the necessary methods to use corresponding `collections.abc` Mixins, I propose to either subclass these, or to satisfy the protocols manually:

- `nn.Sequential` -> `MutableSequence[M]`
- `nn.ModuleList` -> `MutableSequence[M]`
- `nn.ModuelDict` -> `MutableMapping[str, M]`
- `nn.ParameterList` -> `Sequence[P]` (missing: `__delitem__` and `insert` for `MutableSequence[M]`)
- `nn.ParameterDict` -> `MutableMapping[str, P]`
- `M` would be a `TypeVar` bound to `nn.Module`.
- `P` would be a `TypeVar` bound to `nn.Parameter`.

### Alternatives

_No response_

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
[DDP + Dynamo] Traceable DDP hooks
Labels: oncall: distributed, triaged, oncall: pt2

### 🚀 The feature, motivation and pitch

https://github.com/pytorch/pytorch/pull/106738 has support to trace FSDP.  Is there any plan to add similar support for traceable DDP along with using functional collectives?

With support for CompiledAutoGrad + traceable DDP hook I am expecting the complete graph including the backward graph + (mul + allreduce) for all buckets which are present in the DDP hooks to be available at the compiler backend.  This helps the custom compiler backend to schedule the Allreduce calls for better compute + communication overlap.

def compiler_fn(gm):
    return torch.compile(gm, backend="inductor", fullgraph=True)

ddp_m = DDP(m, gradient_as_bucket_view=True, static_graph=True)
ddp_m = torch.compile(ddp_m, backend="inductor")
with compiled_autograd.enable(compiler_fn):
    loss = ddp_m(inp, target)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

The current DDP hooks are not traceable where the first place dynamo tracing breaks is with this particular stack trace during pre_forward hooks.

> inputs, kwargs = self._pre_forward(*inputs, **kwargs)
> self._lazy_init()
> self._lazy_init_ran = True.

The graph compilation also fails with this particular error:
> AttributeError: type object '_DDPSink' has no attribute '_compiled_autograd_key'

Other than this there are multiple parts of the DDP backward hook implementation which are not traceable.  The expectation is with traceable DDP hook + CompiledAutoGrad by registering a comm hook with functional collectives we can get a combined graph along with the collective calls at the compiler backend.

@wconstab 

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
Extends the functionality of  `nn.BatchNorm1d`.
Labels: oncall: distributed, module: nn, triaged, needs research

### 🚀 The feature, motivation and pitch

This feature request proposes the extension of `BatchNorm1d` to the PyTorch library. `NaiveSyncBatchNorm1d` is an extension of the existing `nn.BatchNorm1d` module, designed to support synchronization across multiple devices, either locally or globally. 

The motivation behind this feature request is to enhance PyTorch's capabilities in distributed deep learning scenarios. the proposed `NaiveSyncBatchNorm1d` module would fill this gap.

I propose the addition of the `NaiveSyncBatchNorm1d` module to PyTorch. The module provides the following features:
- Synchronization of batch normalization statistics (mean and variance) across multiple devices.
- Support for both global synchronization and local synchronization based on user requirements.

### Alternatives

While there are alternative ways to implement synchronized batch normalization, `NaiveSyncBatchNorm1d` offers a simple and efficient solution. 


### Additional context

Here's an example of how `NaiveSyncBatchNorm1d` can be used in PyTorch:

```python
sync_bn = NaiveSyncBatchNorm1d(num_sync_devices=4, global_sync=False, num_features=64)
output = sync_bn(input_tensor)
```

Source: https://github.com/facebookresearch/pytorchvideo/blob/64e5a17ccefcd6b93ad331d1a9c2a130f179ff44/pytorchvideo/layers/batch_norm.py#L10C44-L10C44

I would like to create a PR for the same.

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
[pytree] Make `optree` optional and populate members from `_pytree` when not available
Labels: triaged, open source, ciflow/trunk, release notes: releng, module: pytree, ciflow/inductor, module: export

Follow-up PR for #93139

- #93139

Changes:

- Remove `torch/utils/_pytree.py` from the lint ignore list.
- Move functions in `torch._functorch.pytree_hacks` to `torch.utils._pytree` and update imports.
- Remove duplicate registration for immutable collections.
- Make `optree` optional in `torch.utils._cxx_pytree` and populate members from `_pytree` when not available.
- Register pytree nodes in both Python pytree and CXX pytree.
- Format files and update type hints.

See also: https://github.com/pytorch/pytorch/pull/93139#discussion_r1329479085

> -> Diff 1 add new dep
> Diff 2 Integration behind disabled flag
> Diff 3 pass tests with flag enabled and disabled
> Diff 4 set flag to default on let soak a few weeks
> Diff 5 remove flag, remove old system

cc @zou3519 @avikchaudhuri @gmagogsfm
[WIP] Dynamo CPU backend under Windows
Labels: open source, module: inductor, module: dynamo, ciflow/inductor, release notes: dynamo

Adding dynamo CPU backend support under Windows #90768 

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[FSDP] UnpicklingError when calling save_state_dict in distributed run
Labels: oncall: distributed, triaged, module: fsdp

### 🐛 Describe the bug

we are using multi-nodes training with FSDP, and we got the following error during checkpointing through `torch/distributed/checkpoint/state_dict_saver.py`
```
File "/opt/ml/code/checkpoints.py", line 153, in _save
checkpoint.save_state_dict(
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/state_dict_saver.py", line 119, in save_state_dict
return distW.all_reduce("write", write_data, finish_checkpoint)
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/utils.py", line 236, in all_reduce
final_result = self.broadcast_object(result)
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/utils.py", line 88, in broadcast_object
dist.broadcast_object_list(
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1451, in wrapper
return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2277, in broadcast_object_list
object_list[i] = _tensor_to_object(obj_view, obj_size)
  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1970, in _tensor_to_object
return _unpickler(io.BytesIO(buf)).load()
_pickle.UnpicklingError: invalid load key, '\x00'.
```
originally, we thought that it only happened when we overwrite checkpoint to an existing folder, but we recently observed the error occurred even when writing to fresh folder. thus this error is non-deterministic. it might be related to these issues https://github.com/pytorch/pytorch/issues/109396 

### Versions

```
Collecting environment information...
PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A
```

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin
FSDP: ShardedStateDict support for world_size = 1
Labels: oncall: distributed, module: fsdp

### 🐛 Describe the bug

When iterating with FSDP code, it's sometimes useful to set world_size = 1 to sanity check some things before launching larger job. However, this currently requires switching the state_dict code logic to use FULL_STATE_DICT instead of ShardedStateDict, which is a bit inconvenient from developer velocity perspective.

This can be repro'd by setting world_size = 1 + using StateDictType.ShardedStateDict, and trying to load in a checkpoint.

### Versions

main

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin
Add Pass to move constructors from cpu to cuda
Labels: ciflow/trunk, module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #109665

Sometimes indexing tensors are constructed on cpu and then used to index a cuda tensor. This prevents cudagraphs when it does not need to. Adding a pass which moves constructors from cpu->cuda when we can prove the downstream uses can be safely converted.

This pr allows us to cudagraph `clip` from the blueberries model which improves perf from ~1.5x -> ~4x.




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[WIP] compiled autograd on inductor torchbench
Labels: release notes: releng, module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #109653

![image](https://github.com/pytorch/pytorch/assets/4755252/50104ab8-80f4-45b5-99ef-d4018943cfd2)
InstanceNorm does not catch dim mismatch
Labels: module: nn, triaged, actionable

### 🐛 Describe the bug

InstanceNorm with a specified input C works on tensors of arbitrary dimensions/channels (with the number of channels not necessarily matching C) without throwing a dimension mismatch error. For example:

```
m = torch.nn.InstanceNorm1d(64)
input = torch.randn(4, 80, 300)
output = m(input)
```

### Versions

PyTorch v2.0

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki