gopikrsmscs/torch-issues huggingface dataset:
Module states cannot be fully synchronized due to the DDP broadcast_buffers breaking change
Labels: oncall: distributed

### 🐛 Describe the bug

From the recent breaking change in the `2.1.0` in the https://github.com/pytorch/pytorch/pull/100729, when the `broadcast_buffers` is set to `False`, it caused the DDP to not fully synchronize the module states (i.e., only the parameter is synchronized but the buffer is being left out out-of-sync) during the DDP initialization and the final step joining. This caused some issues in the following scenarios. 
1. With the random initialization, the master process module's parameter is broadcasted but not the buffer. While we can fix this by ensuring that all ranks are initialized with the same random seed, there are however some scenarios where we still want to use a different random seed for each rank. 
2. With the checkpoint loading, previously we only needed to load the checkpoint in the master process and the states (parameters and buffers) will be broadcasted to all other processes. While we can fix this by loading the exact same checkpoint in each process, there's also a scenario in which not all processes would have direct access to the checkpoint, for example, in a multi-node configuration where only the master node has access to the checkpoint which is located in the master node locally.

Also in the DDP [documentation](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel), it specifically states that the `broadcast_buffers` flag is used to control the buffer broadcasting at the beginning of a module forward function, but not during the initialization and module final joining. 
```
broadcast_buffers ([bool] – Flag that enables syncing (broadcasting) buffers of the module at the beginning of the forward function. (default: True)
```

The documentation states a different behavior from the new changes, and the new change also caused some issues mentioned above in the module loading in the DDP, is this an intended behavior or it's a bug? 



### Versions

2.1.0

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu
"Invalid Scalar type" when using bf16 allreduce with Gloo backend
Labels: oncall: distributed

### 🐛 Describe the bug

When I do a bf16 allreduce on GPU tensors with the gloo backend, I hit an "Invalid scalar type" issue. It seems that the `GENERATE_ALL_TYPES` logic in gloo does not support bf16.

### Versions

main

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu
Update ROCm triton pin
Labels: module: rocm, open source, ciflow/trunk, topic: not user facing, ciflow/periodic, ciflow/inductor, keep-going

Changes:
- Enables bfloat16 support in MFMA dot on MI200 (https://github.com/ROCmSoftwarePlatform/triton/commit/23979098c881536a52ba3edf8164e12296ecfe7b)
- Add support for int8 to bfloat16 conversion (https://github.com/ROCmSoftwarePlatform/triton/commit/2d3e38e182e7b9b21aadb90bc2ab0d6514b3c760) fixing a bug in bf16 triton gemm workloads.
- Enable scanOp lowering by adding shfl_up support https://github.com/ROCmSoftwarePlatform/triton/pull/324
- MFMA16 support - support for the mfma_16x16xX instructions - these help perf on smaller sized GEMMs - https://github.com/ROCmSoftwarePlatform/triton/commit/7e34c244c284a84191a1a7bb0cd484c6345de650
- configurable wavefront-per-eu - this helps us increase our occupancy in certain use cases such as Flash Attention - https://github.com/ROCmSoftwarePlatform/triton/commit/e801638b40dac4fd511f973d9899f033ae94dbec
- Many bug fixes and optimisations


cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @hongxiayang
type promotion test for torch.div variants is broken
Labels: triaged, module: testing

The OpInfo for `torch.div` for the variants `floor_rounding` and `trunc_rounding` currently set `promotes_int_to_float=True`:

https://github.com/pytorch/pytorch/blob/b35279dfac2950c053d005fdbe179312c5f9d921/torch/testing/_internal/common_methods_invocations.py#L10882

This is wrong and will be removed in #110939. In turn, this will trigger regular type promotion checks for binary ops:

https://github.com/pytorch/pytorch/blob/b35279dfac2950c053d005fdbe179312c5f9d921/test/test_binary_ufuncs.py#L522-L531

This test however doesn't use the regular sample inputs defined on the op, but rather hand-crafted ones:

https://github.com/pytorch/pytorch/blob/b35279dfac2950c053d005fdbe179312c5f9d921/test/test_binary_ufuncs.py#L474-L476

For regular sample inputs, we set the `rounding_mode` through a `functools.partial` to the sample inputs function

https://github.com/pytorch/pytorch/blob/b35279dfac2950c053d005fdbe179312c5f9d921/torch/testing/_internal/common_methods_invocations.py#L10878

Meaning, the `rounding_mode="floor"` or `rounding_mode="trunc"` never make it to the type promotion test. The test calls `torch.div`, which by default performs true division and thus `promotes_int_to_float=True`, but the test expects it to perform regular type promotion.
torch::serialize::OutputArchive::save_to crash if save on C:\
Labels: module: windows, module: serialization, triaged

### 🐛 Describe the bug

```
string model_path = "C:\\model.pt";
serialize::OutputArchive output_archive;
cp_net.save(output_archive);
output_archive.save_to(model_path);

```
Then crash - empty file are created.

If add some folder like this:
`string model_path = "C:\\Folder\\model.pt";`
Works fine

### Versions

libtorch-win-shared-with-deps-2.1.0+cpu

MSVC++Windows 10 VS 2022

cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite @mruberry @mikaylagawarecki
Add comment to keep PYI041 disabled
Labels: triaged, open source, release notes: distributed (fsdp), topic: not user facing, module: inductor, ciflow/inductor

Enable [redundant-numeric-union (PYI041)](https://docs.astral.sh/ruff/rules/redundant-numeric-union/)

Link: #110950


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
Reduce overhead in cudagraph trees
Labels: ciflow/trunk, module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #111094
* #111016



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
[HigherOrderOp] Move map_impl to torch.ops.higher_order
Labels: ciflow/trunk, module: export, release notes: export

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #111152
* #111147
* __->__ #111092

As titled.

Test Plan:
existing tests.


cc @avikchaudhuri @gmagogsfm @zhxchen17 @tugsbayasgalan

Differential Revision: [D50232731](https://our.internmc.facebook.com/intern/diff/D50232731)
[AOTInductor] Wrapper codegen fixes
Labels: fb-exported, topic: not user facing, module: inductor, ciflow/inductor

Summary:
The changes:
1. When using fake inputs make sure they are on the same device as the original inputs.
1. Don't change the value of `self.cpp_wrapper` from `True` to `False` if can't generate a C++ wrapper, instead have a check and fail early to avoid producing Python code for C++ compiler.

Differential Revision: D50154720




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
Build failure with Xcode 15 linker
Labels: module: build, triaged, module: macos

### 🐛 Describe the bug

When building with Apple Clang 15, I see the following build error:
```
  FAILED: lib/libtorch_cpu.dylib
  ...
  ld: warning: duplicate -rpath '/Users/Adam/spack/opt/spack/darwin-sonoma-m2/apple-clang-15.0.0/pthreadpool-2021-04-13-3jrtr6fgfmbe7yl2ele6dtttwljr3iqk/lib' ignored
  ld: warning: duplicate -rpath '/Users/Adam/spack/opt/spack/darwin-sonoma-m2/apple-clang-15.0.0/protobuf-3.24.3-uj3r4mhyae45adjcmliahvn3rsjyguxu/lib' ignored
  ld: warning: duplicate -rpath '/Users/Adam/spack/opt/spack/darwin-sonoma-m2/apple-clang-15.0.0/abseil-cpp-20230125.3-geqbwotmm5vdh6tdbwwnz6ox66ixihjd/lib' ignored
  ld: warning: ignoring duplicate libraries: 'lib/libcpuinfo.a', 'lib/libnnpack.a'
  ld: multiple errors: duplicate LC_RPATH '/Users/Adam/spack/opt/spack/darwin-sonoma-m2/apple-clang-15.0.0/gcc-13.2.0-qhn4xdjhn2p7k2fnvllfz2ca6xhwl2yu/lib' in '/Users/Adam/spack/opt/spack/darwin-sonoma-m2/apple-clang-15.0.0/openblas-0.3.24-lsn4jcy34p62ysj7qq2ghlsawu6sco6a/lib/libopenblas-r0.3.24.dylib'; duplicate LC_RPATH '/Users/Adam/spack/opt/spack/darwin-sonoma-m2/apple-clang-15.0.0/gcc-13.2.0-qhn4xdjhn2p7k2fnvllfz2ca6xhwl2yu/lib' in '/Users/Adam/spack/opt/spack/darwin-sonoma-m2/apple-clang-15.0.0/openblas-0.3.24-lsn4jcy34p62ysj7qq2ghlsawu6sco6a/lib/libopenblas-r0.3.24.dylib'
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
```
This appears to be due to the new linker added in Xcode 15. I'm able to build successfully using the following workaround:
```
export LDFLAGS="-Wl,-ld_classic"
```
See the build logs for details:

* [spack-build-out.txt](https://github.com/pytorch/pytorch/files/12874928/spack-build-out.txt)
* [spack-build-env-mods.txt](https://github.com/pytorch/pytorch/files/12874929/spack-build-env-mods.txt)

### Versions

Script fails because I don't have pip installed.

* PyTorch 2.1.0
* Apple Clang 15.0.0
* macOS Sonoma 14.0
* Apple M2 Pro (arm64)
* No CUDA/CUDNN/NCCL
* MPS

cc @malfet @seemethere @albanD
Getting "master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified" warning when using rdzv.
Labels: oncall: distributed

### 🐛 Describe the bug

Here we print based on whether `master_addr` is defined:
- https://github.com/pytorch/pytorch/blob/4d29b40299e9af1e4287b48024b6ef28b7cbf738/torch/distributed/run.py#L695-L699

But it is always defined:
- https://github.com/pytorch/pytorch/blob/4d29b40299e9af1e4287b48024b6ef28b7cbf738/torch/distributed/run.py#L565

This PR is related @cz-37 @d4l3k :
- https://github.com/pytorch/pytorch/pull/88922

### Versions

latest

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu
There is a performance drop because we have not yet implemented the batching rule for aten::mkldnn_rnn_layer_backward.
Labels: triaged, module: vmap, module: functorch

### 🐛 Describe the bug

Hi all,

I just received the following warning, and I am doing what is suggested there. Many thanks!

```
/home/kshamsaei/miniconda3/envs/torch/lib/python3.11/site-packages/torch/autograd/__init__.py:303: UserWarning: There is a performance drop because we have not yet implemented the batching rule for aten::mkldnn_rnn_layer_backward. Please file us an issue on GitHub so that we can prioritize its implementation. (Triggered internally at /opt/conda/conda-bld/pytorch_1682343995622/work/aten/src/ATen/functorch/BatchedFallback.cpp:82.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
```

### Versions

N/A

cc @zou3519 @Chillee @samdow @kshitij12345 @janeyx99
There is a performance drop because we have not yet implemented the batching rule for aten::mkldnn_rnn_layer_backward.
Labels: triaged, module: functorch

Hi all,

I just received the following warning, and I am doing what is suggested there. Many thanks!

```
/home/kshamsaei/miniconda3/envs/torch/lib/python3.11/site-packages/torch/autograd/__init__.py:303: UserWarning: There is a performance drop because we have not yet implemented the batching rule for aten::mkldnn_rnn_layer_backward. Please file us an issue on GitHub so that we can prioritize its implementation. (Triggered internally at /opt/conda/conda-bld/pytorch_1682343995622/work/aten/src/ATen/functorch/BatchedFallback.cpp:82.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
```

cc @zou3519 @Chillee @samdow @kshitij12345 @janeyx99
AOTAutograd perf: avoid as_strided() calls when we have intermediate bases
Labels: triaged, oncall: pt2, module: aotdispatch

This is a more targeted version of an existing issue around as_strided calls in AOTAutograd, https://github.com/pytorch/pytorch/issues/109237. Came from an internal issue

Simple repro:
```
import torch

@torch.compile
def f(x):
    out = x.mul(2)
    return out.view(out.shape), out.view(out.shape)

inps = (torch.randn(5), torch.tensor(0))
x = torch.randn(4, requires_grad=True)
out1, out2 = f(x)
print(out1.grad_fn)
```

prints:
```
<AsStridedBackward0 object at 0x7f4c7d4d2260>
```

We end up calling as_strided in the compiled forward, so an `AsStridedBackward` node shows up in the backward, which in general is not implemented to be particularly fast.

Why does this happen?

(1) AOTAutograd has logic for "intermediate bases". If we have two outputs of our graph that are aliases of each other (and of the same graph intermediate), today, AOTAutograd will just have the shared intermediate be an output to the graph. AOTAutograd will then replay the views off of the intermediate, so that autograd properly realizes that the outputs alias.

(2) AOTAutograd has a function to try to do the view replay, but it hits a slow path in that function that causes it to go to as_strided. We should figure out why and fix this: https://github.com/pytorch/pytorch/blob/main/torch/_functorch/aot_autograd.py#L807

cc @ezyang @msaroufim @wconstab @anijain2305
dump_operator_names.cc uses std::cout but dose not include iostream
Labels: triaged, better-engineering, actionable

### 🐛 Describe the bug

I am unable to build pytorch 2.1.0 on arch linux due to this line: https://github.com/pytorch/pytorch/blob/6c7013a3dce50315bb5d3adf33efce1490266c13/binaries/dump_operator_names.cc#L31

neither dump_operator_names.cc nor its intimidate includes include iostream.
Presumably on some platforms this works as iostream is included indirectly somewhere, this should not be relied on.

I also challenge the appropriateness of printing directly to stdout instead of to PyTorch's logging functions even in a function intended for debugging

### Versions

Pytorch 2.1.0
Export swallows exception
Labels: triaged, oncall: pt2, module: export

### 🐛 Describe the bug

torch export on except raises another exception without including original exception.
That  hides failure information.

code point:

https://github.com/pytorch/pytorch/blob/main/torch/_dynamo/utils.py#L1412



### Versions

master

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @avikchaudhuri @gmagogsfm @zhxchen17 @tugsbayasgalan
Option to disable fastpath in MHA
Labels: needs design, oncall: transformer/mha

### 🚀 The feature, motivation and pitch

I have trained a transformer model but I'm not able to run inference with it because of fastpath errors commented here: #88806 and #97128. I would like to run the inference with the normal path but I can't find any options to disable fastpath.

### Alternatives

One option is to add a flag like
```python
torch._fastpath.config.enable=False
```
or use environment variables.

### Additional context

I could disable it by setting the following line with a non-empty string: https://github.com/pytorch/pytorch/blob/d3205f83770f83cfa315a38f4448c64670758b03/torch/nn/modules/activation.py#L1111


cc @jbschlosser @bhosmer @cpuhrsch @erichan1 @drisspg @mikaylagawarecki
DISABLED test_unfuse_bias_addmm (__main__.TestPaternMatcher)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on MI210 runners as part of https://github.com/pytorch/pytorch/pull/105980:
https://github.com/pytorch/pytorch/actions/runs/5744430164/job/15572649474

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
DISABLED test_uint4x2_mixed_mm_gating_works (__main__.TestPaternMatcher)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on MI210 runners as part of https://github.com/pytorch/pytorch/pull/105980:
https://github.com/pytorch/pytorch/actions/runs/5744430164/job/15572649474

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
DISABLED test_uint4x2_mixed_mm_fail_to_match (__main__.TestPaternMatcher)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on MI210 runners as part of https://github.com/pytorch/pytorch/pull/105980:
https://github.com/pytorch/pytorch/actions/runs/5744430164/job/15572649474

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
DISABLED test_uint4x2_mixed_mm_epi (__main__.TestPaternMatcher)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on MI210 runners as part of https://github.com/pytorch/pytorch/pull/105980:
https://github.com/pytorch/pytorch/actions/runs/5744430164/job/15572649474

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
DISABLED test_uint4x2_mixed_mm (__main__.TestPaternMatcher)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on MI210 runners as part of https://github.com/pytorch/pytorch/pull/105980:
https://github.com/pytorch/pytorch/actions/runs/5744430164/job/15572649474

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
DISABLED test_splitwithsizes_cat (__main__.TestPaternMatcher)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on MI210 runners as part of https://github.com/pytorch/pytorch/pull/105980:
https://github.com/pytorch/pytorch/actions/runs/5744430164/job/15572649474

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
[dynamo] Allow autograd.Function tracing to smuggle attributes through lifting
Labels: ciflow/trunk, topic: not user facing, module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #111055

Today, we say an attribute crosses ctx boundaries when it is written to the ctx in Autograd.Function forward and accessed in backwards.

This mostly works. However, if that attribute has a proxy, (tensor, symint) - it will fail during attribute lifting as the proxy.tracer is from the forward speculation. We have prior art of bypassing lifting, as the proxy is already known the tracer, via the save_for_backwards calls in autograd.Function. 

This PR extends a similar hit through the proxy to let attribute lifting know that this proxy is already known to us and is safely tracked in side_effects. 

cc @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
DISABLED test_mm_plus_mm (__main__.TestPaternMatcher)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on MI210 runners as part of https://github.com/pytorch/pytorch/pull/105980:
https://github.com/pytorch/pytorch/actions/runs/5744430164/job/15572649474

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
DISABLED test_mixed_mm_gating (__main__.TestPaternMatcher)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on MI210 runners as part of https://github.com/pytorch/pytorch/pull/105980:
https://github.com/pytorch/pytorch/actions/runs/5744430164/job/15572649474

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
DISABLED test_mixed_mm_epi_works (__main__.TestPaternMatcher)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on MI210 runners as part of https://github.com/pytorch/pytorch/pull/105980:
https://github.com/pytorch/pytorch/actions/runs/5744430164/job/15572649474

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
DISABLED test_mixed_mm (__main__.TestPaternMatcher)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on MI210 runners as part of https://github.com/pytorch/pytorch/pull/105980:
https://github.com/pytorch/pytorch/actions/runs/5744430164/job/15572649474

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
DISABLED test__int_mm (__main__.TestSelectAlgorithm)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on MI210 runners as part of https://github.com/pytorch/pytorch/pull/105980:
https://github.com/pytorch/pytorch/actions/runs/5744430164/job/15572649474

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
[sparse] semi-structured sparse + torch.compile support
Labels: Merged, Reverted, ciflow/trunk, release notes: sparse, ciflow/slow

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #111049
* #110583

Summary:

This PR adds in torch.compile support for semi-structured sparsity,
using the subclass tracing @bdhirsh added.

Based on wether we are using cuSPARSELt or CUTLASS, we return a
different representation of the inner tensors.

Test Plan:
```
python test/test_sparse_semi_structured.py -k compile
```

Reviewers:

Subscribers:

Tasks:

Tags:
[WIP] Persist copy_ in training graph for inputs that don't require grad
Labels: module: inductor, module: dynamo, ciflow/inductor, release notes: AO frontend

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #111046

In this PR, we try to keep the input mutations in the forward graph IFF input mutation is data mutation and not metadata mutation and doesn't require grad. This is for optimizing inductor training graphs. (For more details: https://github.com/pytorch/pytorch/issues/109240) Previously, this was only enable for forward-only path but unconditionally disabled for joint graph. Another caveat is that when we are tracing through subclasses, we won't allow any input mutations to be preserved in the graph. The reason is that it makes the code quite ugly for no obvious performance improvement. 

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
[ROCm] Skip failing tests on ROCm
Labels: module: rocm, open source, topic: not user facing

Cherry picked https://github.com/ROCmSoftwarePlatform/pytorch/commit/2d9f3ec757d0c71d613c59e88c082c74a86f828a


cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
[ROCm] Centos stream9 pytorch image support
Labels: module: rocm, open source, topic: not user facing

This PR is by cherry picking the below two commits. 

Centos stream9 PyTorch image support - https://github.com/ROCmSoftwarePlatform/pytorch/commit/5ab8d09d51acd74ccd483e8d3b128f178dd4bca4
Updated to latest conda for CentOS stream 9 - https://github.com/ROCmSoftwarePlatform/pytorch/commit/2cb18f8d102b12b90a1635e78f8c3eaa7ccc8fcd

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
DISABLED test_multi_return_some_unused (__main__.TestSerialize)
Labels: module: rocm, triaged, module: flaky-tests, skipped, module: dynamo

Platforms: rocm

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/failure/test_multi_return_some_unused) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/17591397405).

Over the past 72 hours, it has flakily failed in 6 workflow(s).

**Debugging instructions (after clicking on the recent samples link):**
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Grep for `test_multi_return_some_unused`


Test file path: `export/test_serialize.py`

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
[dynamo] allow_in_graph decorator doesn't work on autograd.Function
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

sample:

```
import torch
from torch.autograd import Function
import torch._dynamo

@torch._dynamo.allow_in_graph
class Foo(Function):
    @staticmethod
    def forward(ctx, x):
        ctx.x0 = x.size(0)
        return x * 2

    @staticmethod
    def backward(ctx, grad_out):
        return grad_out * ctx.x0


@torch.compile(backend="eager", fullgraph=True, dynamic=True)
def foo(x):
    return Foo.apply(x)

foo(torch.randn(2, requires_grad=True))
```

you can see we're still trying to speculate subgraph (and failing due to https://github.com/pytorch/pytorch/issues/111031 ).

Workaround is to make a function that calls Foo.apply, and then allow in graph that.

### Versions

main

cc @msaroufim @wconstab @bdhirsh @anijain2305
[PT2.1] SIGSEGV seen with view + sgn operator inside torch.compile
Labels: triage review, triaged, ZeroTensor, oncall: pt2

### 🐛 Describe the bug

when view operator with sgn used inside torch.compile, then signal segmentation violation error show. 
Please use below code to reproduce the issue.

```
import torch

def fn(a):
    b = a.view((2, 2))
    return b.sgn()

x_cpu =torch.tensor([[2.0, 2], [-2, -2]], requires_grad=True)
compiled_fn = torch.compile(fn)        
y_cpu = compiled_fn(x_cpu)
print("y_hpu", y_cpu)
```

### Error logs

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffeb281440 in c10::intrusive_ptr<c10::TensorImpl, c10::UndefinedTensorImpl>::reset_() ()
   from /tmp/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
(gdb) bt
#0  0x00007fffeb281440 in c10::intrusive_ptr<c10::TensorImpl, c10::UndefinedTensorImpl>::reset_() ()
   from /tmp/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#1  0x00007fffeb2a9999 in at::FunctionalTensorWrapper::replace_(at::Tensor const&) () from /tmp/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#2  0x00007fffeb2aa48c in at::FunctionalTensorWrapper::regenerate_from_base() () from /tmp/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#3  0x00007ffff6710e3b in torch::autograd::THPVariable__sync(_object*, _object*, _object*) ()
   from /tmp/lib/python3.8/site-packages/torch/lib/libtorch_python.so
#4  0x00000000005f6939 in PyCFunction_Call ()
#5  0x00000000005f7506 in _PyObject_MakeTpCall ()
#6  0x0000000000570b8e in _PyEval_EvalFrameDefault ()
#7  0x00007ffff673afcb in custom_eval_frame_shim () from /tmp/lib/python3.8/site-packages/torch/lib/libtorch_python.so
#8  0x00000000005f6ce6 in _PyFunction_Vectorcall ()
#9  0x000000000056b4ed in _PyEval_EvalFrameDefault ()
#10 0x00007ffff673afcb in custom_eval_frame_shim () from /tmp/lib/python3.8/site-packages/torch/lib/libtorch_python.so
#11 0x00000000005697da in _PyEval_EvalCodeWithName ()
#12 0x00000000005f6ec3 in _PyFunction_Vectorcall ()
#13 0x000000000056b4ed in _PyEval_EvalFrameDefault ()
#14 0x00007ffff673afcb in custom_eval_frame_shim () from /tmp/lib/python3.8/site-packages/torch/lib/libtorch_python.so
#15 0x00000000005697da in _PyEval_EvalCodeWithName ()
#16 0x00000000005f6ec3 in _PyFunction_Vectorcall ()
#17 0x0000000000570556 in _PyEval_EvalFrameDefault ()
#18 0x00007ffff673afcb in custom_eval_frame_shim () from /tmp/lib/python3.8/site-packages/torch/lib/libtorch_python.so
#19 0x00000000005697da in _PyEval_EvalCodeWithName ()
#20 0x00000000005f6ec3 in _PyFunction_Vectorcall ()

### Minified repro

_No response_

### Versions

[pip3] numpy==1.24.4
[pip3] torch==2.1.0
[pip3] torchaudio==2.0.1
[pip3] torchdata==0.6.1
[pip3] torchmetrics==1.2.0
[pip3] torchtext==0.15.2a0
[pip3] torchvision==0.15.1a0

cc @ezyang @gchanan @zou3519 @kadeng @msaroufim @wconstab @bdhirsh @anijain2305
[Dynamo] Error in speculate_subgraph doesn't report inner user stack trace
Labels: triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

self explanatory

### Versions

main

cc @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
[Dynamo] Support more argument types for autograd Function speculate: HigherOrderOperator with body that accepts non-Tensors as input
Labels: triaged, module: dynamo

### 🐛 Describe the bug

If I pass a user defined object, or a list, etc to an autograd.Function, speculate subgraph should work as long as I can flatten all accessed elements into the supported types.

### Versions

main

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Disable FlashAttenion for is_causal=True when seqlen q not equal kv
Labels: ciflow/trunk, topic: not user facing, module: multi-headed-attention

# Summary:
This pull request **removes** support for non-square sequence lengths in causal attention when using FlashAttention V2.

### Why are doing this
  // FlashAttention 2 updated the default mask meaning for causal in this PR:
  // 9e5e8bc91e it is now aligned to lower_right which would be a BC break
  // for non-square masks. We will not support non-square masks for causal w/ FAV2
  
 For more context see:
 https://github.com/pytorch/pytorch/issues/108108
 
 ### Followup
 A large number of people will likely want to use FAV2 with lower_right causal attention for non equal sequence lengths. See this RFC : https://github.com/pytorch/pytorch/issues/110681
Dynamo inlining should compile partial subgraphs
Labels: triaged, module: dynamo

Consider a simple case of:

```
def h(x):
    x = x.cos()
    print(x)
    x = x.cos()
    return x

def g(x):
    x = x.cos()
    x = h(x)
    x = x.cos()
    return x


def f(x):
    x = x.cos()
    x = g(x)
    x = x.cos()
    return x
```

In an ideal state, dynamo would produce 2 graphs.

```
cos() 
cos()
cos()
```
`print` in eager
and
```
cos() 
cos()
cos()
```

However, today, we produce 6 due to how inlining runs (nesting inlining down) and how it fails (bubbling up). Each graph has a cos() in it.

This is pretty inefficient, and we lose a lot of compilation value from this - it is also probably representative of a real world problem where graph breaks in leaves have an outsized impact on the # of graphs we produce, and 1 graph break != 1 split in the graph. 



cc @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
DISABLED test_mem_leak (__main__.TestProfilerCUDA)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on MI210 runners as part of https://github.com/pytorch/pytorch/pull/105980:
https://github.com/pytorch/pytorch/actions/runs/5744430164/job/15572649474

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
DISABLED test_cublas_baddbmm_large_input_1_10000_10000_10000_cuda_float16 (__main__.TestMatmulCudaCUDA)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on MI210 runners as part of https://github.com/pytorch/pytorch/pull/105980:
https://github.com/pytorch/pytorch/actions/runs/5744430164/job/15572649474

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
DISABLED test_cublas_baddbmm_large_input_2_1000_1000_1000_cuda_float16 (__main__.TestMatmulCudaCUDA)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on MI210 runners as part of https://github.com/pytorch/pytorch/pull/105980:
https://github.com/pytorch/pytorch/actions/runs/5744430164/job/15572649474

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
DISABLED test_cublas_baddbmm_large_input_2_100_100_100_cuda_float16 (__main__.TestMatmulCudaCUDA)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on MI210 runners as part of https://github.com/pytorch/pytorch/pull/105980:
https://github.com/pytorch/pytorch/actions/runs/5744430164/job/15572649474

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
DISABLED test_jvp_linalg_det_singular_cpu_float32 (__main__.TestOperatorsCPU)
Labels: triaged, module: macos, module: regression, skipped, module: functorch

Platforms: macos

This test was disabled because it is failing on main branch ([recent examples](http://torch-ci.com/failure/functorch%2Ftest_ops.py%3A%3ATestOperatorsCPU%3A%3Atest_jvp_linalg_det_singular_cpu_float32)).

cc @albanD @zou3519 @Chillee @samdow @kshitij12345 @janeyx99
[Nested tensor]support nested tensor in hook
Labels: fb-exported, topic: new features, release notes: nested tensor

Summary:
## Motivation:
In hook, the check_variable_result check the grad size to make sure that the hook function doesn't change the grad size.
Since C++ Nested tensor doesn't support size, it got blocked when using hook.

## This diff:
Replace the sym_sizes call with is_same_size which supports nested tensor.
To use is_same_size, also change the signatures from TensorBase to Tensor.

Test Plan: add a test case that the hook changes the tensor size and it throws.

Differential Revision: D50106928
[dynamo] add infinite generators `itertools.{count, repeat, cycle}`
Labels: triaged, open source, module: dynamo, ciflow/inductor, release notes: dynamo

Fixes https://github.com/pytorch/pytorch/pull/110953/files#r1352868935

Depends on: https://github.com/pytorch/pytorch/pull/110953

Why not use these for `repeat(item, count)`:
> These are not preferred as they return an opaque VariableTracker. In particular, one cannot do `enumerate(repeat(1))`. `repeat(1, 10)` benefits from the integration enjoyed by `ListVariableIterator`

Follow ups:
- [ ] make listiterator an IteratorVariable, define iterator integrations on base IteratorVariable where unspecialized https://github.com/pytorch/pytorch/pull/110967#discussion_r1356656469
    - Please make a new issue for this 

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @ezyang
`torch.is_autocast_enabled()` always False on CPU
Labels: high priority, triage review, module: amp (automated mixed precision)

### 🐛 Describe the bug

`torch.is_autocast_enabled()` seems to always returns `False` on CPU. For example:

```python
import torch

device = torch.device("cpu")
with torch.autocast(device.type, torch.bfloat16, enabled=True):
    assert torch.is_autocast_enabled()
```

### Versions

Collecting environment information...
PyTorch version: 2.1.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.2.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.202)
CMake version: version 3.26.2
Libc version: N/A

Python version: 3.10.10 (main, Feb 16 2023, 02:46:59) [Clang 14.0.0 (clang-1400.0.29.202)] (64-bit runtime)
Python platform: macOS-13.2.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Max

Versions of relevant libraries:
[pip3] flake8==6.0.0
[pip3] mypy==1.2.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.22.4
[pip3] pytorch-ranger==0.1.1
[pip3] torch==2.1.0
[pip3] torch-optimizer==0.3.0
[pip3] torchaudio==2.1.0
[pip3] torchmetrics==0.11.4
[pip3] torchvision==0.16.0
[conda] Could not collect

cc @ezyang @gchanan @zou3519 @kadeng @mcarilli @ptrblck @leslie-fang-intel @jgong5
nccl flight recorder
Labels: release notes: distributed (c10d)

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110960

Keep a buffer of the last 16384 nccl work actions, including the stack
trace that launched the event.

When torch._C._distributed_c10d._dump_nccl_trace(), it an dump these to
a pickled archive.

For each action we get:
process_group_id, seq_id, collective_name, size_of_first_tensor, stack trace

state - issued, started, completed (based on cuda events and queried if
necessary when the dump is requested)

I tested that it is possible to query event state when the streams are
otherwise stuck.

Differential Revision: [D50138956](https://our.internmc.facebook.com/intern/diff/D50138956)
kBackendDefaultTimeout is causing a timeout exception when rank 0 process exceeds 30 minutes preparing a dataset.
Labels: oncall: distributed

### 🐛 Describe the bug

If rank 0 process takes more than 30 minutes to process a training dataset, processes of rank > 0 assume it is dead and throw an exception, therefore killing the distributed training program.

Attempts to change default_pg_timeout within deepspeed's code failed to work, implying that value is either ignored or being overridden by the value in  ~/pytorch/torch/csrc/distributed/c10d/Backend.hpp

____________________________________________________________________________________________________________

#pragma once

#include <condition_variable>
#include <memory>
#include <mutex>
#include <stdexcept>
#include <unordered_map>
#include <utility>
#include <vector>

#include <ATen/ATen.h>
#include <c10/macros/Macros.h>

#include <torch/csrc/distributed/c10d/Work.hpp>
#include <torch/csrc/distributed/c10d/Types.hpp>
#include <torch/csrc/distributed/c10d/Utils.hpp>
#include <torch/csrc/distributed/c10d/debug.h>
#include <torch/csrc/distributed/c10d/sequence_num.hpp>


constexpr auto kBackendDefaultTimeout = // This default of 30 minutes for a timeout is causing problems for long running 
                                                                      // training data prep with multiple GPU setups.
    std::chrono::milliseconds(30 * 60 * 1000);

namespace c10d {

class TORCH_API Backend : public torch::CustomClassHolder {
 public:

  // Backend Options is a base struct that defines the basic options
  // when constructing a Backend. Each Backend subclass should
  // extend this struct and define its options if it wants to provide more
  // config options (beyond basic ones defined here) to end user.
  struct TORCH_API Options : torch::CustomClassHolder {
    explicit Options(
        std::string backend,
        std::chrono::milliseconds timeout = kBackendDefaultTimeout)
        : timeout(timeout), backend(std::move(backend)) {}
    ~Options() override = default;


### Versions

.

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu
Enable more flake8-pyi ruff checks
Labels: good first issue, module: lint, triaged, actionable

### 🚀 The feature, motivation and pitch

https://github.com/pytorch/pytorch/pull/110830 enabled a bunch of use flake8-PYI checks to improve the quality of type hints on the PyTorch codebase. While this was a good first pass, many rules need to removed from the ignore list in the pyproject.toml. Fixing these rules an enabling the lints would be a good starter task.

For other rules to enable, see this starter task: https://github.com/pytorch/pytorch/issues/106571

### Alternatives

_No response_

### Additional context

_No response_
RuntimeError: out_ptr == out_accessor[thread_count_nonzero[tid + 1]].data() INTERNAL ASSERT FAILED
Labels: needs reproduction, triaged, module: advanced indexing

### 🐛 Describe the bug

Running a python script from within a SLURM job. Running the same SLURM job multiple times will have inconsistent results with jobs sometimes completing and sometimes failing with this error message.

### Error logs

```
Traceback (most recent call last):
  File "/<myprojectpath>/src/SLURM/pythonScript.py", line 45, in <module>
    dot_detect_deconv(fl, spot_dir, support_dir, th_fit = 80, delta=2, delta_fit=4, sigmaZ=2.5, sigmaXY=1.85)
  File "/<myprojectpath>/src/SLURM/pythonScript.py", line 33, in dot_detect_deconv
    Xh = get_local_maxfast_tensor(imDn,th_fit=th_fit,im_raw=im_,dic_psf=None,delta=delta,delta_fit=delta_fit,sigmaZ=sigmaZ,sigmaXY=sigmaXY)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<myprojectpath>/src/SLURM/../ioMicro_Schier_v5.py", line 2768, in get_local_maxfast_tensor
    z_ = get_ind(im_centers0,zmax)
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<myprojectpath>/src/SLURM/../ioMicro_Schier_v5.py", line 2741, in get_ind
    x_[bad]=-x_[bad]
             ~~^^^^^
RuntimeError: out_ptr == out_accessor[thread_count_nonzero[tid + 1]].data() INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1682343970094/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp":2232, please report a bug to PyTorch. 
```

### Minified repro

_No response_

### Versions

```
Collecting environment information...
Traceback (most recent call last):
  File "collect_env.py", line 612, in <module>
    main()
  File "collect_env.py", line 595, in main
    output = get_pretty_env_info()
  File "collect_env.py", line 590, in get_pretty_env_info
    return pretty_str(get_env_info())
  File "collect_env.py", line 428, in get_env_info
    pip_version, pip_list_output = get_pip_packages(run_lambda)
  File "collect_env.py", line 400, in get_pip_packages
    out = run_with_pip([sys.executable, '-mpip'])
  File "collect_env.py", line 384, in run_with_pip
    for line in out.splitlines()
AttributeError: 'NoneType' object has no attribute 'splitlines'
```
sorry I guess this is not what you were expecting... 

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
remove \ in cache_dir
Labels: triaged, open source, ciflow/trunk, topic: not user facing, module: inductor, ciflow/inductor

Fixes #110933 

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
gdb core dump when enable DEBUG mode to compile cpu torch in centos!!!
Labels: needs reproduction, triaged

### 🐛 Describe the bug

try to compile pytorch 2.0 from source with DEBUG mode in centos, 
```
DEBUG=ON USE_CUDA=OFF python3 setup.py build
python3 setup.py bdist_wheel -d build_pip
pip3 install build_pip/torch-2.0.0a0+gitf33faa7-cp38-cp38-linux_x86_64.whl
```

write a test_import.py only to execute "import torch",

test_import.py:
`
import torch
`
execute on gdb terminal,
```
$ gdb python3
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/bin/python3.8...done.
(gdb) r test_import.py
Starting program: /usr/local/bin/python3 test_import.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[Detaching after fork from child process 59264]
Segmentation fault (core dumped)
```
the only info of gdb core dump file is:
```
$ gdb core-gdb.36373.software-h3c-r5300-g5-012.1696993149
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
[New LWP 36373]
Missing separate debuginfo for the main executable file
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/c1/2d5af5d51fcc6bf42dbae84c5c14631fc28025
Core was generated by `gdb python3'.
Program terminated with signal 11, Segmentation fault.
#0  0x000000000079c820 in ?? ()
"./core-gdb.36373.software-h3c-r5300-g5-012.1696993149" is a core file.
Please specify an executable to debug.
(gdb) bt
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fffff7fef20:
(gdb)

```
Noted:
if compile on ubuntu image, it is ok;

### Versions

pytorch v2.1.0 tag
docker: nvidia/cuda:11.2.0-cudnn8-devel-centos7
Noisy warnings from broadcasting in boolean operations
Labels: high priority, triage review, module: regression, module: safe resize

### 🐛 Describe the bug

The following code:
```python
import torch

I,J,K = 5,3,7

x1 = torch.zeros(I,J,1).bool()
x2 = torch.zeros(I,1,K).bool()
torch.logical_and(x1, x2)
```
raises 
```bash
UserWarning: An output with one or more elements was resized since it had shape [5, 3, 1], which does not match the required output shape [5, 3, 7]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:28.)
  torch.logical_and(x1, x2)
```

However the equivalent

```python
import torch

I,J,K = 5,3,7

x1 = torch.zeros(I,J,1).bool()
x2 = torch.zeros(I,1,K).bool()
x1*x2
```

Does not raise any warnings. This seems inconsistent.

There seems to have been some reason (bugfix) for raising this warning, see #107431 .



### Versions

```bash
[pip3] numpy==1.26.0
[pip3] torch==2.1.0
[pip3] torch-tb-profiler==0.4.1
[pip3] triton==2.1.0
```

cc @ezyang @gchanan @zou3519 @kadeng