gopikrsmscs/torch-issues huggingface dataset:
[dynamo] scalar <comp op> tensor not supported
Labels: triaged, oncall: pt2, module: dynamo

Min repro code by @lezcano :

```python
import torch

@torch.compile(fullgraph=True)
def fn(x):
    return 0 <= x

x = torch.randn(512, device="cuda")
fn(x)
```

Here are tracing logs:
```
Step 1: torchdynamo start tracing fn check_compile_le_torch_op.py:4
TRACE starts_line fn check_compile_le_torch_op.py:4
    @torch.compile()
wrap_to_fake L['x'] (512,) [<DimDynamic.STATIC: 2>] [None]
TRACE starts_line fn check_compile_le_torch_op.py:6
        return 0 <= x
TRACE LOAD_CONST 0 []
TRACE LOAD_FAST x [ConstantVariable(int)]
TRACE COMPARE_OP <= [ConstantVariable(int), TensorVariable()]
step triggered compile
Traceback (most recent call last):
  File "/pytorch/torch/_dynamo/symbolic_convert.py", line 691, in step
    getattr(self, inst.opname)(inst)
  File "/pytorch/torch/_dynamo/symbolic_convert.py", line 1107, in COMPARE_OP
    BuiltinVariable(supported_any[op], **options).call_function(
  File "/pytorch/torch/_dynamo/variables/builtin.py", line 618, in call_function
    result = handler(tx, *args, **kwargs)
  File "/pytorch/torch/_dynamo/variables/builtin.py", line 1412, in _comparison
    _unimplemented()
  File "/pytorch/torch/_dynamo/variables/builtin.py", line 1321, in _unimplemented
    unimplemented(f"comparison {typestr(left)} {op} {typestr(right)}")
  File "/pytorch/torch/_dynamo/exc.py", line 176, in unimplemented
    raise Unsupported(msg)
torch._dynamo.exc.Unsupported: comparison ConstantVariable(int) <built-in function le> TensorVariable()
restore_graphstate: removed 0 nodes
COMPILING GRAPH due to GraphCompileReason(reason='step_unsupported', user_stack=[<FrameSummary file check_compile_le_torch_op.py, line 6 in fn>], graph_break=True)
GUARDS:
hasattr(L['x'], '_dynamo_dynamic_indices') == False           # _dynamo/variables/builder.py:1252 in wrap_fx_proxy_cls
___is_grad_enabled()                                          # _dynamo/output_graph.py:345 in init_ambient_guards
not ___are_deterministic_algorithms_enabled()                 # _dynamo/output_graph.py:341 in init_ambient_guards
___is_torch_function_enabled()                                # _dynamo/output_graph.py:349 in init_ambient_guards
utils_device.CURRENT_DEVICE == None                           # _dynamo/output_graph.py:347 in init_ambient_guards
check_tensor(L['x'], Tensor, DispatchKeySet(CUDA, BackendSelect, ADInplaceOrView, AutogradCUDA), torch.float32, device=0, requires_grad=False, size=[512], stride=[1])  # _dynamo/variables/builder.py:1252 in wrap_fx_proxy_cls
```

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @ysiraichi
subclasses <> compile <> dynamic shapes: assume only first inner tensor gets dynamic dims
Labels: module: dynamo, ciflow/inductor

This came from some initial exploration of trying to compile @vkuzo 's `Float8Tensor` subclass through AOTAutograd. One issue I ran into is that:

(a) The current dynamic shapes logic assumes that the dynamic dims inferred from dynamo onto the outer wrapper tensor should be applied to every inner tensor in a wrapper tensor subclass

(b) this isn't actually true in many cases: for `Float8Tensor`, we have a subclass with two inner tensors: the first should get dynamic shapes, but the second is an `amax` scalar-tensor that is always a zero-dim scalar tensor, and doesn't need/want dynamic shapes.

I added a larger comment in the code, but I think that "really" fixing this problem will require some API design, since `mark_dynamic()` not longer carries enough info to specific what you want to happen for subclasses. Instead of figuring out the right thing to do here (which feels risky because we don't really know all of the ways that people might (ab)use subclasses in the longer term), I'm just doing the simple thing: we have no use cases today where a wrapper subclasses has **multiple** inner tensors, **all** of which need dynamic shapes. So I updated the fakeifying logic to always assume that the first inner tensor attr returned from `__tensor_flatten__` gets dynamic shapes, and no other tensors after the first do.

cc @ezyang, lmk if this sounds reasonable to you for now.

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108570
* #108243
* #108235
* #108081
Call for a deterministic implementation of scatter_add_cuda_kernel
Labels: triaged, module: scatter & gather ops

### 🚀 The feature, motivation and pitch

I met an error below: 

-----------------------------------------------------------------------------

......
message_agg = scatter(message, index=obj, dim=0, dim_size=n_node, reduce='sum')
  File "/home/gg/anaconda3/envs/KGAT-Pytorch/lib/python3.6/site-packages/torch_scatter/scatter.py", line 155, in scatter
    return scatter_sum(src, index, dim, out, dim_size)
  File "/home/gg/anaconda3/envs/KGAT-Pytorch/lib/python3.6/site-packages/torch_scatter/scatter.py", line 21, in scatter_sum
    return out.scatter_add_(dim, index, src)
RuntimeError: scatter_add_cuda_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.


### Alternatives

_No response_

### Additional context

_No response_

cc @mikaylagawarecki
Allow slicing of Nested Tensors along constant dimensions
Labels: triaged, module: nestedtensor

### 🚀 The feature, motivation and pitch

In many cases several of the nested tensor dimensions are constant, and therefore in principle should allow slicing

```
import torch
a = torch.rand(5,5,5)
b = torch.rand(5,5,10)
c =  torch.rand(5,5,15)
d= torch.nested.nested_tensor([a,b,c])
d[:2]
```

Will currently return the following

`NotImplementedError: Could not run 'aten::slice.Tensor' with arguments from the 'NestedTensorCPU' backend. `

May be useful to be able to mark which dims are and will remain constant? 

### Alternatives

_No response_

### Additional context

_No response_

cc @cpuhrsch @jbschlosser @bhosmer @drisspg
`bytes(...)` support of torch tensor does not match numpy + it would be nice to support tensor.tobytes() as alias
Labels: feature, triaged, module: numpy

### 🐛 Describe the bug

```python
import numpy as np
import torch



a = np.arange(3)
print(bytes(a))
# b'\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00'
# ^ correct ^

print(bytes(a.view(np.uint8)))
# b'\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00'
# ^ correct ^

print(a.view('uint8'))
# array([0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0], dtype=uint8)

print(a.tobytes())
# b'\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00'
# ^ correct ^




b = torch.arange(3)
print(bytes(b))
# b'\x00\x01\x02'
# ^ incorrect ^

print(bytes(b.view(torch.uint8)))
# b'\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00'
# ^ correct ^

print(b.view('uint8'))
# TypeError: view() received an invalid combination of arguments - got (str), but expected one of:
# * (torch.dtype dtype)
#      didn't match because some of the arguments have invalid types: (str)
# * (tuple of ints size)
#      didn't match because some of the arguments have invalid types: (str)

print(b.tobytes())
# AttributeError: 'Tensor' object has no attribute 'tobytes'
``` 

Also, if PyTorch supported string representations of dtype as NumPy does, we could write more polymorphic code: `bytes(tensor.view('uint8'))`, currently we need to have to use backend specific `np.uint8` or `torch.uint8`

Maybe related issues:
- https://github.com/pytorch/pytorch/issues/33041
- https://github.com/pytorch/pytorch/issues/43949

### Versions
```python
np.__version__
# '1.24.2'
torch.__version__
# '2.1.0.dev20230802+cpu'
```

cc @mruberry @rgommers
[Decomposition] unbind
Labels: fb-exported, module: inductor, ciflow/inductor

Summary: Copy decomp from caffe2/torch/_refs/__init__.py and include it in core_aten_decompositions

Test Plan: OSS + Phabricator Tests

Differential Revision: D48871742




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[Decomposition] uniform_
Labels: fb-exported, ciflow/inductor

Summary:
Include decomp in core_aten_decompositions

Decomp already exists

https://www.internalfb.com/code/fbsource/[03ff511cad587fc27ed8fd6a54b87845246e8e0c]/xplat/caffe2/torch/_decomp/decompositions.py?lines=2178

Test Plan: OSS + Phabricator Tests

Differential Revision: D48940435
[Decomposition] split.Tensor
Labels: fb-exported, ciflow/inductor, module: export

Summary:
Include decomp in core_aten_decompositions

Decomp: https://github.com/pytorch/pytorch/blob/1e9b590df989337d809fa33edd7ffc6cb60e70ff/torch/_decomp/decompositions.py#L1198

Test Plan: OSS + Phabricator Tests

Differential Revision: D48871743
[Decomposition] resize
Labels: fb-exported, module: inductor, ciflow/inductor

Summary: Add decomp and include it in core_aten_decompositions

Test Plan: Phabricator + OSS Tests

Differential Revision: D48940336




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[Decomposition] randn_like
Labels: fb-exported, module: inductor, ciflow/inductor

Summary: Moving decomposition from _inductor/decomposition.py and include it in core_aten_decompositions

Test Plan: Phabricator + OSS Tests

Differential Revision: D48940304




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[Decomposition] randint
Labels: fb-exported, module: inductor, ciflow/inductor

Summary: Moving decomposition from _inductor/decomposition.py and include it in core_aten_decompositions

Test Plan: Phabricator + OSS Tests

Differential Revision: D48940203




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[Decomposition] full_like
Labels: fb-exported, module: inductor, ciflow/inductor

Summary: Moving decomposition from _inductor/decomposition.py and include it in core_aten_decompositions

Test Plan: Phabricator + OSS Tests

Differential Revision: D48939842




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[Decomposition] exponential_
Labels: fb-exported, ciflow/inductor

Summary:
Include decomp in core_aten_decompositions

Decomp already exists:
https://github.com/pytorch/pytorch/blob/ff38c0e2f9cae35378553c38ccf7188007fed938/torch/_decomp/decompositions_for_rng.py#L229

Test Plan: Phabricator + OSS Tests

Differential Revision: D48939790
[Decomposition] bernoulli
Labels: fb-exported, module: inductor, ciflow/inductor

Summary: Moving decomposition of bernoulli from _inductor/decomposition.py and include it in core_aten_decompositions

Test Plan: Phabricator + OSS Tests

Differential Revision: D48878434




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
Breaking incompatibility with Cuda 12.2, pytorch stable, torchvision
Labels: oncall: binaries, module: crash, module: cuda, triaged

### 🐛 Describe the bug

With latest Cuda 12.2 and ubuntu 20.04 driver, pytorch stable and torchvision installed as on web site, there is a weird segfault occurring whenever a model runs on Cuda.
When it crashes with a message, it's always about python multiprocessing code with buffers too big. Sorry I didn't capture as it seems random with most of the time a segfault without a message. Reinstall from scratch didn't solve the issue. 

Note that DDP works but segfault happens with simple single threaded inference. 

Solution is to use pytorch-nightly and to force it to install gpu version for pytorch and torchvisio, as by default it installs cpu version! This works very well and even faster. We can't wait to see this nightly becomes the new stable. 

Another solution could be to downgrade as we didn't see issue with Cuda 12.1 and pytorch stable for the last 6 months. Maybe that's what we'd be forced to do if nightly breaks... 

### Versions

Ubuntu 20.04
Python 3.11 or 3.10 in conda environment 
Cuda 12.2, cudnn 8 for Cuda 12, on A100 server. 
Pytorch stable and torchvision latest in a fresh conda environment. 


cc @seemethere @malfet @ptrblck
nn.Transformer has dropout layers that BERT / GPT-2 do not have
Labels: module: docs, triaged, oncall: transformer/mha

### 📚 The doc issue

The [docstring of nn.TransformerEncoder](https://github.com/pytorch/pytorch/blob/51c2e22/torch/nn/modules/transformer.py#L233) reads: "Users can build the BERT(https://arxiv.org/abs/1810.04805) model with corresponding parameters."
However, `TransformerEncoderLayer` (and `TransformerDecoderLayer`) has a dropout layer in between the two linear layers that come after the attention layers:
* https://github.com/pytorch/pytorch/blob/51c2e22/torch/nn/modules/transformer.py#L723
* https://github.com/pytorch/pytorch/blob/51c2e22/torch/nn/modules/transformer.py#L874

BERT does not have this:
* https://github.com/google-research/bert/blob/master/modeling.py#L872
So the docstring is subtly wrong, at least when planning to use the model for training.

For more context, also GPT-2 does not have this:
* https://github.com/openai/gpt-2/blob/master/src/model.py#L118

The original "Attention is all you need" has it in Transformer Base v2 and v3, but not v1:
* https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/transformer.py#L1850

The official TensorFlow tutorial does not have it:
* https://www.tensorflow.org/text/tutorials/transformer#the_feed_forward_network

But the Annotated Transformer has it:
* http://nlp.seas.harvard.edu/annotated-transformer/

An alternative to changing the docstring would be extending the `TransformerEncoderLayer` implementation to optionally accept a dictionary for `dropout` with entries "residual", "mlp", "attention" for the three types of dropout that are employed in the model. But I don't know how much `nn.Transformer` is used as compared to custom implementations.

### Suggest a potential alternative/fix

A correct version would be:
"Users can build the BERT (...) model with corresponding parameters, except that BERT does not employ dropout between the feed-forward layers."
This would draw some attention to the fact that the implementation is slightly different from other transformers. Ideally, there would be a similar note for Transformer, TransformerDecoder, TransformerEncoderLayer, TransformerDecoderLayer.

cc @svekars @carljparker @jbschlosser @bhosmer @cpuhrsch @erichan1 @drisspg
[inductor] CPU int32 overflow behavior differs between clang and gcc
Labels: triaged, bug, oncall: pt2, module: m1, module: inductor, module: cpu inductor

### 🐛 Describe the bug

As a follow on from #108513 the following add operation gives a different output on torch and dynamo on CPU:

```
import torch
import torch._dynamo
import torch._dynamo.config
def mySum64(x):
    return (x+x).to(torch.int64)

x = torch.tensor( (2147483647), dtype=torch.int32)
torchResult = mySum64(x)
dynamoResult = torch.compile(mySum64)(x)

print(torchResult)
print(dynamoResult)
```

The output is :
```
tensor(-2)
tensor(4294967294)
```  
It looks like the compiler is not honouring the int32 for the add; instead it is incorrectly treating it as an int64. 

When we fix this a test should be added for this situation in the same place as #108513.


### Versions

Collecting environment information...
PyTorch version: 2.1.0a0+gite68b3ad
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.5.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: version 3.26.4
Libc version: N/A

Python version: 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:41:52) [Clang 15.0.7 ] (64-bit runtime)
Python platform: macOS-13.5.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Max

Versions of relevant libraries:
[pip3] flake8==6.0.0
[pip3] flake8-bugbear==23.3.23
[pip3] flake8-comprehensions==3.12.0
[pip3] flake8-executable==2.1.3
[pip3] flake8-logging-format==0.9.0
[pip3] flake8-pyi==23.3.1
[pip3] flake8-simplify==0.19.3
[pip3] mypy==0.960
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.25.2
[pip3] torch==2.1.0a0+git1004706
[pip3] torchgen==0.0.1
[conda] numpy                     1.25.2                   pypi_0    pypi
[conda] torch                     2.1.0a0+git1004706           dev_0    <develop>
[conda] torchgen                  0.0.1                    pypi_0    pypi

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
Multi-Head Attention: Only require attn_mask if actually needed
Labels: triaged, open source

`torch.nn.MultiheadAttention` internally uses `scaled_dot_product_attention()`. For the common case of causal attention, the latter accepts an `is_causal` flag, which computes causal attention inside the kernel without having to compute an attention mask in memory. Still, `torch.nn.MultiheadAttention` and `torch.nn.functional.multi_head_attention_forward()` ask for an attention mask whenever `is_causal=True` is given:
```python
import torch
mha = torch.nn.MultiheadAttention(100, 4)
x = torch.random(2, 10, 100)
mha(x, x, x, is_causal=True, need_weights=False)  # RuntimeError
```
This short PR tightens the check for a missing attention mask so it is not required when it would be set to `None` 11 lines later anyway.
Disclaimer: I currently do not have a development setup for PyTorch and will rely on the CI, sorry.

As an aside, the docstring of `torch.nn.functional.multi_head_attention_forward()` currently reads:
> is_causal: If specified, applies a causal mask as attention mask, and ignores
>        attn_mask for computing scaled dot product attention.

This suggests that the mask is completely ignored, while it is actually still required when either `need_weights` or `key_padding_mask` is given. This could be fixed either by updating the docstring, or by creating a causal `attn_mask` on the fly when needed, and not ever complaining about a missing mask. The latter would be convenient, but it would hide to the user the opportunity to precompute the mask once and reuse it in the case of fixed sequence lengths or multiple same-size transformer layers.
S390x complex division
Labels: module: cpu, triaged, open source

Adopt algorithm from AVX2 implementation.
This change fixes test test_complex_div_underflow_overflow_cpu_complex128
from test/test_binary_ufuncs.py

At the same time it breaks some of Arithmetics/*.Division tests
from vec_test_all_types_ZVECTOR,
but it's also broken on AVX2 and AVX512.

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
Eliminate calls of c10::guts::conjunction,c10::guts::disjunction,c10::guts::negation,c10::guts::void_t, c10::invoke and c10::guts::apply
Labels: module: cpu, triaged, open source, ciflow/binaries, ciflow/trunk, release notes: distributed (c10d), ciflow/periodic, ciflow/slow

C++17 provides alternations. 


cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
@huydhn  @Skylion007
[Dynamo][Guard]expose guard code
Labels: triaged, open source, module: dynamo, ciflow/inductor

This is part of ongoing efforts to expose more information to users so that they can check and understand how dynamo works.

This PR exposes the source code of guards to users. Together with the `_debug_get_cache_entry_list` API, users can inspect a compiled function he wants, and look into its guards.

Another proposal is to wire things up so that `inspect.getsource` works for guards. However, `inspect.getsource` works by reading the source code **file**, which does not work for dynamically generated function.

cc @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @jansel
[complex][cpu] nansum & nanmean
Labels: module: cpu, open source, topic: not user facing

Fixes #71472

This PR adds complex support for nansum and nanmean in the CPU.
Previous PR (in CUDA): #93199

cc @Skylion007 @kshitij12345 @lezcano 

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
[cond] cache size limit exceeeded
Labels: triaged, oncall: pt2, module: higher order operators

### 🐛 Describe the bug

Repro

1) Get this stack of branch - https://github.com/pytorch/pytorch/pull/108402
2) Run ` test/functorch/test_control_flow.py::TestControlFlowTraced::test_nested_map_cond_real` or `pytest test/functorch/test_control_flow.py::TestControlFlowTraced::test_cond_nested_with_closure`

Background - I am trying to reduce the cache size limit to 4. Its unclear to me why torch.cond requires even more than one compile. I see many recompiles. Its also counter intuitive because torch.cond is targeted for torch.export.

cc @ezyang @msaroufim @wconstab @bdhirsh @ydwu4 @zou3519 




### Error logs

_No response_

### Minified repro

_No response_

### Versions

N/A
backend-friendly distributions
Labels: module: distributions, feature, module: cuda, triaged

### 🚀 The feature, motivation and pitch

As of now many basic distributions are not supported on important backends, for instance for a categorical or multinomial distribution on inductor/cuda graphs
```python
@torch.compile(fullgraph=True, backend='inductor')
def fn():
    cat = torch.distributions.categorical.Categorical(torch.tensor([0.1,0.2,0.7])) # seems to use torch.multinomial under the hood
    return cat.sample()

fn()
```
gives
```json
{
	"name": "Unsupported",
	"message": "call_method SizeVariable() numel [] {}\n\nfrom user code:\n   File \"/tmp/ipykernel_1333478/2921103239.py\", line 6, in fn\n    return cat.sample()\n  File \"/usr/local/lib/python3.8/dist-packages/torch/distributions/categorical.py\", line 118, in sample\n    samples_2d = torch.multinomial(probs_2d, sample_shape.numel(), True).T\n\nSet torch._dynamo.config.verbose=True or TORCHDYNAMO_VERBOSE=1 for more information\n\n\nYou can suppress this exception and fall back to eager by setting:\n    torch._dynamo.config.suppress_errors = True\n",
}
```

### Alternatives

It it possible to have it implemented in a way compatible with GPU-friendly backends?

### Additional context

_No response_

cc @fritzo @neerajprad @alicanb @nikitaved @ptrblck
Suppport Fused AdamW on CPU
Labels: module: performance, feature, module: optimizer, triaged, needs research

### 🚀 The feature, motivation and pitch

I would like to benefit from the speed advantages of fused-adamw while doing CPU only training, but this is not supported. It currently throws an error indicating that it only works with GPUs.

While CPU only training is not a priority it seems, supporting it well makes it possible for downstream developers to for example run CI/CD tests on CPU only instances.

### Alternatives

Not using fused-adamw.

### Additional context

_No response_

cc @vincentqb @jbschlosser @albanD @janeyx99 @crcrpar
DistributedDataParallel to support __getattr__
Labels: oncall: distributed, triaged

### 🚀 The feature, motivation and pitch

The DistributedDataParallel wrapped module, beside being a subclass of nn.Module, can have its own additional methods for different situations. Now, under DDP, in order to reference those methods, it'll be quite cubersome to do the following everywhere that we want to reference those methods

```
if is_ddp():
    mymodel.module.method1()
else:
    mymodel.method1()
```

or 

```
if is_ddp():
    my_wrapped_module.method1()
else:
    mymodel.method1()
```


Now if we add `__getattr__` support to the DistributedDataParallel class and have it delegate to the wrapped module for the call, we won't need to either keep track of the wrapped module or reference the wrapped module for the method call needed.

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu
Don't call release_lock_on_cudamalloc on ROCm
Labels: module: rocm, ciflow/trunk, topic: not user facing, test-config/default

This change https://github.com/pytorch/pytorch/pull/108367 breaks ROCm tests in trunk.  This was landed internally, so it might be easier to forward fix the test instead.

The failure https://hud.pytorch.org/pytorch/pytorch/commit/b8af8ac784905ff4d20792959e3920d01acfa8cf is that the new `release_lock_on_cudamalloc` function is not available on ROCm.

### Testing

The test passes on ROCm https://github.com/pytorch/pytorch/actions/runs/6057040907/job/16437921431

Fixes #108476

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
CNN w variable sized input performance regression 1.10.2 cu113 -> 2.0.1 cu117
Labels: module: performance, module: nn, module: cuda, triaged, module: regression

### 🐛 Describe the bug

I'm running an image processing service that takes images of varying sizes and runs them through a couple of CNNs, one with fixed size input (always w0 * h0) and one with variable size input (w & h vary from input to input). 

I've been trying to upgrade from v1.10.2 cu113 to v2.0.1 cu117 and encountered https://github.com/bytedeco/javacpp-presets/issues/1409 / https://github.com/pytorch/pytorch/pull/104369. When I apply the workaround (`TORCH_CUDNN_V8_API_DISABLED=1`) and run a load test I see the following performance regression: 

![image](https://github.com/pytorch/pytorch/assets/90846/f4a2eefe-ed81-4a7e-a5c3-278e473af6ca)

In addition, the first couple of runs through my networks can easily take ~30 seconds on v2.0.1 when they take <1 second on v1.10.2. 

This performance regression makes it hard to upgrade and I'm not really sure how to proceed here - I have an entire production edifice where it exhibits, but it's all proprietary so hard to share in a way that could enable debugging. 

I guess I'd need to find a way to replicate it in a smaller python-only example? 

### Versions

The issue exhibits in v2.0.1 cu117 but not in v1.10.2 cu113

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
[unwind.cpp] In process unwind symbolization

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108469

(WIP to debug build issues)
Replaces the Symbolizer which calls `addr2line` with one that uses
libbfd in process to do the same work. This is a little nicer to deal with
because it doesn't have to fork, which can be a problem for large applications.
It also doesn't keep subprocesses around that might not clean up nicely.

libbfd is the same library addr2line uses, so the results should be equivalent
and take about the same time to generate.

This uses `std::async` to launch threads because libbfd is pretty slow,
and doing so matches the performance of the previous version which launched
processes instead of threads.

Differential Revision: [D48980893](https://our.internmc.facebook.com/intern/diff/D48980893)
[WIP] lazy list length guarding
Labels: module: dynamo, ciflow/inductor

Fixes #ISSUE_NUMBER


cc @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov
_foreach_copy_ with scalar second arg
Labels: feature, module: optimizer, triaged, actionable, module: mta

### 🚀 The feature, motivation and pitch

There are a lot of uses (specifically in the optimizer) for launching a single kernel to fill multiple tensors with a scalar. Other foreach ops can handle scalar args and broadcasting as well. 

### Alternatives

_No response_

### Additional context

_No response_

cc @vincentqb @jbschlosser @albanD @janeyx99 @crcrpar @mcarilli
Revert D48801487: Multisect successfully blamed "D48801487: [export] Copy gm before calling PassManager" for test or build failures
Labels: fb-exported

Summary:
This diff is reverting D48801487
D48801487: [export] Copy gm before calling PassManager by digantdesai has been identified to be causing the following test or build failures:

Tests affected:
- [on_device_ai/Assistant/Jarvis/compiler:test_passes - test_calculate_peak_memory_pass (on_device_ai.Assistant.Jarvis.compiler.tests.test_passes.TestMemPlanningPasses)](https://www.internalfb.com/intern/test/844425014917237/)

Here's the Multisect link:
https://www.internalfb.com/multisect/2930767
Here are the tasks that are relevant to this breakage:

We're generating a revert to back out the changes in this diff, please note the backout may land if someone accepts it.

If you believe this diff has been generated in error you may Commandeer and Abandon it.

Test Plan: NA

Differential Revision: D48901081
[dynamo] Add DictView variable tracker
Labels: open source, module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108420
* #110524
* #111196
* #110523
* #110522

As per title

cc @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov
torch.einsum() computes different results on cpu and cuda on A100 GPU.
Labels: module: cuda, triaged, module: linear algebra

### 🐛 Describe the bug

**Dear Developers:**

Currently, `torch.einsum()` is commonly used to compute the attention scores of query and key matrix. However, we found this operator computes different results on CPU and CUDA on a **A100 GPU**. And V100 doesn't have this problem.

Below is the reproduced code, where `n` is the sequence length, `h` is the number of heads, `d` is the hidden_size per head. We fixed `h, d = 64, 96`, and tested the results of different `n`.

```python
import torch

for n in [1, 10, 50, 64, 100, 200, 500]:
    h, d = 64, 96

    qq = torch.ones((1, n, h, d), dtype=torch.float32)  # use ones tensor
    kk = torch.ones((1, n, h, d), dtype=torch.float32)
    vv = torch.ones((1, n, h, d), dtype=torch.float32)

    kk /= torch.sqrt(torch.tensor(d))

    scores1 = torch.einsum(
        "bthd,bshd->bhts", qq.cuda(), kk.cuda()
    ).cpu()  # cuda
    scores2 = torch.einsum("bthd,bshd->bhts", qq, kk)  # cpu
    print(n, (scores1 - scores2).abs().max())   # compute the max different abs value.
```

A100 results, when `n>=64`, this problem arises.
```bash
1 tensor(7.6294e-06)
10 tensor(0.)
50 tensor(0.)
64 tensor(0.0011)
100 tensor(0.0011)
200 tensor(0.0011)
500 tensor(0.0011)
```

V100 results, almost no error.
```bash
1 tensor(7.6294e-06)
10 tensor(7.6294e-06)
50 tensor(7.6294e-06)
64 tensor(0.)
100 tensor(7.6294e-06)
200 tensor(7.6294e-06)
500 tensor(7.6294e-06)
```

### Versions

Python Envs

```
Versions of relevant libraries:
[pip3] numpy==1.22.2
[pip3] pytorch-quantization==2.1.2
[pip3] pytorch-triton==2.1.0+e6216047b8
[pip3] torch==2.1.0.dev20230812+cu121
[pip3] torch-tensorrt==1.5.0.dev0
[pip3] torchdata==0.7.0a0
[pip3] torchtext==0.16.0a0
[pip3] torchvision==0.16.0a0
[pip3] triton==2.0.0
[conda] numpy                     1.24.4                   pypi_0    pypi
```

cc @ptrblck @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano
enhance documentation around the developer build
Labels: triaged, topic: docs

### 📚 The doc issue

At the following or similar to the source build docs
```
To resolve the issue of your local folder not being picked up first in the Python path when building PyTorch, you can try the following steps:

Remove the previously installed package:
Since you have already run python setup.py develop, you need to undo that step first. To do this, go to the root directory of the PyTorch project and run:
python setup.py develop --uninstall
This will remove the package installed with develop mode.
Install the package in editable mode again:
Navigate to the root directory of your PyTorch project and run:
pip install -e .
This will install the package in editable mode, which will create a link to your local code, ensuring that your local folder is picked up first in the Python path.
```

I can implement this 




### Suggest a potential alternative/fix

_No response_
[dynamo] Reduce cache size to 4
Labels: module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108402
* #108161



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov
quantized module serialization through prepack function registration
Labels: module: cpu, triaged, open source, release notes: quantization

Fixes #108399

# Motivation

PyTorch only can serialize&deserialize parameters in quantized model for official supported QEngine now. There's no chance to extend it to achieve this goal for out-of-tree QEngine based on current code structure.  `torch.save/torch.load` would fail with a ScriptModule on QuantizedXPU backend.

Serializing quantized module (e.g. `ConvPackedParamsBase, LinearPrepackParamsBase`) depends on specific `prepack&unpack` implementation with regard to each QEngine.  We want to propose a mechanism that allows out-of-tree QEngines register their `prepack` implementation in PyTorch to finish the overall serialization process.

# Modification 
1. Add `register_prepack` `get_device_prepack_fn` utils pair for `qconv`, `qlinear` to register and query the pointer to prepacking function of each QEngine.
2. Add qengine `QXPU`, which is the engine for quantized XPU

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
Generalize weight prepacking during quantized model deserialization
Labels: oncall: quantization, triaged

### 🚀 The feature, motivation and pitch

## Motivation

PyTorch only can serialize&deserialize parameters in quantized model for official supported QEngine now. There's no chance to extend it to achieve this goal for out-of-tree QEngine based on current code structure.  `torch.save/torch.load` would fail with a ScriptModule on QuantizedXPU backend.

Serializing quantized module (e.g. `ConvPackedParamsBase, LinearPrepackParamsBase`) depends on specific `prepack&unpack` implementation with regard to each QEngine.  We want to propose a mechanism that allows out-of-tree QEngines register their `prepack` implementation in PyTorch to finish the overall serialization process.

## Current behaviour
 Current pickling method call `prepack` of different backend based on compilation marco and global context `QEngine`. For each `QEngine`, code of using prepacking function to deserialize weight need to be added in https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/quantized/cpu/fbgemm_utils.cpp#L481.  Like, `PackedLinearWeightsQnnp::prepack`, `PackedLinearWeightsOnednn::prepack` `PackedLinearWeight::prepack`

 There's no chance to extend it to out-of-tree QEngines.
 
Also, it is inconvenient for official supported qengine. For any new qengine added, it is required to add the code for deserialization, and maintainers need to know the some details of pickling method.

## Plan

Considering the `prepack` function has same signature for each `QEgnine`, we suggest to generalize the process of calling prepacking function. The prepacking function could be found at runtime based on the global context `QEngine`. Following is our proposal:

A dictionary is maintained by the library, where the key for each entry is `QEngine` and the value is the pointer to prepacking function of such engine. Each engine only need register the prepack function to the dictionary. When deserializing weight file, the `__getstate__` method can use `QEngine` as key to find the proper pointer to prepacking function. 

For maintainers of each official supported `QEngine`, only registering the prepacking function is needed (through `register_prepack_fn`).  No need to add code in https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/quantized/cpu/fbgemm_utils.cpp#L481 anymore.

For maintainers of out-of-tree `QEngine`, maintainers can register prepacking function within their library.  Then function pointer is inserted to the map after registration and be found at runtime. No modifications is needed in PyTorch.  (Maintainers still need add enum `QEngine` in PyTorch.）

![quantsave0](https://github.com/pytorch/pytorch/assets/12457857/8c86a538-0b53-4640-9cb9-41ac83cfd08a)


 @ezyang  @arthuryuan1987  @gujinghui 

### Alternatives

_No response_

### Additional context

_No response_

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel
FSDP always puts parameters to fp32 when loading state_dict, even if state_dict has bf16 params
Labels: oncall: distributed

### 🐛 Describe the bug

In cases where a non-FSDP trained model is loaded with FSDP and the state_dict to load is bf16, loading into FSDP sharded_state_dict will make the parameters to fp32.

In my particular case, I am converting LLaMA weights to PyTorch FSDP format for FSDP training, but the checkpoint I'm working with is in bf16.

This could cause issues in cases where users want to benchmark numerical equivalence.

### Versions

main

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu
[dynamorunner] init FSDP on meta device to fix OOM on large models
Labels: ciflow/trunk, topic: not user facing, module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #108359

By default init models with `meta` device when using FSDP.

Example command (use own HF token):
```
HUGGING_FACE_HUB_TOKEN=hf_xxx python benchmarks/dynamo/torchbench.py --float16 -dcuda --output=./performance.csv --training --backend=eager --only llama_v2_7b_16h --fsdp --performance --timing --print-memory --multiprocess
```

cc @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov
ONNX export constant folding messes up with shared weight deduplication
Labels: module: onnx, triaged

### 🐛 Describe the bug

Hi, it appears that using `do_constant_folding=True` in the ONNX export will undo some weight deduplication. For example, an `nn.Linear` weight will go from

![image](https://github.com/pytorch/pytorch/assets/9808326/5e0a9e87-8c2a-4f23-ae7c-4c4548e95641)

& 

![image](https://github.com/pytorch/pytorch/assets/9808326/752aabad-eeda-44eb-bb36-326c692ba242)

to


![image](https://github.com/pytorch/pytorch/assets/9808326/19c8e6c4-00f5-4973-bf0c-356670348c78)

effectively transposing the weight. Given that the `DeduplicateInitializersByDataPtr` relies on the tensor size, in case the shared weight has a different size (e.g. an embedding weight), the deduplication pass will fail.

It seems to me that the initializer deduplication should happen before the constant folding, and constant folding should be done only for non-shared weights.

WDTY @justinchuby @BowenBao ?

Thank you!


Repro:
```
pip install optimum
optimum-cli export onnx -m bigscience/bloom-560m bloom_onnx --no-post-process
```

and inspect the output with netron


### Versions

Both on 2.0.1 and nightly
Added attention mechanism error,Need to modify torch.use_deterministic_algorithms(True)
Labels: triaged, module: determinism

### 🐛 Describe the bug

RuntimeError: adaptive_avg_pool2d_backward_cuda does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
Error in adding attention mechanism when using YOLOv5,Need to set torch.use_deterministic_algorithms(False).
However, the code cannot be reproduced. Can you help me?


### Versions

def init_seeds(seed=0, deterministic=False):
# Initialize random number generator (RNG) seeds https://pytorch.org/docs/stable/notes/randomness.html
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed) # for Multi-GPU, exception safe
# torch.backends.cudnn.benchmark = True # AutoBatch problem https://github.com/ultralytics/yolov5/issues/9287
if deterministic and check_version(torch.__version__, '1.12.0'): # https://github.com/ultralytics/yolov5/pull/8213
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.deterministic = True
os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8'
os.environ['PYTHONHASHSEED'] = str(seed)

cc @mruberry @kurtamohler
[dynamo][stream]support device-agnostic stream in dynamo and capture stream/event method in fx graph
Labels: triaged, module: mkldnn, open source, ciflow/trunk, topic: not user facing, module: dynamo, ciflow/inductor

This PR implements 2 things:
1. support the device agnostic stream and runtime APIs captured by the dynamo.
2. support the stream methods(include the event) captured by the dynamo.

Here are details for 1st.
Previously the stream captured in dynamo was tightly bind to CUDA. Here we implement a global singleton container named `StreamMethodContainer` for different backends to register their associated stream methods to dynamo. When import the backend’s product, the stream operations can be registered directly by calling 

```
device_stream_method = {'current_stream': method_1,
                         'create_stream_context': method_2,
                         'set_stream': method_3,
                         'set_stream_by_id': method_4}
torch._dynamo.stream.register_stream_method(device_name, device_stream_method)
```

Stream methods need to be passed in this API according to the precise semantics represented by the dict key in `device_stream_method`. After register, these methods can be used by dynamo to capture the stream operations in users’ script, for example, get the current stream or set the specific stream. Additionally, the wrapped stream variable and the stream context variable are changed to be the device-agnostic, the proxy functions of these variables are assigned by the associated methods in the container. All of this are illustrated in the below. Below is a illustration.

![image](https://github.com/pytorch/pytorch/assets/74231238/37ac7350-c539-4167-9886-c3744ecab65d)


cc @gujinghui @PenghuiCheng @XiaobingSuper @jianyuh @jgong5 @mingfeima @sanchitintel @ashokei @jingxu10 @min-jean-cho @yanbing-j @Guobing-Chen @Xia-Weiwen @voznesenskym @penguinwu @EikanWang @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @anijain2305 @yanboliang
torchbench mfu mem bandwidth+ refactors
Labels: topic: not user facing, module: dynamo, ciflow/inductor

```
(sourcetorch) ubuntu@ip-172-31-1-136:~/pytorch$ ./benchmarks/dynamo/torchbench.py --inference --performance --no-skip --inductor --freezing --only stable_diffusion

cuda eval  stable_diffusion                   
Average memory bandwidth per image over 1 samples: 107.47 GB/s
Model FLOPS: 0.48736161619492085 TF/s
```

EDIT: 

I'm angrily refactoring, we can probably just merge by rebasing at https://github.com/pytorch/pytorch/pull/108296/commits/6019d5a559d9a7ef167e5b151614057f5619f5e4

or I might just do https://github.com/pytorch/pytorch/pull/108296/commits/34145b169ac234ca64ca2cd669101a699245521d in a seperate PR

EDIT 2: These numbers are quite low so either its well known for stable diffusion or a bug in my measurement, will baseline with llama instead

EDIT 3: I will research and summarize other implementations of the metric, try to find a denominator for MFU and finally run a larger sweep, it is expected for most blueberries models that mfu will be low since we are doing bs=1 so need to find other things

cc @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov
ONNX-FX based exporter documentation/tutorial topics for PyTorch 2.1
Labels: module: onnx, triaged, release notes: onnx

```[tasklist]
### Tasks
- [ ] https://github.com/pytorch/tutorials/issues/2543
- [ ] Tutorial: Using diagnostics to find and report bugs (create new diag, generate sarif and vie sarif?)
- [x] Tutorial: Using dynamic shapes
- [x] Tutorial: Using fake tensor to export large scale models
- [ ] [Tutorial: Register custom operators](https://microsoft.sharepoint.com/:w:/t/ONNX2/Ee7O0MS48bZEr7ENbFt35-sBQO5xJsgeuZY4f-CU-ltl3w?e=gLvas8)
- [ ] Deep dive: ONNX-FX architecture
- [ ] Deep dive: Design comparison with TorchScript-based ONNX exporter
- [ ] https://github.com/pytorch/pytorch/issues/108274
- [ ] https://github.com/pytorch/tutorials/issues/2552
```
Back out "Serialize pytree to json string (#106116)"
Labels: fb-exported, module: dynamo, ciflow/inductor, module: export

Summary:
Original commit changeset: 7341758b5249

Original Phabricator Diff: D47848392

Test Plan: Backout of backward compat breaking diff.

Differential Revision: D48834000



cc @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov
[REDO][WIP][cuDNN][cuDNN V8 API] Add experimental cuDNN MHA/Flash Attention support 

This is a copy of https://github.com/pytorch/pytorch/pull/101916 which has gone stale, I am going to rebase and update to try and get some numbers for speed comparison


"""
Initial implementation of forward pass cuDNN Flash Attention; current major restrictions are:

Only packed data layout (e.g., QKV tensors as chunks of the same tensor) supported
Only SM 9.0 and SM 8.0 support
Only head dim and sequence length divisible by 64 supported
Gated by TORCH_CUDNN_MHA_ENABLED=1 environment variable.
The plan is to eventually avoid pattern matching against strides as the support matrix from the cuDNN side is improved

"""