gopikrsmscs/torch-issues huggingface dataset:
Quadric Layer
Labels: feature, module: nn, triaged, needs research

### 🚀 The feature, motivation and pitch

Introducing a layer with second order (quadric) hypersurface separability which in turn reduces model size significantly at the same performance on a high level before even utilizing sparsity/quantization. This approach can be used everywhere as a drop-in for a Linear layer but with substantially reduced size. 

This paradigm is based on my research:
https://www.researchgate.net/publication/221582251_Using_Quadratic_Perceptrons_to_Reduce_Interconnection_Density_in_Multilayer_Neural_Networks

There is also other research about higher order neurons in the field, although later afaik

I have further explained the paradigm in my GitHub repo : https://github.com/diro5t/deep_quadric_learning

In this repo there are further examples of reducing model size in concrete applications  for a singular quadric neuron as well as for quadric layers demonstrated for the MNIST dataset in PyTorch as well as in tinygrad.

The proposed implementation can be seen in my fork https://github.com/diro5t/pytorch in the torch.nn.modules.linear.py.

This feature is also on the PyTorch 2.1 feature list https://docs.google.com/spreadsheets/d/1TzGkWuUMF1yTe88adz1dt2mzbIsZLd3PBasy588VWgk/edit#gid=2032684683

### Alternatives

_No response_

### Additional context

comment regarding this feature:
@[dirk.roeckmann@fivetroop.com](mailto:dirk.roeckmann@fivetroop.com) : Please first create an issue (feature request, here: [https://github.com/pytorch/pytorch/issues/new/choose](https://www.google.com/url?q=https://github.com/pytorch/pytorch/issues/new/choose&sa=D&source=docs&ust=1689635667681400&usg=AOvVaw2iZID1OC247jRrAx4e8vGu)) against pytorch/pytorch.  With this feature description. This issue needs to be accepted by pytorch maintainers in order to be considered.
cc @[albandes@meta.com](mailto:albandes@meta.com) @[nshulga@meta.com](mailto:nshulga@meta.com)

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
[pytorch][codev] add current test finding logic to find_matching_merge_rule
Labels: fb-exported, Stale, topic: not user facing

Summary:
Currently we enable skipping pytorch ci tests from codev which is analagous to "pytorchbot merge -f". I think it'd be reasonable to add functionality analagous to "pytorchbot merge -i" where we only skip currently failing tests. The easiest way to do this is to just have a boolean flag to trigger in find_matching_merge_rule and have the logic to find the currently failing tests there.

#FACEBOOK
D47485107 is the diff that adds a pytorch ci skipping label.

Test Plan: testing for trymerge should cover this change as its just a refactor.

Differential Revision: D47530806
Set dir for aot_inductor output files
Labels: triaged, open source, fb-exported, module: inductor, ciflow/inductor

Summary: Generate a random hash name as dir name for aot_inductor, all aot_inductor output files should write into this hash_name dir. This enables merge net predictor file package.

Differential Revision: D47487758



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
torch.onnx.export does not support divisor_override in AvgPool2d
Labels: module: onnx, triaged



```
import torch
 
# export works fine if divisor_override=None, but fails when divisor_override is integer value
model = torch.nn.AvgPool2d(kernel_size=5, stride=2, padding=1, divisor_override=3)
model.eval()
 
rand_inp = torch.randn(1, 3, 8, 8)
 
torch.onnx.export(model, rand_inp, "AvgPool2dModel.onnx", verbose=True)
```

Above code fails during export with the error “torch/onnx/symbolic_helper.py:243: UserWarning: ONNX export failed on avg_pool2d because divisor_override not supported”
Please add support for export of AvgPool2d with an integer divisor_override value.

### Versions

Torch version: 1.9.1
FSDP Full Shard compatibility with BF16 AMP
Labels: oncall: distributed, triaged, module: fsdp

### 🐛 Describe the bug

Hi all,

Per conversation with @Chillee, I am opening this issue.

I was wondering if there were any potential compatibility issues when using FSDP Full Shard in conjunction with BF16 AMP during training?

I do understand that different Mixed Precision configurations can be selected through:
```python
precision = MixedPrecision(
    param_dtype=torch.float32,
    # Gradient communication precision.
    reduce_dtype=torch.bfloat16,
    # Buffer precision.
    buffer_dtype=torch.bfloat16,
)
```
But I am not sure if these predefined configurations will directly conflict with BF16 AMP:
```python
    with autocast(dtype=torch.bfloat16):
```
Or if they are even compatible at all?

I greatly appreciate your help.

Thank you,

Enrico

### Versions

PyTorch - Stable (2.0.1) CUDA 11.8

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
[ONNX] Refactor `test_fx_op_consistency.py`
Labels: module: onnx, triaged, onnx-triaged

`test_fx_op_consistency.py` references `ops_test.py` in onnx-script. However, onnx-script tests has refactored to include all dtype tests and prims test. A refactor is needed on `test_fx_op_consistency.py`.
```[tasklist]
### Tasks
- [ ] Use OpInfo rtol/atol
- [ ] Add coverage
- [ ] Prims
```
Enable SLEEF on ARM
Labels: module: build, triaged, module: sleef, module: arm, topic: improvements

### 🚀 The feature, motivation and pitch

SLEEF implementations of [functions](https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/cpu/vec/vec256/vec256_float_neon.h#L391-L410) (like exp, erf, etc) are much faster than their corresponding STD implementations. Currently, on Intel, the SLEEF implementation is the [default](https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/cpu/vec/vec256/vec256_float.h) one for many of these functions, but not on ARM. 

On ARM, it is controlled by the flag `AT_BUILD_ARM_VEC256_WITH_SLEEF`. It seems like this flag is controlled by the `USE_SLEEF_FOR_ARM_VEC256` [option](https://github.com/pytorch/pytorch/blob/e5f5bcf6d4ec022558caf4d0611d928497394a88/CMakeLists.txt#L278) in CMakeLists. But setting that option to ON and building Pytorch did not set the `AT_BUILD_ARM_VEC256_WITH_SLEEF` flag, and the SLEEF code was still not being executed. 

I'd like to understand what is the correct way to enable SLEEF on ARM, and if there is any issue with enabling it by default (like on Intel).


### Alternatives

_No response_

### Additional context

_No response_

cc @malfet @seemethere
DISABLED test_super_resolution_cuda (__main__.TestModels)
Labels: oncall: jit, module: flaky-tests, skipped

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_super_resolution_cuda&suite=TestModels) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/15097403578).

Over the past 3 hours, it has been determined flaky in 3 workflow(s) with 5 failures and 3 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_super_resolution_cuda`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_jit.py`

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Softmax doesn't support sparse tensors with the CSR layout
Labels: module: sparse, triaged, topic: new features

### 🚀 The feature, motivation and pitch

Hi,
both `torch.softmax` and `torch.sparse.softmax` don't support the CSR format.
Namely, when I try to apply any softmax to `torch.sparse_csr tensor`, I receive the following error:

``RuntimeError: unsupported tensor layout: SparseCsr``

Would it be possible to start supporting this format?

### Alternatives

_No response_

### Additional context

_No response_

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer
TorchInductor Hack-a-Day on July 19th
Labels: triaged, tracker

### Better Engineering
- [x] #105571
- [ ] #105572

### Unit test
- [ ] skipIfTorchInductor tracker: https://github.com/pytorch/pytorch/issues/102207
- [ ] #105534

### AOTInductor
- [x] #105552
- [x] #105553
- [ ] #105554
- [ ] Support rand fallback, https://github.com/pytorch/pytorch/issues/103415
- [ ] #105555

###  Decompositions (write decomp, verify that Inductor has good performance on them and fuses as expected):
- [ ] #105556
- [x] #105557
- [x] #105558
- [x] #105559
- [ ] #105560
- [ ] #105561
- [ ] #105562
- [x] #105563
- [x] #105564
- [x] #105565
- [ ] #105566
- [ ] #105567
- [ ] #105568

### Decomposition optimizations
- [ ] Lowering matmuls to pointwise operators when they’re small or bandwidth bound (https://github.com/pytorch/pytorch/issues/103313) 
- [ ] #105569

### Lowering:
- [ ] #105570
aten.grid_sampler_2d_backward (investigate using decomposition for backward as well)
aten.avg_pool3d

### Performance:
- [ ] Improve concat fusion with matmuls,   https://github.com/pytorch/pytorch/issues/102804
- [ ] Do smarter layout planning with concatenate, https://github.com/pytorch/pytorch/issues/102805
- [ ] Do concatenate copies with a foreach kernel if applicable, https://github.com/pytorch/pytorch/issues/103475
- [ ] Pattern-match operators into foreach kernels
- [ ] Stop zero-ing out non-differentiable outputs in AOTAutograd, https://github.com/pytorch/pytorch/issues/104272 

### Compilation time
- [ ] `will_fusion_create_cycle` takes a long time  , https://github.com/pytorch/pytorch/issues/98467
Can't vmap over torch.tensor constructor
Labels: triaged, module: functorch

### 🐛 Describe the bug

Initially reporeted by @mra-h over [here](https://github.com/pytorch/pytorch/issues/102109#issuecomment-1634596330)
```py
def skew_matrix(w):
   torch.tensor([[0.0, -w[2], w[1]],
                 [w[2], 0.0, -w[0]],
                 [-w[1], w[0], 0.0]])

points = torch.rand((1024, 3))
fn = torch.vmap(skew_matrix, in_dims=(0))
fn(points)
```


### Versions

main

cc @Chillee @samdow @kshitij12345 @janeyx99
Padded tensor subclass
Labels: feature, triaged, module: nestedtensor, tensor subclass

### 🐛 Describe the bug

Suppose you have a network which operates on dynamically sized inputs / has data dependent dynamism internally. Our default policy is to represent such a tensor as compactly as possible (e.g., with no padding) to minimize storage and FLOPs needed to operate on it.

However, in some situations, it could be profitable / cost free to pad out the tensor:

* If you are CUDA graphing with dynamic shapes and you know your maximum size, padding in the outermost (e.g., batch) dimension is effectively free, because the CUDA graph will require you to maintain memory equivalent to the maximum memory usage for your dynamic shapes. In fact, it is better than free, because ensuring you always allocate the same amount of memory every iteration ensures that you will use the same allocations; the allocator otherwise can make bad decisions in the name of "saving" memory (e.g., if you previously allocated a tensor out of a 10MB block, but this time you only need 5MB because you halved your sequence length, instead of serving the allocation out of the 10MB, it might allocate an *extra* 5MB to "save" the 10MB for later (even though it will never be used!)
* Increasing the size of tensors can improve the performance on kernels. @Chillee has a good explainer about this phenomenon in matmuls at https://twitter.com/cHHillee/status/1630274804795445248 In fact, @msaroufim and @Chillee tried to add this optimization directly to PyTorch but the post facto layout change was a bit hard to implement. Doing the layout change "early" with a tensor subclass should be easier to implement (albeit less automatic.) These improvements generalize beyond matmuls, although mostly for making sure your sizes are divisible by something nice. Fully automatic size increases here are a little difficult to do, because you have to know that later you're going to do a matmul, and you also have to know that you aren't losing all your gains from non-contiguous kernels. However, if you have a net where one of the input dimensions is dynamic, you can choose to bucket to reduce the number of CUDA graphs you need. That being said, if the dynamic dimension is batch size (or even sequence length but you have embeddings on the inner dimensions so there's no padding problems), you aren't going to get kernel perf wins.

Padded batch tensor for dynamic batch size is probably the easiest to implement to start, because you can use symbolic shape propagation rules to propagate batch dim and ensure they're properly padded (I can't think of a good way to make vmap do this.) Annoyance is avoiding wasted FLOPs by "adjusting down" the logical size.

Related: https://github.com/pytorch/pytorch/issues/65156
Related: nested tensor

cc @cpuhrsch @jbschlosser @bhosmer @drisspg @msaroufim @albanD 

### Versions

main
DeadKernel when training GNN for Cora on MPS
Labels: triaged, module: mps

              > With torch ver: 2.0.1 MPS is faster, using the original toy example.
> CPU: 16.08941249999998
> MPS: 3.2959765830000833

Hey I'm also using PyTorch 2.0
My config: Apple M2 16gb ram
Im trying to train a simple GNN for the Cora dataset.(which is < 1 Mb) I use Jupyter notebook. When my device is CPU-  it runs quickly. When I use MPS, I get DeadKernel. Am I missing something while using MPS?

_Originally posted by @narenq7 in https://github.com/pytorch/pytorch/issues/77799#issuecomment-1635749861_
            

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
Implementation of torch.sparse.sampled_baddmm
Labels: module: sparse, triaged, topic: new features

### 🚀 The feature, motivation and pitch

Hi,
I would like to perform a batch matrix-matrix product with a per-sample mask.
It's similar to [torch.sparse.sampled_addmm](https://pytorch.org/docs/stable/generated/torch.sparse.sampled_addmm.html), the only difference is that `input` would be a (b, m, n) sparse tensor in the CSR format, unless we could provide masks as a list consisting of b (m, n) tensors.
It might be blocked by https://github.com/pytorch/pytorch/issues/104193 though.

### Alternatives

_No response_

### Additional context

_No response_

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer
[docs] torch.sigmoid to make clear equivalence relations to other sigmoid functions
Labels: module: docs, triaged, actionable

### 📚 The doc issue

Currently https://pytorch.org/docs/stable/generated/torch.sigmoid.html only points to https://pytorch.org/docs/stable/special.html?highlight=expit#torch.special.expit

Both do not link to https://pytorch.org/docs/stable/generated/torch.nn.Sigmoid.html?highlight=sigmoid#torch.nn.Sigmoid or https://pytorch.org/docs/stable/generated/torch.Tensor.sigmoid.html?highlight=sigmoid#torch.Tensor.sigmoid

https://pytorch.org/docs/stable/special.html?highlight=expit#torch.special.expit (and other special functions) should deserve their own `.html` pages btw (as all other functions). expit should mention explicitly if it's just an alias to torch.sigmoid and torch.nn.functional.sigmoid

Ideally all these function variants should just say "See [some single variant]" (maybe to torch.sigmoid? or torch.nn.Sigmoid as it contains currently the function plot)

Currently there exists 4 functional variants (without considering quantized variants) and 1 module variant and findable in docs:
- torch.sigmoid
- torch.Tensor.sigmoid
- torch.nn.functional.sigmoid
- torch.special.expit
- torch.nn.Sigmoid

As usual, search results also contain duplicates and snippets are quite bad (it even finds sth like `[FXE0004:fx-pass-convert-neg-to-sigmoid]` - I would say that Sphinx search should not index code examples/source code itself - only text) :(
<img width="305" alt="image" src="https://github.com/pytorch/pytorch/assets/1041752/adc1ff18-1ebf-45b4-be62-1ac6577d6b87">


### Suggest a potential alternative/fix

_No response_

cc @svekars @carljparker
Support ONNX opset 20 to export GELU to one single op
Labels: module: onnx, triaged

### 🚀 The feature, motivation and pitch

ONNX added GELU (and its tanh approximation ) as a new op in its opset 20: https://github.com/onnx/onnx/blob/main/docs/Operators.md#Gelu

It will be great to upgrade Pytorch ONNX export to support it. Model visualization and GELU pattern matching will be a lot easier.

### Alternatives

_No response_

### Additional context

_No response_
Optimize PyTorch C++ part with Profile-Guided Optimization (PGO)
Labels: module: performance, module: internals, triaged, oncall: pt2

### 🚀 The feature, motivation and pitch

Profile-Guided Optimization (PGO) helps a lot with optimizing different kinds of software based on runtime execution profiles.

PyTorch has a huge C++ [part](https://github.com/pytorch/pytorch/tree/main/torch/csrc) that could be optimized with PGO. E.g. compiler load and JIT. Similar projects like Clang, GCC, Rustc, CPython and others are already built with PGO and proved that PGO (and more advanced techniques like LLVM BOLT) can help here.

So would be great to see PGO results on PyTorch codebase. I am not familiar (at least yet) with the codebase but I think starting with optimizing the compiler is a good starting point. If you know a better place - please correct me. If anyone already tried to optimize these parts with PGO and has some results - would be great if you can share them here.

### Alternatives

Leave things as is.

### Additional context

More about real-life results of applying PGO on different kinds of software you can find (and more materials about PGO and other optimization techniques like LLVM BOLT) [here](https://github.com/zamazan4ik/awesome-pgo/).

cc @ezyang @bhosmer @smessmer @ljk53 @bdhirsh @msaroufim @wconstab @anijain2305
recompile fx.GraphModule lazily
Labels: release notes: quantization, release notes: fx, module: dynamo, ciflow/inductor, module: export, suppress-bc-linter

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #105257

Context: @eellison 's review comment [here](https://github.com/pytorch/pytorch/pull/103642#discussion_r1244418586) complains about my code calling `torch.fx.GraphModule.recompile` after I changed the graph.  We didn't simply remove the call to `recompile` at that time since that increases the risk that user see or run stale python code. In this PR, I recompile GraphModule lazily without increasing the risk that user see/run stale python code.

When training BertForMaskedLM, the `GraphModule.recompile` is called 707 times and takes 1.8s in total. The whole compilation takes around 60 seconds.

By spot checking, I found the main reason we call recompile so frequently is due to inductor pattern matcher. E.g., if we want to replace src_fn with dst_fn, we need trace both src_fn and dst_fn. After tracing is done, we create a GraphModule. The init method of GraphModule will call recompile.

By doing recompile lazily, we reduce the number of calls for `GraphModule._real_recompile` (in this PR, `recompile` just mark the class as needing recompilation and is very light weight. `_real_recompile` does the real recompilation) to 37 times and reduces its total execution time to 0.045s.



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @anijain2305 @ipiszy
[discussion] Integrate widely used utilities from fvcore into the core repo
Labels: oncall: distributed, feature, module: nn, triaged, needs research, module: LrScheduler

### 🚀 The feature, motivation and pitch

Some things like FLOP counter have already been reimplemented in core. But I propose to systematically study widely used things in fvcore (primarily uses in detectron2: https://github.com/search?q=repo%3Afacebookresearch%2Fdetectron2+fvcore+language%3APython&type=code&l=Python) and maybe move some of them into core (it would allow people to take fewer dependencies. that repo has existed now for a long time, so popularity/success of utils there can now be estimated more or less well):

- For instance, in vision/detectron2-related code weight inits from https://github.com/facebookresearch/fvcore/blob/main/fvcore/nn/weight_init.py are often used

- Another useful one is https://github.com/facebookresearch/fvcore/blob/main/fvcore/nn/print_model_statistics.py

- There are also some distributed utils

- It also has stateless schedulers used by detectron2: "A stateless, scale-invariant hyperparameter scheduler: see its [API doc](https://detectron2.readthedocs.io/en/latest/modules/fvcore.html#fvcore.common.param_scheduler.ParamScheduler)". It is also related to https://github.com/pytorch/pytorch/issues/68332, so maybe can be this design can be taken as base for schedulers enhancement / redesign in core (it has float iterations and uses `__call__` syntax; might be more generic to allow for int64 iterations and derive from nn.Module? Or even better: fully stateless free functions)

There are also a bunch of losses there that seem to have been already implemented in core. It would be useful then to deprecate / add warnings to the impls in fvcore or at least add a word in README there.

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
DISABLED test_fused_optimizers_with_large_tensors (optim.test_optim.TestOptim)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on master ([recent examples](http://torch-ci.com/failure/test_optim.py%3A%3ATestOptim%3A%3Atest_fused_optimizers_with_large_tensors)).

cc @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
DISABLED test_cross_entropy_large_tensor_reduction_mean_cuda (__main__.TestNNDeviceTypeCUDA)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on master ([recent examples](http://torch-ci.com/failure/test_nn.py%3A%3ATestNNDeviceTypeCUDA%3A%3Atest_cross_entropy_large_tensor_reduction_mean_cuda)).

cc @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
Multiple linux jobs are failing with version `GLIBCXX_3.4.30' not found 
Labels: module: ci, triaged

## Current Status
mitigated by reverting

## Error looks like
Linux jobs are failing with following error:
```
+ python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/__init__.py", line 236, in <module>
    from torch._C import *  # noqa: F403
ImportError: /opt/conda/envs/py_3.10/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
```

## Incident timeline (all times pacific)
5:00PM EST Issue start happening
5:45PM EST Issue was noticed and OSS CI channel advised

## User impact
multiple inductor, periodic, pull and trunk jobs are failing

## Root cause
tbd

## Mitigation
tbd

## Prevention/followups
tbd


cc @seemethere @malfet @pytorch/pytorch-dev-infra
Significant time difference of calculating Jacobian matrix using jacrev and oracle functions
Labels: module: autograd, triaged, module: functorch

### 🚀 The feature, motivation and pitch

Sorry I'm not sure if this is a new feature, but I'd be very grateful if you guys would like to improve this part.

Specifically, I found that using `jacrev` would be much slower than using the oracle Jacobian function:
```python
from torch.func import vmap, jacrev
import torch
import time

a = torch.rand(10000, 10000)

def f(x):
    return (x ** 2).sum(-1)

def df(x):
    return 2 * x


t0 = time.time()
b = df(a)
t1 = time.time()
c = vmap(jacrev(f))(a)
t2= time.time()

assert torch.allclose(b, c)
print(t1 - t0, t2 - t1)
```
result: 0.10568618774414062 0.9206998348236084

Given that oracle's Jacobian is readily available in neural networks, I wonder why using `jacrev` is so much slower? Is there something wrong with me?

Of course, I can actually rewrite each layer of the neural network to obtain the value and Jacobian at the same time, but calculating the Hessian matrix is too troublesome. It would be great if `jacrev` could be faster!



### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @Chillee @samdow @kshitij12345 @janeyx99
[DTensor] Dtensor API should report the correct device when GPU is used
Labels: triaged, module: dtensor

### 🚀 The feature, motivation and pitch

The DTensor API is not reporting the correct device id of GPU.

```python
from torch.testing._internal.common_distributed import spawn_threads_and_init_comms
import torch
import torch.distributed as dist
from torch.distributed._tensor import DTensor, DeviceMesh, Shard, Replicate, distribute_tensor

from torch.distributed.tensor.parallel import (
  PairwiseParallel,
  RowwiseParallel,
  ColwiseParallel,
  parallelize_module,
  make_input_replicate_1d,
  make_output_replicate_1d,
  make_output_shard_1d,
)

WORLD_SIZE = 2

@spawn_threads_and_init_comms(world_size=WORLD_SIZE)
def shard_big_tensor(world_size):
    mesh = DeviceMesh('cuda', list(range(WORLD_SIZE)))
    big_tensor = torch.randn((7, 3, 1024, 2048))
    spec = [Shard(3)]
    dtensor = distribute_tensor(big_tensor, mesh, spec)
    print(f'Rank: {dist.get_rank()}, dtensor global shape: {dtensor.shape}, local shape: {dtensor.to_local().shape}, dtensor.device: {dtensor.device}, dtensor.to_local().device: {dtensor.to_local().device}')

if __name__ == "__main__":
    shard_big_tensor(WORLD_SIZE)
```

LOG
```bash
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
Rank: 1, dtensor global shape: torch.Size([7, 3, 1024, 2048]), local shape: torch.Size([7, 3, 1024, 1024]), dtensor.device: cuda:0, dtensor.to_local().device: cuda:0
Rank: 0, dtensor global shape: torch.Size([7, 3, 1024, 2048]), local shape: torch.Size([7, 3, 1024, 1024]), dtensor.device: cuda:0, dtensor.to_local().device: cuda:0
```

Why `dtensor.to_local().device` both report `cuda:0`  in two ranks? There are two GPUs used actually, and it should be `cuda:0` and `cuda:1` respectively.

### Alternatives

_No response_

### Additional context

_No response_
[DTensor] Module parallelized with ColwiseParallel should return a sharded tensor
Labels: triaged, module: dtensor

### 🚀 The feature, motivation and pitch

I do not know why a module parallelized with ColwiseParallel returns a replicated tensor. It should return a sharded tensor according to the document.

```python
@spawn_threads_and_init_comms(world_size=WORLD_SIZE)
def demo_colwise_parallel(world_size):
    rank = dist.get_rank()
    mesh = DeviceMesh('cuda', list(range(WORLD_SIZE)))
    spec = [Replicate()]
    model = torch.nn.Linear(10, 32)

    LR = 0.25
    optimizer = torch.optim.SGD(model.parameters(), lr = LR)
    model = parallelize_module(model, mesh, ColwiseParallel())

    for i in range(ITER_TIME):
        inp = torch.randn(20, 10)
        dtensor = distribute_tensor(inp, mesh, spec)
        output = model(dtensor)
        output.sum().backward()
        print(f'Iter {i}, rank: {dist.get_rank()}, ', dtensor.to_local().shape, output.to_local().shape)
        optimizer.step()
```

log
```bash
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
Iter 0, rank: 1,  torch.Size([20, 10]) torch.Size([20, 32])
Iter 0, rank: 0,  torch.Size([20, 10]) torch.Size([20, 32])
```

I expect `output.to_local().shape` is (20, 16) instead of the current (20, 32).


### Alternatives

_No response_

### Additional context

_No response_
Repro str could be displayed with slightly wrong env vars
Labels: module: docs, module: cuda, triaged, actionable

## Issue description

While reproducing https://github.com/pytorch/pytorch/pull/102409#issuecomment-1615254393, I got
```console
...
To execute this test, run the following from the base repo dir:
    TEST_CUDA_MEM_LEAK_CHECK=1 python test/test_foreach.py -k test_outplace_forward_mode_AD__foreach_expm1_cuda_float64

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
```

but I speculate the correct way to enable `CUDA_MEM_LEAK_CHECK` is to set `PYTORCH_TEST_CUDA_MEM_LEAK_CHECK`, not `TEST_CUDA_MEM_LEAK_CHECK` as follows
```
PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/test_foreach.py -k test_outplace_forward_mode_AD__foreach_expm1_cuda_float64
```

## Code example

N/A

## System Info

- PyTorch or Caffe2: pytorch of https://github.com/crcrpar/pytorch/commit/1cc2ec0fe83af238e7880b1db9e5cda4a53bb145
- How you installed PyTorch (conda, pip, source): source

cc @svekars @carljparker @ptrblck
[DO NOT MERGE][NCCL][CUDA][CUDA Graphs] Set watchdog runtime capture mode to thread local to handle cleaning straggling work
Labels: module: cuda, triaged, module: nccl, open source, module: cuda graphs, ciflow/trunk, topic: not user facing, ciflow/periodic, ciflow/inductor

A more minimal alternative to #104487 and #104555 that should not unwittingly suppress errors during graph captures.

I've tested this locally and it appears to fix the repro used to test #104487 and #104555, while checking that the watchdog is attempting to call `cudaEventQuery` (on an event recorded before the capture) in the middle of a capture on another thread.

X-posting summary from #104555 here:
1. The issue is a crash caused by the watchdog thread calling `cudaEventQuery` on work that was enqueued before a graph capture.
2. The initial proposal was to change the capture mode of the thread performing the capture to the "thread local" mode, so that other threads querying events (the watchdog) don't crash as the events in question were created before the actual capture and therefore shouldn't interfere with the capture.
3. Many such as @eellison and @albanD were concerned with how we changed the capture mode of the capturing thread in this PR (by globally changing the constructor), as we could potentially miss errors/failures caused by always using this more lenient mode.
4. @albanD pointed out that it is really just the watchdog thread that we are concerned with issuing a disallowed `cudaEventQuery` call, so another potential solution is to just have the watchdog thread operate in a more lenient mode. This seems to be fairly minimal change that shouldn't have unexpected effects on the correctness or debuggability of code involving graph captures. To clarify, while the watchdog thread itself is not capturing anything, since the watchdog thread never calls `cudaThreadExchangeStreamCaptureMode`, it is operating with the "global" mode when another thread is in the middle of a capture, and hence crash the capture because `cudaEventQuery` is disallowed in this scenario.
4a. Our current understanding is the following (assuming the `cudaEventQuery` is for an event outside of a capture).

capturing thread mode | watchdog thread mode | watchdog event query allowed |
--------------------------|---------------------------|-----------------------------------|
global (default)             | global (default)              | no                                                |
global (default)             | thread local                    | yes  (this PR)                               |
thread local                   | global (default)              | yes  (#104555)                             |
thread local                   |thread local                     | yes                                              |


CC @kwen2501 @eellison @albanD @ptrblck @Aidyn-A 

cc @ptrblck @mcarilli @ezyang
[Dynamo]`__torch_function__` tracing support
Labels: module: inductor, module: dynamo, ciflow/inductor, release notes: dynamo

[Design Doc](https://docs.google.com/document/d/1WBxBSvW3NXhRp9ncmtokJloMLCtF4AYNhJaffvHe8Kw/edit#heading=h.vacn73lozd9w)

In this PR:
* Refactored `__torch_function__` handling to handle subclasses in any argument position. 
* `__torch_function__` dispatch ordering is exactly as specified in the reference dispatcher
* Added handling for `__torch_function__` overrides on custom objects (allows wrapper classes)
* Added handling for `__torch_function__` attribute accesses  and method calls on TensorWithTFOverrideVariable
* More tests

Future work needed:
* Properly make TensorWithTFOverrideVariable subclass TensorVariable with overridden functions in the correct place
* Support custom attributes on TensorWithTFOverrideVariable
* Make subclass handling sound; for example, if `__torch_function__` mutates guardable properties of a tensor, this could introduce constant recompiles, so we should graph break in this scenario
* Unwrap inputs to the graph so that their `__torch_function__` impl will not get triggered within the compiler/graph code, since the behavior of the `__torch_function__` impl should already be traced into the graph

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @anijain2305
PT2 custom ops does not work with future annotations
Labels: triaged, module: custom-operators, oncall: pt2

              PT2 custom ops does not work with future annotations

_Originally posted by @BowenBao in https://github.com/pytorch/pytorch/pull/105156#discussion_r1262892912_
            

Repro:
```python
from __future__ import annotations

import torch
from torch._custom_op import impl as custom_op

@custom_op.custom_op("mylibrary::foo_op")
def foo_op(x: torch.Tensor) -> torch.Tensor:
    ...

```
`ValueError: custom_op(...)(func): Parameter x has unsupported type torch.Tensor. The valid types are: dict_keys([<class 'torch.Tensor'>, typing.Union[torch.Tensor, NoneType], typing.Sequence[torch.Tensor], typing.Sequence[typing.Union[torch.Tensor, NoneType]], <class 'int'>, typing.Union[int, NoneType], typing.Sequence[int], typing.Union[typing.Sequence[int], NoneType], <class 'float'>, typing.Union[float, NoneType], typing.Sequence[float], typing.Union[typing.Sequence[float], NoneType], <class 'bool'>, typing.Union[bool, NoneType], typing.Sequence[bool], typing.Union[typing.Sequence[bool], NoneType], <class 'str'>, typing.Union[str, NoneType], typing.Union[int, float, bool], typing.Union[int, float, bool, NoneType], typing.Sequence[typing.Union[int, float, bool]], <class 'torch.dtype'>, typing.Union[torch.dtype, NoneType], <class 'torch.device'>, typing.Union[torch.device, NoneType]]). Got func with signature (x: 'torch.Tensor') -> 'torch.Tensor')`

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
[ROCm] Add ROCm AMDGPU support for inductor cpp codegen
Labels: module: rocm, triaged, open source, Merged, Reverted, ciflow/trunk, ciflow/periodic, module: inductor, ciflow/inductor, rocm, ciflow/unstable, release notes: inductor

Follows from previous enablement attempt: https://github.com/pytorch/pytorch/pull/101797

Adds support for hsaco binaries in inductor's cpp_wrapper codegen and enables the CUDA tests in test_cpp_wrapper.

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @hongxiayang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @ngimel
TypeError: 'NoneType' object is not subscriptable (Occurred when translating col2im). Can't translate torch.nn.functional.fold in opset_version 18.
Labels: module: onnx, triaged

### 🐛 Describe the bug

```python
import torch
import torch.nn as nn
import torch.nn.functional as F

x=torch.randn(176,64,56,56)
y=torch.randn(176,128,28,28)

input_names=['input']
output_names=['output']

class embedding_concat(nn.Module):
    def __init__(self):
        super(embedding_concat,self).__init__()
        
    def forward(self,x):
        
        y=torch.randn(176,128,28,28)
        B, C1, H1, W1 = x.size()
        _, C2, H2, W2 = y.size()
        s = int(H1 / H2)
        x = F.unfold(x, kernel_size=s, dilation=1, stride=s)
        x = x.view(B, C1, -1, H2, W2)
        z = torch.zeros(B, C1 + C2, x.size(2), H2, W2)
        for i in range(x.size(2)):
            z[:, :, i, :, :] = torch.cat((x[:, :, i, :, :], y), 1)
        z = z.view(B, -1, H2 * W2)
        print(z.shape)
        fold=nn.Fold(output_size=(H1,W1),kernel_size=s,stride=s)
        z = fold(z)
        
        return z



model=embedding_concat()
torch.onnx.export(model,
                  x,
                  "embedding_concat.onnx",
                  input_names=input_names,
                  output_names=output_names,
                  opset_version=18, 
                  do_constant_folding=False, # 是否压缩常量
                  #export_params=False,
                  dynamic_axes={"input":{1: "channel",2:"h",3:"w"},}
                  )
```

### Versions

```
Collecting environment information...
PyTorch version: 2.0.0+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 家庭中文版
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.8.17 (default, Jul  5 2023, 20:44:21) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22621-SP0
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3050 Laptop GPU
Nvidia driver version: 528.79
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=2500
DeviceID=CPU0
Family=205
L2CacheSize=4096
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2500
Name=12th Gen Intel(R) Core(TM) i5-12500H
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] flake8==6.0.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.24.4
[pip3] numpydoc==1.5.0
[pip3] torch==2.0.0+cu117
[pip3] torchaudio==2.0.1+cu117
[pip3] torchvision==0.15.1+cu117
[conda] numpy                     1.24.4                   pypi_0    pypi
[conda] numpydoc                  1.5.0                    pypi_0    pypi
[conda] torch                     2.0.0+cu117              pypi_0    pypi
[conda] torchaudio                2.0.1+cu117              pypi_0    pypi
[conda] torchvision               0.15.1+cu117             pypi_0    pypi
```
DISABLED test_conv (quantization.jit.test_quantize_jit.TestQuantizeJit)
Labels: module: rocm, triaged, skipped

Platforms: rocm

Disabling this unit test since it showed up in CI runs for ROCm5.6 CI upgrade PR https://github.com/pytorch/pytorch/pull/103092: https://github.com/pytorch/pytorch/actions/runs/5527567640/jobs/10083707569

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @BLOrange-AMD
DISABLED test_conv_transpose (quantization.jit.test_quantize_jit.TestQuantizeJit)
Labels: module: rocm, triaged, skipped

Platforms: rocm

Disabling this unit test since it showed up in CI runs for ROCm5.6 CI upgrade PR https://github.com/pytorch/pytorch/pull/103092: https://github.com/pytorch/pytorch/actions/runs/5527567640/jobs/10083707569

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @BLOrange-AMD
DISABLED test_observer_with_ignored_function (quantization.jit.test_quantize_jit.TestQuantizeJit)
Labels: module: rocm, triaged, skipped

Platforms: rocm

Disabling this unit test since it showed up in CI runs for ROCm5.6 CI upgrade PR https://github.com/pytorch/pytorch/pull/103092: https://github.com/pytorch/pytorch/actions/runs/5527567640/jobs/10083707569

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @BLOrange-AMD
DISABLED test_single_linear (quantization.jit.test_quantize_jit.TestQuantizeJit)
Labels: module: rocm, triaged, skipped

Platforms: rocm

Disabling this unit test since it showed up in CI runs for ROCm5.6 CI upgrade PR https://github.com/pytorch/pytorch/pull/103092: https://github.com/pytorch/pytorch/actions/runs/5527567640/jobs/10083707569

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @BLOrange-AMD
DISABLED test_nested (quantization.jit.test_quantize_jit.TestQuantizeJit)
Labels: module: rocm, triaged, skipped

Platforms: rocm

Disabling this unit test since it showed up in CI runs for ROCm5.6 CI upgrade PR https://github.com/pytorch/pytorch/pull/103092: https://github.com/pytorch/pytorch/actions/runs/5527567640/jobs/10083707499

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @BLOrange-AMD
DISABLED test_unary_ops (__main__.TestTensorExprFuser)
Labels: module: rocm, triaged, skipped

Platforms: rocm

Disabling this unit test since it showed up in CI runs for ROCm5.6 CI upgrade PR https://github.com/pytorch/pytorch/pull/103092: https://github.com/pytorch/pytorch/actions/runs/5527567640/jobs/10083707389

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @BLOrange-AMD
MacOS arm64 runners are not available in CI
Labels: module: ci, triaged

## Current Status
mitigated

## Error looks like
MacOS arm64 runners are not available following build will fail: ``trunk / macos-12-py3-arm64 / build``

## Incident timeline (all times pacific)
Jul 12 7:40 pm EST First job failed
Jul 12 8:56 pm EST as OSS CI oncall this issue is noticed and investigation is started
Jul 13 12:40AM EST runner reprovisioned

## User impact
MacOS arm64 runners are not available following build will fail: ``trunk / macos-12-py3-arm64 / build``

## Root cause
github daemon auto-updated in all instances at the same time, downloading a new version and then failing to start.

## Mitigation
tbd

## Prevention/followups
tbd


cc @seemethere @malfet @pytorch/pytorch-dev-infra
Remaining functions without meta registrations
Labels: triaged, module: meta tensors

### 🐛 Describe the bug

I think some of these can't have meta registrations (like unique), and some of them are legacy and we shouldn't bother with.

https://gist.github.com/ezyang/99a46bab8992ad142e7973e0e53d6eff

Computed with

```
print('\n'.join(sorted([n for n in torch._C._dispatch_get_all_op_names() if not torch._C._dispatch_has_computed_kernel_for_dispatch_key(n, "Meta") and n.startswith('aten::')])))
```

cc @eellison @bdhirsh @nkaretnikov 

### Versions

main
workaround for using vmap when .item() is being used internally
Labels: triaged, module: vmap, module: functorch

Example code:

    def func(x):
        for i in torch.arange(x):
            pass
        return None

    y = torch.tensor([3, 3])
    torch.vmap(func)(y)

Output:

    RuntimeError                              Traceback (most recent call last)
          4     return None
          6 y = torch.tensor([3, 3])
    ----> 7 torch.vmap(func)(y)

    ...

          1 def func(x):
    ----> 2     for i in torch.arange(x):
          3         pass
          4     return None

    RuntimeError: vmap: It looks like you're calling .item() on a Tensor. We don't support vmap over calling .item() on a Tensor, please try to rewrite what you're doing with other operations. If error is occurring somewhere inside PyTorch internals, please file a bug report.

Commentary: I would like to use functorch's vmap, with one of the input arguments of the function an integer representing a variable number of iterations of a for loop. Apparently, torch.arange() internally uses .item() and vmap cannot handle that. Is there any workaround?

This issue seems similar to pytorch/functorch#747, but it is not exactly the same.



cc @zou3519 @Chillee @samdow @kshitij12345 @janeyx99
[RFC] Proposal to upgrade LLVM version
Labels: triaged, NNC, module: cpu inductor

### 🚀 The feature, motivation and pitch

Currently, LLVM 9.0.1 is being used by PyTorch, which is quite old.
Some features we would like to add down the line require LLVM 10 or above.
If upgrading LLVM wouldn't break anything, then we would like to do so.

### Alternatives

Linking two separate versions of LLVM is an alternative, but it might cause symbol-resolution issues, and a workaround might increase the binary-size.

### Additional context

We'd like to upgrade the LLVM version to a version between 10 & 16.0.6.

cc @EikanWang @jgong5 @malfet @seemethere
Fix kwargs for `checkpoint`; composition with  `fully_shard`
Labels: Stale, release notes: distributed (fsdp), topic: bug fixes

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #105090

This PR fixes kwargs for `checkpoint` and some composability issues with `fully_shard`, where the FSDP pre/post-forward incorrectly runs twice in backward causing errors.

Unlike for the module wrapper path, `checkpoint`'s `_no_hook` context only disables its own hooks, not FSDP's hooks. Hence, we need to make sure that FSDP does not re-run its forward hooks in backward. Checking against `BACKWARD_PRE` training state does this.
[ONNX] Support aten::var_mean
Labels: module: onnx, triaged

              Is there a issue to track it? or at least add a `TODO:`

_Originally posted by @thiagocrepaldi in https://github.com/pytorch/pytorch/pull/104491#discussion_r1261282006_
            

* ONNX shape inference segfaults in CI. Suspect it is a known fixed issue in ONNX main.
* Support type promotion for `var_mean`.
test_view_dynamic_zero_dim no longer testing zero input
Labels: module: onnx, triaged

### 🐛 Describe the bug

https://github.com/pytorch/pytorch/pull/104828#discussion_r1261361082

Filing an issue as requested by @thiagocrepaldi @titaiwangms 

I don't think the exported model should work with a zero size input. The reason is that we 0/1 specialize on tracing; when you trace a model that has size 2, we assume that if you dynamically vary the shape, you won't vary it to 0 or 1. It is strange to have a model that exports and works with either size 0, or size 2.

If there is a more realistic example, we should put it in here instead.

### Versions

main