gopikrsmscs/torch-issues huggingface dataset:
Raise TypeErrors if Tensor cannot be cast to scalar
Labels: release notes: python_frontend, topic: bc breaking

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110642
* #110619
* #110618

From Python documentation on [built-in exceptions](https://docs.python.org/3/library/exceptions.html#TypeError):
> Passing arguments of the wrong type (e.g. passing a [list](https://docs.python.org/3/library/stdtypes.html#list) when an [int](https://docs.python.org/3/library/functions.html#int) is expected) should result in a [TypeError](https://docs.python.org/3/library/exceptions.html#TypeError), but passing arguments with the wrong value (e.g. a number outside expected boundaries) should result in a [ValueError](https://docs.python.org/3/library/exceptions.html#ValueError)


which should result in the following behavior:
```python
>>> float(torch.tensor([1, 2]))
TypeError: only one element tensors can be converted to Python scalars
>>> torch.tensor([1,2]).item()
TypeError: a Tensor with 2 elements cannot be converted to Scalar
>>> float(torch.tensor(1.5j))
ValueError: value cannot be converted to type double without overflow
```

Tested by:
 -  `test_errors_masked_fill` and `test_errors_item` in `test_ops` 
 -  `test_interpolate_undefined_behavior_casting` in `test_nn.py`
 -  `test__complex__should_not_work` in `numpy_tests/core/test_multiarray.py` 
 -  `test_simple_scalar_cast` and `test_randperm` in `test_tensor_creation_ops.py`
 - `test_pointwise_op_with_tensor_of_scalarlist_overload` in `test_foreach.py`
 -  [`aten/src/ATen/test/scalar_test.cpp`](https://github.com/pytorch/pytorch/blob/261cae793a19d91fda0d44d2c2bd8cd7e9a0b93f/aten/src/ATen/test/scalar_test.cpp#L58)

Fixes https://github.com/pytorch/pytorch/issues/110605
`pytest test/dynamo -v ` fails locally
Labels: high priority, triage review, oncall: pt2

This fails locally for me (and for @voznesenskym, so it's probably not just us). We're not sure why CI is green. The number of failures is deterministic, I get between 10-150 each time I run this.

Possibly related:
- https://github.com/pytorch/pytorch/issues/103213

Example failures:
![image](https://github.com/pytorch/pytorch/assets/5652049/c76d8566-87e8-4e3c-a6ea-5d57d4ec1bd6)



cc @ezyang @gchanan @kadeng @msaroufim @wconstab @bdhirsh @anijain2305
[discussion] Have PyTorch functions support python scalars (like NumPy) + introduce convenience constants like `torch.pi` and `torch.e`
Labels: triaged, module: numpy, module: python frontend

### 🚀 The feature, motivation and pitch

OP: https://github.com/pytorch/pytorch/pull/110351#issuecomment-1742016370

As an example, it would be nice to have torch.sqrt(python scalar) -> python scalar without having to dispatch between torch.sqrt and math.sqrt and to enable a bit more polymorphic code

Another idea is to also have `torch.pi` and other constants (like NumPy), in order to avoid importing numpy or math in order to get these constants.

Please close this if it's duplicate. I tried to search for similar previous discussions, but the keywords are a bit too generic :(

For torch.sqrt specifically, the polymorphic equivalent currently exists: `x ** 0.5` which works both for tensor and python scalar inputs. But it is useful to have this behavior for many (at least simple) functions like torch.exp and so forth

### Alternatives

_No response_

### Additional context

_No response_

cc @mruberry @rgommers @albanD
[aimp][pt2] allow FakeTensor to find non-meta common devices
Labels: fb-exported, topic: not user facing, ciflow/inductor

Summary: In some cases, we have an inference model with embedding weights stored on the meta device in order to allow low-memory transformations on the model during publishing. Other parameters are on the cpu device. When tracing with PT2, this causes an error like "Unhandled FakeTensor Device Propagation for aten.mm.default, found two different devices meta, cpu". In this case, we know the parameters placed on the meta device are not final, so we want to ignore them when finding a common FakeTensor device.

Differential Revision: D49654689
[CUDA] Errors when building with cuda 12.2 
Labels: module: sparse, module: build, module: cuda, triaged

# Summary

While attempting to build PyTorch from source with Cudatoolkit 12-2 I ran into a number of issuess:

``` Shell
In file included from /home/drisspg/meta/pytorch/aten/src/ATen/cuda/CUDASparseDescriptors.cpp:4:
/home/drisspg/meta/pytorch/aten/src/ATen/cuda/CUDASparseDescriptors.h:101:46: error: 'cusparseDestroyBsrsv2Info' is deprecated: The routine will be removed in the next major release [-Werror,-Wdeprecated-declarations]
    : public CuSparseDescriptor<bsrsv2Info, &cusparseDestroyBsrsv2Info> {
                                             ^
/usr/local/cuda-12.2/include/cusparse.h:468:1: note: 'cusparseDestroyBsrsv2Info' has been explicitly marked deprecated here
CUSPARSE_DEPRECATED
^
/usr/local/cuda-12.2/include/cusparse.h:108:12: note: expanded from macro 'CUSPARSE_DEPRECATED'
         [[deprecated("The routine will be removed in the next major release")]]
```

This is confirmed here: https://docs.nvidia.com/cuda/cusparse/#cusparsedestroybsrsv2info-deprecated
Interestingly they don't provide a path forward.

### Potential issues with Symbool
``` Shell
en/src/ATen/cuda/cub-RadixSortKeys.cu.o
/home/drisspg/meta/pytorch/c10/core/SymBool.h:66:8: error: no viable conversion from returned value of type 'optional<typename std::decay<const bool &>::type>' to function return type 'optional<bool>'
return c10::make_optional(data_);
```

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer @malfet @seemethere @ptrblck
torch.compile CPU backend is slower than eager for several transcendental functions
Labels: triaged, module: inductor, module: cpu inductor

I tried a handful of somewhat important transcendentals (sigmoid, tanh, log, logit), and they all appear to be slower than eager.  On my 96c/192t Intel(R) Xeon(R) Platinum 8339HC CPU:

```
$ OMP_NUM_THREADS=1 numactl -C 94 python bench.py
log
eager:    363.3282147347927
compiled: 1091.864187270403
sigmoid
eager:    756.7794565111399
compiled: 872.3638635128736
logit
eager:    731.156948953867
compiled: 1594.5508629083633
tanh
eager:    475.9031366556883
compiled: 867.8419776260853

$ python bench.py
log
eager:    28.686294332146645
compiled: 106.62426799535751
sigmoid
eager:    91.70412458479404
compiled: 108.63539576530457
logit
eager:    25.92829428613186
compiled: 88.3821751922369
tanh
eager:    28.41925248503685
compiled: 113.94776217639446
```


```
import time

import torch

def do_bench(fn, iters=1000):
    fn()  # warmup
    s = time.perf_counter()
    for _ in range(iters):
        fn()
    e = time.perf_counter()

    return (e - s) /  iters * 1e6

x = torch.clamp(torch.rand(1 << 20), 1e-6, 1 - 1e-6)  # good range for log, logit

for fn in [torch.log,
           torch.sigmoid,
           torch.logit,
           torch.tanh,
           ]:
    cfn = torch.compile(fn)
    print(fn.__name__)

    eager = do_bench(lambda: fn(x))
    compiled = do_bench(lambda: cfn(x))

    print(f"eager:    {eager}")
    print(f"compiled: {compiled}")
```

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
DISABLED test_type_promotion__foreach_sub (__main__.ForeachTests)
Labels: triaged, module: flaky-tests, skipped, module: inductor

Platforms: linux, slow

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_type_promotion__foreach_sub&suite=ForeachTests) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/17422466410).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 6 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_type_promotion__foreach_sub`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `inductor/test_foreach.py`

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
[skip ci] voz/fsdp_autograd3 tracker PR
Labels: release notes: mps, ciflow/mps, module: inductor, module: dynamo, ciflow/inductor

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
ValueError issued instead of TypeError when tensor is cast to a scalar
Labels: module: error checking, triaged, module: numpy

### 🐛 Describe the bug

When using `math.isfinite` with a tensor a wrong `ValueError` is issued instead of a `TypeError`. This creates bugs (see #109819). Fixing this issue will fix the other one as well.

```python
>>> import math, torch
>>> a = torch.randn(5)
>>> math.isfinite(a)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[3], line 1
----> 1 math.isfinite(a)

ValueError: only one element tensors can be converted to Python scalars

>>> math.isfinite(a.numpy())
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[4], line 1
----> 1 math.isfinite(a.numpy())

TypeError: only size-1 arrays can be converted to Python scalars
```

### Versions

`PyTorch version: 2.0.1`

cc @malfet @mruberry @rgommers
AOTAutograd logging: log autograd graphs
Labels: module: logging, triaged, module: aotdispatch

### 🐛 Describe the bug

For debugging purposes, it would be helpful to have something like this:

```
diff --git a/torch/_functorch/aot_autograd.py b/torch/_functorch/aot_autograd.py
index 84a176222ef..b094952890e 100644
--- a/torch/_functorch/aot_autograd.py
+++ b/torch/_functorch/aot_autograd.py
@@ -1313,6 +1313,9 @@ def create_joint(
         backward_out = []
         # Call the backwards pass
         if grad_primals:
+            from torchviz import make_dot
+            for out in needed_outs:
+                log.warning("DOT GRAPH\n%s", make_dot(out))
             with fx_traceback.preserve_node_meta():
                 # for full graph export, we always export a joint graph where we assume no tangents are needed.
                 if aot_config.no_tangents:
```

cc @bdhirsh

### Versions

main
[torch.compile] Multiple set operations don't work
Labels: good first issue, triaged, oncall: pt2, module: dynamo, release notes: dynamo

### 🐛 Describe the bug

Originally reported by @fmassa 

Examples:
```py
@torch.compile(backend="eager", fullgraph=True)
def f(x):
    y = set({1, 2, 3})
    if 1 in y:
        return x
    return x - 1

f(x)
x = torch.randn(3)
```
```py
y = set({1, 2, 3})
@torch.compile(backend="eager", fullgraph=True)
def f(x):
     if x in y:
         return 1
     return 0

x = torch.randn(3)
f(x)
```

### Error logs

_No response_

### Minified repro

_No response_

### Versions

main

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Multiprocessing takes forever after on .get()  with mp.Queue() (Possible Deadlock)
Labels: needs reproduction, module: multiprocessing, triaged

### 🐛 Bug

This bug came up when i decided to Train ColBERT on a custom Dataset, but it was taking Forever, so I tried diagnosing the problem, seems that it uses torch.multiprocessing to divide tasks, but whenever a Task Queue is formed, the code gets stuck on the get() method

```Python
#Sample code to reproduce the Problem
import torch
import torch.multiprocessing as mp
try:
    mp.set_start_method('spawn', force=True)
except RuntimeError:
    print('Hello')
return_value_queue = mp.Queue()
#return_values = sorted([return_value_queue.get() for _ in all_procs])  #The Code gets stuck here
print(return_value_queue.get()) #To Reproduce
```

### Versions

torch version = 1.13.1+cu117

cc @VitalyFedyunin
DISABLED test_cond_with_quantization (__main__.MiscTests)
Labels: oncall: quantization, triaged, module: flaky-tests, skipped, module: unknown

Platforms: linux, mac, macos

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_cond_with_quantization&suite=MiscTests) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/17411684051).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 2 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_cond_with_quantization`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `dynamo/test_misc.py`

ResponseTimeoutError: Response timeout for 5000ms, GET https://raw.githubusercontent.com/pytorch/pytorch/main/test/dynamo/test_misc.py 200 (connected: true, keepalive socket: false, socketHandledRequests: 1, socketHandledResponses: 1)
headers: {"connection":"keep-alive","content-length":"246923","cache-control":"max-age=300","content-security-policy":"default-src 'none'; style-src 'unsafe-inline'; sandbox","content-type":"text/plain; charset=utf-8","etag":"\"c7c8187891006854d573a956b6ecb352949216e485aedde4223e6a7001644ccb\"","strict-transport-security":"max-age=31536000","x-content-type-options":"nosniff","x-frame-options":"deny","x-xss-protection":"1; mode=block","x-github-request-id":"EA54:73BB:2687EB:2FD366:651E844E","accept-ranges":"bytes","date":"Thu, 05 Oct 2023 09:39:27 GMT","via":"1.1 varnish","x-served-by":"cache-sjc10070-SJC","x-cache":"MISS","x-cache-hits":"0","x-timer":"S1696498767.228529,VS0,VE228","vary":"Authorization,Accept-Encoding,Origin","access-control-allow-origin":"*","cross-origin-resource-policy":"cross-origin","x-fastly-request-id":"fedadda00189e53d6e9e2f621f2c2e1b18d32c76","expires":"Thu, 05 Oct 2023 09:44:27 GMT","source-age":"0"}

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel
wip - Hook new guard system
Labels: module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110590
* #110735
* #110589
* #110265
* #108839



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
[dynamo][guard-refactor] TypeGuardAccessor
Labels: module: dynamo

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #110590
* #110735
* __->__ #110589
* #110265
* #108839



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
libtorch.so: error adding symbols: file in wrong format
Labels: needs reproduction, module: build, triaged

### 🐛 Describe the bug

Ubuntu 22.04.2 LTS
Architecture aarch64
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
cmake version 3.22.1
libtorch version 1.9.1

When i tried to build libtorch in the above environment, i get error like this

[ 95%] Linking CXX shared library libstt_sdk.so
/usr/bin/ld: ../third_party/libtorch_cpu/lib/libtorch.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status

can you tell me what happened?

### Versions

Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.2 LTS (aarch64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.19.0-1025-aws-aarch64-with-glibc2.35
Is CUDA available: N/A
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

CPU:
Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          2
On-line CPU(s) list:             0,1
Vendor ID:                       ARM
Model name:                      Neoverse-N1
Model:                           1
Thread(s) per core:              1
Core(s) per socket:              2
Socket(s):                       1
Stepping:                        r3p1
BogoMIPS:                        243.75
Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
L1d cache:                       128 KiB (2 instances)
L1i cache:                       128 KiB (2 instances)
L2 cache:                        2 MiB (2 instances)
L3 cache:                        32 MiB (1 instance)
NUMA node(s):                    1
NUMA node0 CPU(s):               0,1
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; CSV2, BHB
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.0
[conda] Could not collect

cc @malfet @seemethere
Switch eigen to GitHub mirror
Labels: topic: not user facing, test-config/default

I found this mirror https://github.com/eigen-mirror/eigen/tree/master that we can use.  Should we make the switch?  One less flaky source is better than twos, i.e. https://hud.pytorch.org/pr/pytorch/pytorch/110530#17400038975

Please let me know if there are any concerns as the gitlab one is still considered official.
Support de-functionalizing _c10d_functional.all_reduce in AOTInductor
Labels: module: inductor, ciflow/inductor, module: export

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110577
* #110570

Differential Revision: [D49939256](https://our.internmc.facebook.com/intern/diff/D49939256/)

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @avikchaudhuri @gmagogsfm
Unify torch.SymInt and torch.types.SymInt
Labels: topic: not user facing, module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110573
* #110572
* #110112



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
Enable more mypy import following for torch/_inductor/
Labels: topic: not user facing, module: inductor, module: dynamo, ciflow/inductor, module: export

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110572

I've excluded importing several modules & added a bunch of
type-ignore comments to make this more tractable. I think it's valuable
to turn this on sooner rather than later given that it greatly improves
our coverage; `follow_imports=skip` actually makes a lot of things into
Any types when we have more precise type info.

I also set us up to ignore errors from torch modules that are already
handled through the MYPY lintrunner command. This was not done
just for perf reasons: `mypy-nofollow.ini` is actually stricter in some
ways than the `mypy.ini`, so MYPYNOFOLLOW was raising errors in
the rest of the code.


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @avikchaudhuri @gmagogsfm
Native c10d_functional ops
Labels: release notes: distributed (c10d)

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110570

This PR introduces a native version of c10d_functional ops. The main goal is to add collective support in AOTInductor and allow collective ops to work in multi-threaded native runtimes.

The native version also incorporated API improvements we wished to implement in Python c10d_functional:

- Removed `ranks` and `group_size` from collective op signatures which were proven to be redundant.
- Use tensor storage as opposed to `void*` to resolve in-flight work.

The native process group registration/resolution mechansim is only used for native c10d_functional in the PR. It will become the single source of truth in upcoming PRs.

The upcoming PRs will implement Inductor/AOTInductor support for c10d_functional, after which native c10d_functional will replace Python c10d_functional.

Differential Revision: [D49932255](https://our.internmc.facebook.com/intern/diff/D49932255/)
automate the full source tarball release asset (sdist)
Labels: oncall: releng, triaged

### 🚀 The feature, motivation and pitch

This release asset - pytorch-vX.Y.Z.tar.gz - appears a few weeks after an official release, like an afterthought, yet is crucial for source-based packages, because it contains all Git submodules: https://github.com/stefantalpalaru/gentoo-overlay/blob/c69c2c2be65abe76649493af8c1099fb406806e7/sci-libs/caffe2/caffe2-2.0.1-r101.ebuild#L14-L16

Please generate it automatically, using a CI workflow, and make sure it runs when a new release is created.

### Alternatives

_No response_

### Additional context

_No response_
[Inductor] ABI-fy some aten fallback kernels
Labels: module: inductor, ciflow/inductor

Fixes #ISSUE_NUMBER


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
[Optimus][pt2] Initial opportunity finder
Labels: fb-exported, module: inductor, ciflow/inductor

Summary:
Add a new fx pass to help analyze model.
In this initial version, we can find some opportunities for horizontal fusion. Will add more

Test Plan:
torch level: P832449483
aten level:
fwd P832451192
bwd P832452174

Differential Revision: D49408003




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
Call RECORD_KERNEL_FUNCTION_DTYPE
Labels: fb-exported, release notes: linalg_frontend

Summary:
Modified the .cpp, then run the tracer command to generate the rest:
cd ~/fbsource/fbcode && buck run caffe2/torch/fb/mobile/cli:cli -- --gen_model_config --model_name MultitaskPeopleSegmentation --model_version 9000 --asset_name PYTORCH_MODEL

Test Plan: CI

Differential Revision: D49926634
Create nested _sdpa_flash
Labels: module: cpu, ciflow/inductor

# Summary

Follow up PR on #110527 and #110533

This creates a new aten function for SDPA_flash that only has a nested registration. This is because the information needed for backward is different between the tensor layouts. 


cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
[CI] Add inductor workflow for rocm
Labels: module: rocm, open source, topic: not user facing, module: inductor, ciflow/inductor

This PR is to create a separate CI job for inductor UTs on ROCm.

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
Clean way to distinguish python subclass NT vs. C++ NT
Labels: triaged, module: nestedtensor

## Issue description

We introduced a jagged layout backed NT as a python tensor subclass in #108314. Since the `NestedTensor` and `AutogradNestedTensor` dispatch keys are set for this subclass, calling `t.is_nested` will return True for both the subclass and the traditional C++-side NT. We need a clean way to distinguish the two for cases where they should be handled differently on the python side (e.g. during multiprocessing serialization).

This currently works but it's ugly:
```python
from torch.nested._internal.nested_tensor import NestedTensor
if isinstance(t, NestedTensor):
    # python subclass
    ...
elif t.is_nested:
    # C++ subclass
    ...
```

Relevant comment link: https://github.com/pytorch/pytorch/pull/110292#discussion_r1344297126

cc @cpuhrsch @bhosmer @drisspg @soulitzer
On the correctness of torch.signal.windows.cosine
Labels: triaged, module: numpy, topic: bc breaking, topic: docs

As of writing, `torch.signal.windows.cosine` is implementing the following function:

$h[n] = \sin(\frac{\pi(n+0.5)}{M}),$

where $M$ is the length of the window. (The denominator is $M+1$ when `sym=False`, but my point will be the same.)

This implementation is identical to scipy's, but it deviates from the definition I'm familiar with, which is

$h[n] = \sin(\frac{\pi n}{M-1}).$

As far as I know, the cosine window is the square root of the Hann window. If we use the current implementation, this equality is not satisfied, while the latter definition does indeed satisfy it. For what it's worth, MATLAB also follows the latter definition. I could not find an explanation for scipy's deviation.

### Solution
I suppose there are two choices:
1. Continue following scipy, but make it clear in the documentation that it deviates from the correct definition.
2. Modify the code to implement the correct definition, and perhaps mention how it deviates from scipy in the docs.

cc @mruberry @rgommers
Add bandwidth to extern kernel calc
Labels: fb-exported, ciflow/trunk, topic: not user facing, module: inductor, ciflow/inductor

Summary: - Modify the result of get_estimated_runtime() for ExternKernelSchedulerNode to count both bytes and FLOPs and return the maximum of the two.

Reviewed By: xmfan

Differential Revision: D48987490




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
[MPS] Unsupported operand type for * with complex tensors
Labels: triaged, module: complex, module: mps

### 🐛 Describe the bug

```python
import torch
import torch.nn as nn

class MyModule(nn.Module):
    def __init__(self, x, y):
        super().__init__()
        self.register_parameter("x", nn.Parameter(x))
        self.register_buffer("y", y)

x = torch.rand(3, dtype=torch.complex64)
y = torch.rand(3, dtype=torch.complex64)
model = MyModule(x, y).to(device="mps")
model.x * model.y
```
returns
```python
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[1], line 17
     15 device = torch.device("mps")
     16 model = MyModule(x, y).to(device)
---> 17 model.forward()

Cell In[1], line 11, in MyModule.forward(self)
     10 def forward(self):
---> 11     return self.x * self.y

TypeError: unsupported operand type(s) for *: 'Parameter' and 'Tensor'
```

### Versions

PyTorch version: 2.1.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.0 (arm64)
GCC version: Could not collect
Clang version: 15.0.0 (clang-1500.0.40.1)
CMake version: Could not collect
Libc version: N/A

Python version: 3.11.4 (main, Jul  5 2023, 08:40:20) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-14.0-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M2

Versions of relevant libraries:
[pip3] flake8==6.1.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.25.2
[pip3] torch==2.1.0
[conda] numpy                     1.25.2                   pypi_0    pypi
[conda] torch                     2.1.0                    pypi_0    pypi

cc @ezyang @anjali411 @dylanbespalko @mruberry @Lezcano @nikitaved @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
[dynamo] Implement set in terms of dict
Labels: open source, module: dynamo, ciflow/inductor, release notes: dynamo

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #108420
* __->__ #110524
* #111196
* #110523
* #110522

This allows to heavily simplify the implementation of set, which was
"quite unique". Now we represent a set a as a dict where all its values
are None.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
[dynamo] Simplify add_dict in preparation to refactor it with call_set
Labels: open source, module: dynamo, ciflow/inductor, release notes: dynamo

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #108420
* #110524
* #111196
* __->__ #110523
* #110522

The previous implementation had a fair amount of repeated code, and did
things like calling `add_options` where options was always empty (which
is fine, as the guards are already set within ConstDictVariable).

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
[dynamo] [easy] Move Set to dicts.py
Labels: open source, module: dynamo, ciflow/inductor, release notes: dynamo

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #108420
* #110524
* #111196
* #110523
* __->__ #110522

A set is more of a dict than a list if you ask me.
This comes before the refactor where we implement sets and dicts via the
same logic.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Torch Nested Issue With Backward Pass In Transpose
Labels: triaged, has workaround, module: nestedtensor

### 🐛 Describe the bug

When transposing a nested tensor and doing a matrix multiplication, the backward pass requires the tensor to be contiguous. I'm guessing this issue has to do with transpose causing tensors to be non-contiguous after the operation which is fine with the forward pass, but the backward pass may have a bug for nested tensors when going through the backward transpose operation.

```python
import torch
X1 = torch.nested.as_nested_tensor([
    torch.tensor([[1,2],[3,4.],[5,6]]),
    torch.tensor([[7,8],[9,10]])
    ]).requires_grad_().cuda()
X2 = torch.nested.as_nested_tensor([
    torch.tensor([[1,2],[3,4.],[5,6]]),
    torch.tensor([[7,8],[9,10]])
    ]).requires_grad_().cuda()
Y = X2.transpose(-1, -2).contiguous()@X1
L = sum([i.sum() for i in Y])
L.backward()
```


```
Traceback (most recent call last):
  File "<pyshell#51>", line 1, in <module>
    L.backward()
  File "../torch/_tensor.py", line 487, in backward
    torch.autograd.backward(
  File ".../torch/autograd/__init__.py", line 200, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: empty_like only supports contiguous memory format for Nested Tensors
```

### Versions

Collecting environment information...
PyTorch version: 2.0.1+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Pro
GCC version: (Rev6, Built by MSYS2 project) 13.1.0

Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.10 (tags/v3.10.10:aad5f6a, Feb  7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22621-SP0
Is CUDA available: True
CUDA runtime version: 11.8.89

CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3050 Ti Laptop GPU
Nvidia driver version: 537.42
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9


CurrentClockSpeed=2500


DeviceID=CPU0


Family=207


L2CacheSize=11776


L2CacheSpeed=


Manufacturer=GenuineIntel


MaxClockSpeed=2500


Name=12th Gen Intel(R) Core(TM) i9-12900HK


ProcessorType=3


Revision=

Versions of relevant libraries:
[pip3] clip-anytorch==2.5.2
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.24.1
[pip3] torch==2.0.1+cu118
[pip3] torch-geometric==2.3.1
[pip3] torchaudio==2.0.2+cu118
[pip3] torchdata==0.6.1
[pip3] torchdiffeq==0.2.3
[pip3] torchsde==0.2.5
[pip3] torchtext==0.6.0
[pip3] torchvision==0.15.2+cu118
[pip3] vector-quantize-pytorch==1.6.19
[conda] Could not collect

cc @cpuhrsch @jbschlosser @bhosmer @drisspg @soulitzer
doc modification of torch.nn.softshrink api
Labels: module: docs, module: nn, triaged

### 📚 The doc issue

Hi! I am try to built a api that calling the torch torch.nn.softshrink. or you can consider any other non linear activation function. and i want to pass the torch tensor with it's dtype. but in the documentation it doesn't given which dtype the x tensor is supports for this activation function. here is documentati of [softshrink](https://pytorch.org/docs/2.1/generated/torch.nn.Softshrink.html#torch.nn.Softshrink). can you help me by adding the supported dtype. for the input tensor in the activation function.

and in the softshrink activation. what will the constrain on the lambd value.  ?<lambd<?

### Suggest a potential alternative/fix

by adding the supported dtype to the all activation doc for the input tensor.

cc @svekars @carljparker @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
[DEMO] cached allocate across devices

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110500

This is a demo of functionality that allows the caching allocator to create
as single tensor that exists allocated across all devices. The outer stride of
the tensor steps across the allocations on different devices.

The implementation uses the expandable segment functionality to control the
mapping of memory to ensure a single stride to cross devices. For N devices,
it creates N blocks of virtual memory that are sequential. Then it performs
the same map/unmap operations in each block, but wiring memory from the
corresponding devices. We use the standard caching allocator to manage
the allocation as if it were on just one of the devices and then hack
in the extra stride that reveals that we also allocated memory on the other devices.

Bug - the tensor returned from _reveal_multiple_devices does not own the data. The original tensor needs to be kept around to keep it alive. This can probably be fixed using the right DataPtr callbacks to keep a handle on the right storage object.

    import torch

    # tensors allocated under this guard will be
    # allocated as a copy per device,
    # torch will think the tensor is allocated on
    # whatever the 'current' device is.
    with torch.cuda.memory._multi_device_allocator():
        x = torch.empty(3, 4, device='cuda')

    # use this function to get a tensor with a stride over
    # all the devices. the second argument tells torch
    # where it should run the kernels on it
    y = torch._C._reveal_multiple_devices(x, torch.device('cuda', 1))
    print(y.size()) # 2, 3, 4
    # note: no need for enable peer access!
    y.fill_(4)
    print(y)
[export] `torch.tensor(0)` should not get burned in as a constant
Labels: triaged, oncall: pt2, module: aotdispatch, module: dynamo, module: export

```
import torch

class Mod(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.init_value = 0.1

    def forward(self, x):
        return x + torch.tensor(self.init_value)

def main() -> None:
    m = Mod()
    foo = torch.export.export(m, (torch.randn((2, 3)),))
    print(foo.graph)
```

prints
```
graph():
    %_lifted_tensor_constant0 : [num_users=1] = placeholder[target=_lifted_tensor_constant0]
    %arg0_1 : [num_users=1] = placeholder[target=arg0_1]
    %lift_fresh_copy : [num_users=1] = call_function[target=torch.ops.aten.lift_fresh_copy.default](args = (%_lifted_tensor_constant0,), kwargs = {})
    %add : [num_users=1] = call_function[target=torch.ops.aten.add.Tensor](args = (%arg0_1, %lift_fresh_copy), kwargs = {})
    return (add,)
```

I think `torch.tensor(0)` should behave like the other tensor factory functions and get preserved in the graph. This most directly preserves user intent. Besides, if someone wants it to be a constant they can just fold it later.

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @avikchaudhuri @gmagogsfm
[export] Constant tensors should not get lifted to buffers
Labels: triaged, module: export

Today, if we have a constant tensor (e.g. generate by a `torch.tensor(0)` call), it will get lifted to be a buffer (https://github.com/pytorch/pytorch/blob/main/torch/_export/passes/lift_constant_tensor_pass.py).

This is potentially problematic—the state dict is a well-established abstraction in PyTorch, and mutating it due to export implementation details may be surprising for users. As a concrete example, we have many users that guard model compatibility based on state dict keys. If export adds a key based on some details of how tracing happened, those compatibility checks will break and users will be confused as to why.

Instead, we should handle tensor constants in the tensor specification. Nodes should be able to take constant tensors as inputs, and we should have a table mapping constant tensor handles to in-archive serialized blobs. The serialization/deserialization systems will need to be updated to properly load/save these tensors.

cc @avikchaudhuri @gmagogsfm
Support mutating constant attributes in export
Labels: ciflow/trunk, topic: not user facing, module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110471

In this PR, we implement simulating setattr on nn.Module and HuggingFacePretrainedConfig objets while not changing the original object. For nn.Module, we do:

1. If the nn.Module has custom setattr, we inline it
2. If the nn.Module has default setattr, we inline it in torch.compile and simulate this behavior in export only. The reason is inlining setattr will cause a graph break (https://github.com/pytorch/pytorch/blame/f2d7faf4ba92a6ed43890775ca6ca174ddbf99ea/torch/nn/modules/module.py#L1755) This is fine because this simulation only happens for constant attributes so it is pretty easy to keep track.

For HuggingFacePreTrained, we do:
1. Inline setattr for torch.compile 
2. Simulate the setattr in torch.export. This seems bit problematic for attributes that are stored inside https://github.com/huggingface/transformers/blob/2f3ea08a077ba3133fa8a604b22436cad250b055/src/transformers/configuration_utils.py#L254 but i am not really sure how likely these occur in practice. If the issue rises, we can probably revisit this issue? 

In the future, I think we should create ExportInstructionTranslator that knows how to deal with mutations. 

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng
Upgrade CI to ROCm5.7
Labels: module: rocm, open source, ciflow/trunk, topic: not user facing, ciflow/periodic, keep-going

This PR is to upgrade CI to ROCm5.7

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @zstreet87
Custom tensor attributes not preserved with registered functions
Labels: triaged, module: custom-operators, module: library

In the following snippet
```.py
import torch

class Fn(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        return x/x.coef

    @staticmethod
    def backward(ctx):
        pass


def my_fn_impl(x):
    x.coef = 2.
    print("COEF", x.coef)
    return Fn.apply(x)

from torch.library import Library

ns = 'capturable'
lib = Library(ns, "DEF")
fwd_name = lib.define(f"capturable::myop(Tensor x) -> Tensor")
lib.impl(fwd_name, my_fn_impl, 'CPU')

x = torch.randn(2, 3)
print(my_fn_impl(x))  #fine
print(torch.ops.capturable.myop(x)) # COEF is printed, but autograd fn says Tensor has no attribute coef
```
when `my_fn_impl` is registered on CPU key, an attribute coef that's attached to tensor in my_fn_impl is stripped by Autograd function
According to @zou3519 
>For some reason, HermeticPyObjectTLS is set, which disables PyObject preservation (the thing that would make .coef show up on the Tensor)


cc @anjali411
[fuzzing result][fuzz_torch_jit_lite_interpreter] read-heap-buffer-overflow-far-from-bounds (size 4) in c10::IValue::IValue()
Labels: fb-exported, release notes: mobile

Summary: This diff fixes an OOB read found by fuzzing in torch/../jit/mobile

Test Plan:
CI and
```
arc lionhead crash reproduce 853835926354224
```
doesn't crash anymore.

Differential Revision: D49537377
[pytorch][PR][inductor] Change log to debug for Optimus
Labels: fb-exported, module: inductor, ciflow/inductor

Summary: The log breaks one of ads-model export flows, and we change the log to debug

Test Plan: see details in D49710166

Differential Revision: D49844303




cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler
`test_pytorch_onnx_onnxruntime_cuda.py` is not run in CI
Labels: module: onnx, module: cuda, module: tests, triaged

Currently this file fails at import time because of this import:

https://github.com/pytorch/pytorch/blob/4069d1de59bb5cd60c5364acd2835fbf9c6e601c/test/onnx/test_pytorch_onnx_onnxruntime_cuda.py#L17-L21

which imports the `*_OPSET_VERSION` constants from the wrong file
https://github.com/pytorch/pytorch/blob/4069d1de59bb5cd60c5364acd2835fbf9c6e601c/test/onnx/onnx_test_common.py#L436-L439

I am incidentally fixing the import issue in https://github.com/pytorch/pytorch/pull/110310#discussion_r1342669352 but the fact it fails at all means it must not be run in CI.

cc @ptrblck @mruberry @ZainRizvi
[C10D] Split watchdog into CUDA and non-cuda calls.
Labels: open source, release notes: distributed (c10d)

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #110449

A common problem with the current watchdog is that it hangs
trying to abort NCCL or destroy events, those take CUDA locks
that might be held by the main thread that is starved due to a stuck collective.

We do it by introducing a second thread called the monitor thread.

Here some details of this design.

We introduce started_ and _completed_ atomics that track progress.

We set them as part of isCompleted and isStarted. Note that we can't
detect that a collective started if we're not recording those events.

Important visibility rule. We set started_ then completed_ so we must
read them in the opposite order.

Beyond that, we use a pair of queues so collectives can flow between them
as they complete. The new queue is used for the sole purpose of carrying
the destruction on the monitor thread.
Explore Hybrid (CPU+GPU) Graphs in Scalar parameters
Labels: oncall: pt2, module: inductor

### 🚀 The feature, motivation and pitch

As observed across https://github.com/pytorch/pytorch/issues/107006#issuecomment-1741841283, https://github.com/pytorch/pytorch/pull/110345#discussion_r1344183030
Inductor currently is unable to reason properly about graphs that have tensors spread across CPU and GPU.

This forces us to coerce to GPU the scalar computations and corresponding scalar inputs in Adam-like and Adagrad/Adamax. Inductor then launches a single small prelude kernel. This choice may be suboptimal as this small kernel launch is unlikely to saturate the GPU. 

https://github.com/pytorch/pytorch/blob/ff96f6d04f062e660224e7ba1f00da867d25d0de/test/inductor/test_compiled_optimizers.py#L128

One possible solution would be to have inductor generate CPU kernels for the scalar part of the computation, and GPU kernels for the main optimizer step.

### Alternatives

It is unclear if this incurs a large enough cost to justify the effort. 

It is also unclear if saturating the GPU with this kernel is necessary - if multiple chunks/groups are launched, then a small kernel can be running concurrently alongside a big kernel launch.

@mlazos @janeyx99 for advice on whether this direction does deserve more investigation.

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler