gopikrsmscs/torch-issues huggingface dataset:
Reuse autograd.grad graph for rapid, repeated gradient calculation
Labels: feature, module: autograd, triaged, module: cuda graphs

### 🚀 The feature, motivation and pitch

In Scientific Machine Learning (SciML), there is often the need to run a function forward and obtain gradient wrt inputs. This is not to tune nn parameters but for solving large systems of equations. This task happens at every time step with fixed input shapes and is often the largest time consumer. The workflow can be written concisely as:

```
for i in range(nt):
  torch.set_grad_enabled(True)
  gg = G(x, p, aux)   
  #x, p are tensors.  They may have very many elements, but fixed shapes over time steps. 
  dGdx = torch.autograd.grad(gg,x,grad_outputs=v) ## the jacobian dGdx is often sparse
  torch.set_grad_enabled(False)
  #use dGdx in some calculations, without need for Automatic Differentiation
```

through some optimization with v we were already making torch.autograd.grad parallel for two dimensions, but it is still the majority of the time. I suspect it is due to repeatedly calling the autograd engine, but the graph is fixed for every time! if we can capture the graph for the two main lines "gg=G" and "autograd.grad", only change the data (x,y) and reuse the graph every time step, it can go much faster:

```
# somehow initialize the graph capturedGraph for both G and autograd.grad
for i in range(nt):
  dGdx = capturedGraph(x,p)
  # use dGdx in some calculations
```

Motivation:
This is a core task for almost all engineering domains. It's a super big deal!! For example, read here.
https://www.mathworks.com/help/optim/ug/nonlinear-equations-with-analytic-jacobian.html


### Alternatives

I've been reading torch.fx to no avail.

### Additional context

_No response_

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @mcarilli
Inductor guards are not propagated to Dynamo with dynamic shapes
Labels: triaged, oncall: pt2, module: dynamic shapes

### 🐛 Describe the bug

```
@dataclasses.dataclass
class PositiveGuard:
    """
    An expression we should check for > 0
    Guards are currently not checked.  Plan to add this later.
    """

    expr: Expr
```

This means we may incorrectly reuse inductor kernels for sizes they are not valid for.

### Versions

master

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh
Better error message when trying to run fp16 weights on CPU
Labels: good first issue, module: error checking, triaged

### 🚀 The feature, motivation and pitch

Hey :wave: from the Hugging Face Open-Source team,

We're seeing the following issue over and over again across libraries
```
RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'
```

or:

```
RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'
```

E.g.: https://github.com/runwayml/stable-diffusion/issues/23

The problem here is that a PyTorch model has been converted to fp16 and the user tried to run it on CPU, e.g. the following:

```py
from torch import nn
import torch

linear = nn.Linear(2,2, dtype=torch.float16)

tensor = torch.ones((2,), dtype=torch.float16)

linear(tensor)
```

yields:

```
"addmm_impl_cpu_" not implemented for 'Half'
```

Could we maybe catch such errors in the forward of https://pytorch.org/docs/stable/_modules/torch/nn/modules/module.html#Module

and return a simpler error message that just says "Float16 cannot be run on CPU"?

### Alternatives

_No response_

### Additional context

_No response_

cc @malfet
[MPS] F.conv1d and F.conv2d produce incorrect gradients when minibatch >= 2^16
Labels: triaged, module: mps

### 🐛 Describe the bug

Both `F.conv1d` and `F.conv2d` (and `torch.nn.Conv1d` and `torch.nn.Conv2d`) produce incorrect gradients
when the minibatch size is 65_536 (=2^16) or greater. In the example below, the bottom two assertions fail.
```python
import torch

def test_grad(minibatch, dimensions=1):
    if dimensions == 1:
        view = (-1, 1, 1)
        op = torch.nn.functional.conv1d
    elif dimensions == 2:
        view = (-1, 1, 1, 1)
        op = torch.nn.functional.conv2d

    # create identical input tensors
    input_cpu = torch.ones(minibatch)
    input_mps = copy.deepcopy(input_cpu).to("mps")

    # create identical weight tensors
    weight_cpu = torch.ones(1)
    weight_mps = copy.deepcopy(weight_cpu).to("mps")
    weight_cpu.requires_grad_()
    weight_mps.requires_grad_()

    # get output
    out_cpu = op(input_cpu.view(view), weight_cpu.view(view)).mean()
    out_mps = op(input_mps.view(view), weight_mps.view(view)).mean()
    out_cpu.backward()
    out_mps.backward()

    # compare CPU and MPS outputs
    assert torch.allclose(out_cpu, out_mps.to("cpu"))
    assert torch.allclose(weight_cpu.grad, weight_mps.grad.to("cpu")) # fails with minibatch >= 65_536

test_grad(65_535, dimensions=1)
test_grad(65_535, dimensions=2)
test_grad(65_536, dimensions=1) # second assertion fails
test_grad(65_536, dimensions=2) # second assertion fails
```

The output might be non-deterministic, but in the testing below the error rate with `minibatch=65_535` is 6%,
whereas the error rate with `minibatch=65_536` is 100%.
```python
n_iter = 100
errors_65_535 = 0
errors_65_536 = 0

for _ in range(n_iter):
    try:
        test_grad(65_535)
    except AssertionError:
        errors_65_535 += 1
    
    try:
        test_grad(65_536)
    except AssertionError:
        errors_65_536 += 1

print(f"65_535 AssertionError rate = {errors_65_535 / n_iter : .0%}")
print(f"65_536 AssertionError rate = {errors_65_536 / n_iter : .0%}")
```

### Versions

PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.2.1 (arm64)
GCC version: Could not collect
Clang version: 15.0.7
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.6 (main, Aug 29 2022, 10:06:59) [Clang 13.1.6 (clang-1316.0.21.2.5)] (64-bit runtime)
Python platform: macOS-13.2.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Max

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.24.2
[pip3] torch==1.13.1
[pip3] torchtext==0.13.1
[conda] Could not collect

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
[Dynamo] HuggingFace transformers configuration_utils graph break workaround
Labels: triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

HF worked around dynamo problem here https://github.com/huggingface/transformers/pull/21648/files#r1107482201

This should not be necessary, we should fix our code.

### Versions

master

cc @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @soumith
dynamo + dict subclass + tensor instance check: NotImplementedError
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

Torchdynamo fails to understand the `self.__dict__` type when it's `isinstance` checked with `torch.Tensor`

Note that `dict(self.__dict__)` does work as a workaround

### Error logs

```python
Traceback (most recent call last):
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile
    out_code = transform_code_object(code, transform)
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 530, in transform_code_object
    transformations(instructions, code_options)
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform
    tracer.run()
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1862, in run
    super().run()
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 619, in run
    and self.step()
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 583, in step
    getattr(self, inst.opname)(inst)
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 349, in wrapper
    return inner_fn(self, inst)
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1014, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 517, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 561, in call_function
    result = handler(tx, *args, **kwargs)
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 799, in call_isinstance
    arg_type = arg.python_type()
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/torch/_dynamo/variables/base.py", line 146, in python_type
    raise NotImplementedError(f"{self} has no type")
NotImplementedError: GetAttrVariable(UserDefinedObjectVariable(MyDict), __dict__) has no type

from user code:
   File "/home/carmocca/git/lightning/kk4.py", line 7, in foo
    print(isinstance(self.__dict__, torch.Tensor))
```

### Minified repro

```python
import torch

class MyDict(dict):
    def foo(self):
        print(isinstance(self.__dict__, torch.Tensor))

def x():
    d = MyDict()
    d.foo()

c_x = torch.compile(x)
c_x()
```

### Versions

`torch==2.1.0.dev20230302+cpu`

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
`ld: error: unknown argument '-force_load'` when linking libtorch on Android
Labels: module: build, oncall: mobile

### 🐛 Describe the bug

What I did:
1. Built PyTorch 1.13.1 from source by running `scripts/build_android.sh -DUSE_LITE_INTERPRETER_PROFILER=OFF` for arm64-v8a (it builds as static libraries this way).
2. Linked against the resulting installation in CMake roughly like so:
```cmake
find_package(Torch REQUIRED PATHS "${torch_SOURCE_DIR}/share/cmake/Torch")
target_link_libraries(my_lib PRIVATE ${TORCH_LIBRARIES})
target_include_directories(my_lib PRIVATE ${TORCH_INCLUDE_DIRS})
```
3. Tried to build my Android app that uses `my_lib` though JNI.

During the build I got `ld: error: unknown argument '-force_load'` -- this flag is set [here](https://github.com/pytorch/pytorch/blob/12ab4f08b77e1f2685ff15e7cd2a07d15ed80a43/cmake/TorchConfig.cmake.in#L32) in `TorchConfig.cmake`.

Full error message:
```
FAILED: <project dir>/build/intermediates/cxx/Debug/4m2x1c2x/obj/arm64-v8a/libmy_lib.so 
cmd.exe /C "cd . && C:\Users\<my username>\AppData\Local\Android\Sdk\ndk\23.1.7779620\toolchains\llvm\prebuilt\windows-x86_64\bin\clang++.exe --target=aarch64-none-linux-android21 --sysroot=C:/Users/<my username>/AppData/Local/Android/Sdk/ndk/23.1.7779620/toolchains/llvm/prebuilt/windows-x86_64/sysroot -fPIC -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fexceptions -frtti -stdlib=libc++ -g  -fno-limit-debug-info  -Wl,--build-id=sha1 -Wl,--no-rosegment -Wl,--fatal-warnings -Qunused-arguments -Wl,--no-undefined -shared -Wl,-soname,libmy_lib.so -o <other .o and .a/.so files>  -pthread  -ldl  -Wl,-force_load <project dir>/.cxx/Debug/4m2x1c2x/arm64-v8a/_deps/torch-src/lib/libtorch.a  -Wl,-force_load <project dir>/.cxx/Debug/4m2x1c2x/arm64-v8a/_deps/torch-src/lib/libtorch_cpu.a  _deps/torch-src/lib/libc10.a  _deps/torch-src/lib/libnnpack.a  _deps/torch-src/lib/libpytorch_qnnpack.a  _deps/torch-src/lib/libXNNPACK.a  _deps/torch-src/lib/libcpuinfo.a  _deps/torch-src/lib/libclog.a  _deps/torch-src/lib/libpthreadpool.a -static-libstdc++ -latomic -lm && cd ."
ld: error: unknown argument '-force_load'
ld: error: unknown argument '-force_load'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
```

### Versions

- Windows 11 x64
- Android NDK 23.1.7779620 (also tried 25.2.9519653)
- CMake 3.22.1
- PyTorch built from sources, `v1.13.1` git tag as described above

cc @malfet @seemethere
[torchdistx] Future of the large model initialization
Labels: module: nn, triaged, ezyang's list, module: meta tensors, module: fsdp

For deep learning, when the model is large, model creation and initialization on host device will require tremendous time and sometimes causes host OOM. The existing [torchdistx](https://github.com/pytorch/torchdistx) package resolved this issue efficiently by introducing the `deferred_init` API which will first create model with fake tensor then materialized those tensor after the model is already partitioned, either by FSDP or other model parallelism methods.

However, it seems there is no active development in the torchdistx package, while the current FSDP implementation still depends on it for `deferred_init`. I'm wondering if there is any plan to merge the `deferred_init` API into the native Pytorch, and if no, what will be the native solution for large model initialization?

This issue is associated with this [PR](https://github.com/pytorch/xla/pull/4664) in the Pytorch/XLA repo.

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @ezyang @eellison @bdhirsh @zhaojuanmao @mrshenli @rohan-varma @awgu @fegin @penguinwu @saketh-are @soumith
mps bug: failed assertion `[MPSNDArrayDescriptor sliceDimension:withSubrange:] error: subRange.start (6) is not less than length of dimension[0] (6)'
Labels: triaged, module: regression, module: viewing and reshaping, module: mps

### 🐛 Describe the bug

I got a bug that looks similar to #95883 that is just closed. Not sure if it's the same. I'm running on the latest nightly version.

Code:
```
import torch

t = torch.ones((2,6,), device='mps')[1].reshape(2,3)
print("No error yet")
t = t + 1
```

Output:
```
No error yet
/AppleInternal/Library/BuildRoots/a0876c02-1788-11ed-b9c4-96898e02b808/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSCore/Types/MPSNDArray.mm:82: failed assertion `[MPSNDArrayDescriptor sliceDimension:withSubrange:] error: subRange.start (6) is not less than length of dimension[0] (6)'
zsh: abort      python test/torch_mps_test.py
```

Interesting, if I change the code to
```
import torch

t = torch.ones((2,6,), device='mps')[1].reshape(3,2)
print("No error yet")
t = t + 1
```
then there is no error.

### Versions

```
Collecting environment information...
PyTorch version: 2.1.0.dev20230306
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.6.3 (arm64)
GCC version: Could not collect
Clang version: 12.0.0 (clang-1200.0.32.28)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.16 (main, Mar  1 2023, 12:19:04)  [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-12.6.3-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==1.24.2
[pip3] torch==2.1.0.dev20230306
[pip3] torchdiffeq==0.2.3
[conda] numpy                     1.24.2                   pypi_0    pypi
[conda] torch                     2.1.0.dev20230306          pypi_0    pypi
[conda] torchdiffeq               0.2.3                    pypi_0    pypi
```

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
`torch.utils.checkpoint` should avoid updating BatchNorm statistics twice
Labels: module: checkpoint, triaged

### 🚀 The feature, motivation and pitch

`torch.utils.checkpoint` provides two checkpointing flavors
1. Reentrant
2. non-reentrant 

In both cases if the wrapped module contains `BatchNorm` layers, the batch norm will be updated twice because the forward pass has to be ran twice. This behavior makes reproducing experiment results between checkpointed and un-checkpointed runs harder. 
To repro the problem, one can run https://github.com/facebookresearch/fairscale/blob/main/tests/nn/checkpoint/test_checkpoint_activations_norm.py test without the fairscale wrapper. 

### Alternatives

Fairscale provides a patched Reentrant flavour of activation checkpoint wrapper, that uses 2 hooks to avoid the second forward pass updating the batchnorm stats. 
But currently we don't have a clean API to support this behavior in the non-reentrant case. I can think of one method to achieve this effect is to make the following modification in the non-reentrant pass, but not sure if this will work. UPDATE: it doesn't seemed to work. 
1. mark the `running_mean` and `running_var` tensor inside BatchNorm with a special attribute.
2. detect that special attribute during [pack](https://github.com/pytorch/pytorch/blob/master/torch/utils/checkpoint.py#L378), and return the normal tensor instead of the holder object
3. during `unpack`, if a tensor is passed in as argument, return the tensor directly instead of loading it from `storage`
4. during `unpack`, run the forward pass with `track_running_stats=False`, so the `running_mean` and `running_var` does not get updated. 

### Additional context

_No response_
make_fx tracing with dynamic shapes should also disable_slice_optimization
Labels: triaged, module: aotdispatch

### 🐛 Describe the bug

This will help remove some guards related to slicing.

Test case:

```
diff --git a/test/test_proxy_tensor.py b/test/test_proxy_tensor.py
index d1f5de669b3..d8f2d6f1c0b 100644
--- a/test/test_proxy_tensor.py
+++ b/test/test_proxy_tensor.py
@@ -970,6 +970,16 @@ def forward(self, crop_camera_1, mask_1):
     index_put_ = torch.ops.aten.index_put_.default(crop_camera_1, [mask_1], view_2);  crop_camera_1 = mask_1 = view_2 = None
     return None""")
 
+    def test_unbacked_slice(self):
+        def f(x, m):
+            x = x[m]
+            return x[slice(None, None, None), slice(None, None, None), slice(None, 2, None)]
+
+        make_fx(f, tracing_mode="symbolic")(
+            torch.randn((12, 3, 3)),
+            torch.randint(0, 2, (12,), dtype=torch.bool)
+        )
+
     @unittest.skipIf(not USE_TORCHVISION, "test requires torchvision")
     def test_unbacked_batch_resnet(self):
         mod = torchvision.models.resnet18()
```

### Versions

master
Static size boolean masking
Labels: triaged, module: advanced indexing

### 🐛 Describe the bug

A long standing request is https://github.com/pytorch/pytorch/issues/62320 ; Executorch team has agreed to implement it.

Once this is implemented, we can also get static size boolean masking to work too. The easiest way is to convert the boolean mask into an index tensor. You can use the meta implementation for indexing to do this:

```
@register_meta(aten.index.Tensor)
def meta_index_Tensor(self, indices):
    result: List[Optional[Tensor]] = []
    for i, index in enumerate(indices):
        if index is not None:
            check(
                index.dtype in [torch.long, torch.int, torch.int8, torch.bool],
                lambda: "tensors used as indices must be long, int, byte or bool tensors",
            )
            if index.dtype in [torch.int8, torch.bool]:
                nonzero = index.nonzero()
                k = len(result)
                check(
                    k + index.ndim <= self.ndim,
                    lambda: f"too many indices for tensor of dimension {self.ndim}",
                    IndexError,
                )
                for j in range(index.ndim):
                    check(
                        index.shape[j] == self.shape[k + j],
                        lambda: f"The shape of the mask {index.shape} at index {i} "
                        f"does not match the shape of the indexed tensor {self.shape} at index {k + j}",
                        IndexError,
                    )
                    result.append(nonzero.select(1, j))
            else:
                result.append(index)
        else:
            result.append(index)
    return result

```

The main annoyance is that if there are not enough elements to fill the nonzero, it will be zero padded. For a boolean mask, a zero pad is inappropriate; instead you want an invalid index, and then to fill the indexing op with some placeholder element like 0.

### Versions

master
[RFC] Extend CPU AMP to add FP16 support on eager mode
Labels: triaged, module: amp (automated mixed precision)

### 🚀 The feature, motivation and pitch

Currently low precision of PyTorch CPU AMP on eager mode is BF16 and it does not support FP16 i.e. torch.half. If we want to add FP16 support on CPU #97068, we need also to extend PyTorch CPU AMP.

This RFC is proposed to extend PyTorch CPU AMP to support both BF16 and FP16 on eager mode so that users can use `torch.cpu.amp.autocast` to set the lower precision data type to `torch.half/torch.bfloat16` and autocast will automatically cast the inputs to float, lower precision data type i.e. FP16/BF16 or make inputs fall through.

### Proposed Implementation:
**Extend autocast to support setting lower precision data type to FP16 on CPU.**

* Extend autocast init method on CPU: 

   Usage of `torch.autocast`:
   Before: `torch.autocast()` only supports setting dtype to torch.bfloat16 on CPU.  if users use `torch.autocast(device_type="cpu", dtype=torch.half)`, autocast will be disabled with the warning: "In CPU autocast, but the target dtype is not supported. Disabling autocast".

    After: users can set dtype=torch.bfloat16 or dtype=torch.half.:
    ```Python
    with torch.autocast(device_type="cpu", dtype=torch.bfloat16):
        ...
    with torch.autocast(device_type="cpu", dtype=torch.half):
        ...
    ```

* Nest behavior of autocast on CPU:

    Nest AMP with mixed torch.bfloat16 and torch.half is not defined yet in PyTorch. For example:
    ```Python
   with torch.autocast(device_type="cpu", dtype=torch.bfloat16):
        ...
        with torch.autocast(device_type="cpu", dtype=torch.half)::
            ...
    ```
    Both BF16 and FP16 are allowed on CUDA, and it also does not handle such usage specially. In addition,  in such case `cache_enabled` will be also enabled by default, and the dtype of cached tensor (e.g., BF16) may be different with the target data type (e.g., FP16 set by nested amp). Ops may report errors with inconsistent data types.

    Nesting AMP with mixed torch.bfloat16 and torch.half seems unreasonable. To simplify things we can disable such mixed usage or give warning and document it.

### Test
* Add test cases for FP16 autocast on CPU.

### Additional context

This RFC depends on FP16 supports of operators https://github.com/pytorch/pytorch/issues/97068

cc @mcarilli @ptrblck @leslie-fang-intel @jgong5
DISABLED test_nn_sequential_invocation_dynamic_shapes (torch._dynamo.testing.DynamicShapesMiscTests)
Labels: triaged, module: flaky-tests, skipped, module: unknown

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_nn_sequential_invocation_dynamic_shapes&suite=torch._dynamo.testing.DynamicShapesMiscTests) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/11781994621).

Over the past 3 hours, it has been determined flaky in 3 workflow(s) with 3 failures and 3 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_nn_sequential_invocation_dynamic_shapes`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/testing.py` or `/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/testing.py`

ConnectionTimeoutError: Connect timeout for 5000ms, GET https://raw.githubusercontent.com/pytorch/pytorch/master/test//opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/_dynamo/testing.py -2 (connected: false, keepalive socket: false, socketHandledRequests: 1, socketHandledResponses: 0)
headers: {}
Pytorch 2.0 installation tutorial does not work under Macbook
Labels: module: docs, triaged, module: macos

### 📚 The doc issue

I attempt to install Pytorch and follow this tutorial:
https://pytorch.org/get-started/pytorch-2.0/

However, this command gives the following error:
```
(base) jimmy@Jimmys-MacBook-Air mnist_pytorch % pip3 install numpy --pre torch --force-reinstall --index-url https://download.pytorch.org/whl/nightly/cu117
Looking in indexes: https://download.pytorch.org/whl/nightly/cu117
ERROR: Could not find a version that satisfies the requirement numpy (from versions: none)
```
Later on, I tried another [tutorial](https://pytorch.org/blog/Accelerating-Hugging-Face-and-TIMM-models/) with the following command and suceed:
```
(base) jimmy@Jimmys-MacBook-Air mnist_pytorch % pip3 install numpy --pre torch --force-reinstall --extra-index-url https://download.pytorch.org/whl/nightly/cu117
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/nightly/cu117
Collecting numpy
  Downloading numpy-1.24.2-cp310-cp310-macosx_10_9_x86_64.whl (19.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.8/19.8 MB 50.5 MB/s eta 0:00:00
Collecting torch
  Downloading torch-1.13.1-cp310-none-macosx_10_9_x86_64.whl (135.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.3/135.3 MB 17.2 MB/s eta 0:00:00
Collecting typing-extensions
  Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Installing collected packages: typing-extensions, numpy, torch
  Attempting uninstall: typing-extensions
    Found existing installation: typing_extensions 4.4.0
    Uninstalling typing_extensions-4.4.0:
      Successfully uninstalled typing_extensions-4.4.0
  Attempting uninstall: numpy
    Found existing installation: numpy 1.22.3
    Uninstalling numpy-1.22.3:
      Successfully uninstalled numpy-1.22.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
gensim 4.3.0 requires FuzzyTM>=0.4.0, which is not installed.
daal4py 2021.4.0 requires daal==2021.3.0, which is not installed.
scipy 1.7.3 requires numpy<1.23.0,>=1.16.5, but you have numpy 1.24.2 which is incompatible.
numba 0.56.4 requires numpy<1.24,>=1.18, but you have numpy 1.24.2 which is incompatible.
Successfully installed numpy-1.24.2 torch-1.13.1 typing-extensions-4.5.0
```

### Suggest a potential alternative/fix

replace all `--index-url` with `--extra-index-url` in the installation tutorial

cc @svekars @carljparker @malfet @albanD
Questions and Possible Features: Pytorch RPC 'future.wait()' will not release GIL which will block other thread's execution when using multithreading.
Labels: triaged, module: multithreading

Hi, I'm trying to build a 5 stages pipeline using python multithread. Each stage contains some cuda operation or `.cuda` (cuda memory copy operations) as well as some very light weight cpu compuatation. What's more, in each stage, I will use PyTorch distributed `rpc.rpc_async` to do some computation remotely and transfer them back to local memory. I then called `future.wait()` to wait for the output. One expected behavior is that during the `future.wait()`, the other concurrent stages can execute in parallel so that I can overlap the computation with communication. However, I found that the other threads are also blocked by this operation. My guess is that the `future.wait()` doesn't release the GIL although it seems to be implemented in CPP so the other thread can't get the GIL to execute. I wonder if my guess is right? If it's the true, how can I overlap this communication time with other threads in parallel? Thanks a lot!

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @anjali411 @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu @gcramer23
Encourage dynamo.export users to assume static by default if they call nonzero / unbacked SymInt
Labels: triaged, oncall: pt2, module: dynamic shapes, module: export

### 🐛 Describe the bug

@tugsbayasgalan has observed that on latest master you can end up with a bunch of "cannot guard on unbacked SymInt" failures that look something like `Ne(i0*((s2//2 - 1)//8)**2 + 2*i0*((s2//2 - 1)//8) + i0, ((s2//2 - 1)//8)**2 + 2*((s2//2 - 1)//8) + 1)`. These look complicated but they actually are quite trivial if you can substitute in s2 (in this case, the substitued guard is `Ne(36*i0, 36)`). This could be a legitimate problem if s2 was actually dynamic, but it's pretty common for it to actually be static (and we're just trying to export a "maximally" dynamic model). In this case, upfront marking it as static would solve the problem. We should encourage export users (perhaps by defaulting this way) to assume inputs are static.

cc @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 @soumith @avikchaudhuri @voznesenskym 

### Versions

master
[inductor][cpp] xcit_large_24_p8_224 OOM
Labels: triaged

`xcit_large_24_p8_224` has been hitting OOM intermittently on CI, see https://hud.pytorch.org/hud/pytorch/pytorch/master/1?per_page=50&name_filter=inductor_timm_cpu_acc . https://github.com/pytorch/pytorch/pull/94822 somehow makes the issue repeatable.
Graphstate checkpointing doesn't checkpoint ShapeEnv / shape guards
Labels: triaged, oncall: pt2, module: dynamic shapes

### 🐛 Describe the bug

https://github.com/pytorch/pytorch/pull/90665/files#diff-4c82a5798a61d4cceb176b2700ba6fdd7c3e72d575b8e7e22458589139459caa discusses a hypothetical problem where when we checkpoint graphstate (e.g., before starting inlining a function), we technically also need to checkpoint GraphEnv and roll it back if we hit a graph break.

Well, I managed to actually trigger this problem. As the comment says, all you need to do is some how not specialize on a global. So this is real, and I want an issue to track it.

However, in my particular case, I think I'm going to work around the problem by forcing specialization when the unspecialized int comes from a global. Is it enough? We'll see.

### Versions

master

cc @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 @soumith
The torch.sparse document's typo error
Labels: module: sparse, module: docs, triaged

### 📚 The doc issue

In the torch.sparse module(https://pytorch.org/docs/stable/sparse.html), there is a typo error of the "Operator overview" chapter(https://pytorch.org/docs/stable/sparse.html#operator-overview). The original description is
"The particularities of storage, that is the physical layout of the data, influences the performance of an operation but shhould not influence the semantics." And the error spelling is "shhould", which the right spelling is "should".


### Suggest a potential alternative/fix

Change the "shhould" to "should".

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer @svekars @carljparker
Build from source,  Undefined symbol: c10::detail::maybe_wrap_dim_slow(long long, long long, bool)
Labels: triaged, oncall: mobile, module: ios

### 🐛 Describe the bug

Hi

I tried to build Pytorch for iOS from source (Because we need to support more arch than pre-build binary, like LibTorch-Lite provides, so we're forced to build from source).

We followed the guidance in [build-pytorch-ios](https://pytorch.org/mobile/ios/#build-pytorch-ios-libraries-from-source). Everything turns out good, the building process is smooth. But we got errors while linking Pytorch to our project.

```
Showing Recent Messages
Undefined symbol: at::_ops::repeat_interleave_self_int::call(at::Tensor const&, long long, c10::optional<long long>, c10::optional<long long>)

Undefined symbol: at::_ops::pad::call(at::Tensor const&, c10::ArrayRef<long long>, c10::basic_string_view<char>, c10::optional<double>)

Undefined symbol: c10::detail::maybe_wrap_dim_slow(long long, long long, bool)
```

I have found [issue 13541](https://github.com/pytorch/pytorch/issues/13541), and tried adding ```"-D_GLIBCXX_USE_CXX11_ABI=0"``` flag to ```building_ios``` script but that doesn't fix the issue.

My compiler version:

Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: x86_64-apple-darwin21.6.0
Thread model: posix

If anyone could give a hint that where should I look into, it would be a great help.

Thanks a lot.

### Versions

I build from the master branch, don't know whether it's relevant.
CPU time performance is unstable
Labels: module: performance, module: cpu, triaged

### 🐛 Describe the bug

I retrain the mobilenetV3 with my own data. The accuracy is the same under cpu or gpu. But the time performance has a huge difference. 
For GPU: the costing time is pretty similar under different epoch. It only takes 15ms to inference single image.

But for CPU, epoch 1 takes over 40ms to inference single image. epoch 2 takes over 20ms and epoch 3 takes over 110ms. And everytime i train the model. Inference time is always different. 

I also use other framework to test whether this problem comes from cpu. But the tensorflow is pretty stable and it costs 25ms to inference.

My cpu is i5-7500 and torch version is 2.0.0.dev. I tried losts of version of torch(1.10.0, 1.12.1,even 1.5) .The problems still happen. Does anyone know what happened and how to fix this problem?

### Versions

My cpu is i5-7500 and torch version is 2.0.0.dev. I tried losts of version of torch(1.10.0, 1.12.1,even 1.5) .The problems still happen. Does anyone know what happened and how to fix this problem?

cc @ngimel @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
autograd.functional.jacobian : tensor instead of function as input for reverse mode?
Labels: feature, module: autograd, triaged, needs research, has workaround

### 🚀 The feature, motivation and pitch

For reverse-mode, would it be possible to make an overload that takes a tensor as input instead of function. This will simplify the coding quite a bit, when you are continuing the graph after the jacobian calculation.

Especially when you are going to take the jacobian of a jacobian etc.
```
y= torch.mv(A,x)
jac1 = torch.autograd.functional.jacobian(y, x, create_graph=True)
z = somefunction(y,jac1)
jac2 = torch.autograd.functional.jacobian(z ,x, create_graph=False)

a = anotherfunction(z,jac2,y,jac1)
``` 
Instead of :
```
def yf(x):
    return torch.mv(A,x)

def fun1(x):    
  jac = torch.autograd.functional.jacobian(yf,x,create_graph=True) 
  return jac

def fun2(x):
    y=yf(x)
    jac1=fun1(x)
    z=somefunction(y,jac1)
    return z
jac2 = torch.autograd.functional.jacobian(fun2,x,create_graph=False) 
z=fun2(x)
jac1 = fun1(x)
y=yf(x)
a = anotherfunction(z,fac2,y,jac1)
```

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
[PTD] dist.barrier() unreliable when using collectives from multiple threads.
Labels: oncall: distributed, module: c10d

### 🐛 Describe the bug

This is a hang that happens with https://github.com/pytorch/pytorch/pull/95819 but it's a reliability issue with any code that emits collectives from multiple threads including calls to dist.barrier.

This is likely to be common with PTD tests that uses CUDA, trigger backward in some model and do collectives from them (like when doing DDP or FSDP).

The work around is to replace the barrier call with `dist.all_reduce(torch.zeros((1,), device="cuda" if torch.cuda.is_available() else "cpu"))`.


### Versions

master

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
The Benchmark test for dynamo runs stuck
Labels: needs reproduction, triaged, oncall: pt2

### 🐛 Describe the bug

![image](https://user-images.githubusercontent.com/40196111/222394368-cb101449-5564-41e3-bc1a-84036edb7316.png

Stay here for a long time, no errors and no warnings.I just want to know what happend

### Error logs

_No response_

### Minified repro

_No response_

### Versions

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

cc @ezyang @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 @soumith
corrupted size vs prev size error
Labels: triaged, module: ddp

### 🐛 Describe the bug

I train my model with 16,000,000 data examples and 8 A100 cards. 
  ```
  model = torch.nn.parallel.DistributedDataParallel(
      model,
      device_ids=[self.args.local_rank],
      output_device=self.args.local_rank,
      find_unused_parameters=find_unused_parameters,
  )
```
In this code, it returns corrupted size vs prev size error.
When I tried 15,999,999 data, the error gone.

### Versions

1.13.0+cuda11.7

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Input names provided three but onnx recognizes two inputs only
Labels: module: onnx, triaged

I have given three input names to the model but it is taking only two input names any ideas related to this please?

```python
torch.onnx.export(model,
    (image, caption, cap_mask),
    "model.onnx",
    input_names=input_names,
    output_names=ouput_names,
    export_params=True
)
```

```bash
onnx_graph:
[
name: "caption"
type {
    tensor_type {
        elem_type: 7
        shape {
            dim {
                dim_value: 1
            }
            dim {
                dim_value: 128
            }
        }
    }
},
name: "cap_mask"
    type {
        tensor_type {
            elem_type: 9
            shape {
                dim {
                    dim_value: 1
                }
                dim {
                    dim_value: 128
                }
            }
        }
    }
]
```

Error:
```bash
Traceback (most recent call last):
File "dum.py", line 61, in
ort_inputs = {ort_session.get_inputs()[0].name: image, ort_session.get_inputs()[1].name: caption, ort_session.get_inputs()[2].name: cap_mask}
IndexError: list index out of range
```
[JIT] Support string type annotations in NamedTuples
Labels: oncall: jit

### 🚀 The feature, motivation and pitch

See below for an example clarifying what "string type annotations" means.

Normally we'd just advise users to not annotate like this. But this is an effect of `from __future__ import annotations` which is difficult to work around, because it applies per-file ([PEP](https://peps.python.org/pep-0563)). It's also plausible that users in some scenarios would want to use this in order to be able to import less packages, to reduce import times.

**Q**: Why do NamedTuples have this issue, but not normal types?
**A**: String annotations are supported for normal types, because the IR is loaded into C++ as strings and parsed in C++. But for NamedTuples, we parse the NamedTuple string name and then run a callback to python, where we use `getattr(obj, "__annotations__")` to get the annotations. At this point, we may not have the necessary context in order to evaluate the ForwardRef.

In order to fix this, we need to plumb the extra context through.

- [x] Fix this for type annotations - fixed in #96933
- [ ] Fix this for instances of NamedTuples whose types are inferred. For an example of this, see test_namedtuple_resolution_forwardref introduced in #96933. Suggestions on investigating this: first, un-mark the test as expectedFailure; then put a breakpoint at `registerNamedTuple` (in python_sugared_value.cpp). This will show the C++ stack leading to the named tuple issue. One option for fixing this is to plumb the RCB through to this callsite. (To run the test, `python test/test_jit.py -k test_namedtuple_resolution_forwardref`).

### Alternatives

Alternatively we can try to support this the way non-nn.Module classes are supported. Roughly speaking, these are pre-compiled and then cached in `torch/jit/_state.py`. Then, we don't need to worry about plumbing the context through. I didn't look at the details, so maybe there's other complications here or other context why NamedTuples were supported differently than other classes.

### Additional context

Rough idea of repro (untested)

```python
import torch
from typing import NamedTuple

class MyNT(NamedTuple):
    x: "str"
    y: "torch.Tensor"

def fn(x: MyNT):
    return x.y.relu()

torch.jit.script(fn)
```

If we also add `from __future__ import annotations`, then we'll find that `MyNT.__annotations__` shows a dict of `str -> FutureRef` instead of the `str -> typing.*` that we normally see.

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
weakref.proxy issue with torch.compile
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

The snippet below should work with the compiled function. Thre seems to be an undesired interaction with `weakref.proxy`

This causes https://github.com/Lightning-AI/lightning/issues/16822

### Error logs

```py
Good!  # regular `log`
Traceback (most recent call last):
  File "/home/carmocca/git/lightning/kk3.py", line 34, in <module>
    compiled_log()
  File "/home/carmocca/git/py310/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 228, in _fn
    return fn(*args, **kwargs)
  File "/home/carmocca/git/lightning/kk3.py", line 21, in log
    raise RuntimeError
RuntimeError
```

### Minified repro

```python
import weakref

import torch


class Trainer:
    def __init__(self):
        self.foo = True


class MyModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.trainer = None

    def forward(self):
        ...

    def log(self):
        if self.trainer.foo is None:
            raise RuntimeError
        print("Good!")


model = MyModel()
trainer = Trainer()
model.trainer = weakref.proxy(trainer)

# works
model.log()

compiled_log = torch.compile(model.log)
# fails
compiled_log()
```

### Versions

Today's nightly: `torch==2.0.0.dev20230301+cpu`

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
Let Nested Tensor Metadata be cached on GPU
Labels: triaged, module: nestedtensor

### 🚀 The feature, motivation and pitch

Filing this issue to track https://github.com/pytorch/pytorch/pull/95518#discussion_r1117869931

Currently nested tensor metadata lives on the CPU. This means that when we construct a nested tensor from say a tuple of (Tensor values, Tensor offsets) that live on the GPU, constructing the nested tensor metadata (e.g. seq_len column of sizes) will incur a sync. 

We want to treat a copy of the metadata on the GPU like a cached value with the ground truth being readily available on the CPU.

At a high level, if Tensor values, Tensor offsets are on GPU, then we can create the GPU copy of the metadata and send it to the CPU using non_blocking=True to a avoid a sync.

More design is needed.

### Alternatives

_No response_

### Additional context

_No response_

cc @cpuhrsch @jbschlosser @bhosmer @drisspg
Static asserts on accessor templates
Labels: module: cpp-extensions, triaged

### 🚀 The feature, motivation and pitch

PyTorch got bug report #54406, which turned out as improper usage of provided accessor API. In the discussion two solutions were suggested to prevent this in the future:
- Update docs
- Introduce static asserts on accessor template

As second solution seems more bullet proof, I'd like to suggest implementing it. My thinking was to restrict set of allowed types only on those which are supported, but don't see enough into the logic to say it's right approach. Can somebody comment on the idea and suggest the way to go?

### Alternatives

Documentation update

### Additional context

_No response_

cc @malfet @zou3519 @jbschlosser
SymInt'ify _gather_sparse_backward
Labels: triaged

When adding the missing sparse_grad parameter support for torch.gather (https://github.com/pytorch/pytorch/issues/95187), it triggers a dynamic shape test error (https://github.com/pytorch/pytorch/blob/9835c93abaf2961c72a8deec16ed9732383fbe0f/test/inductor/test_torchinductor_dynamic_shapes.py#LL43C18-L43C18). 

The problem is the function _gather_sparse_backward at https://github.com/pytorch/pytorch/blob/11f293a74e54fb216952b9b4756df5136659a383/aten/src/ATen/native/TensorAdvancedIndexing.cpp#L2025 needs to be SymInt'ified.
Implementing the batching rule for aten::bucketize.Tensor.
Labels: triaged, actionable, module: vmap, module: functorch

Dear maintainers of functorch,

I tried to use `functorch.vmap` to implement an extended version of `torch.bucketize`, so that I can bucketize a batched tensor with heterogeneous boundaries. 

I got the a `UserWarning: There is a performance drop because we have not yet implemented the batching rule for aten::bucketize.Tensor`.

It would be great if the batching rule could be implemented for `torch.bucketize`.

Thanks a lot.

cc @zou3519 @Chillee @samdow @soumith @kshitij12345 @janeyx99
change stacksize_analysis to worklist algorithm for better result
Labels: triaged, open source, Stale, module: dynamo, ciflow/inductor

The current algorithm for stacksize_analysis is round-robin algorithm with 100 as iteration upper limit, which is not an optimal choice. This PR implements the worklist version for better complexity and precision.


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @soumith @yanboliang @anijain2305 @desertfire