gopikrsmscs/torch-issues huggingface dataset:
[torch.fx] Upgrade on node info
Labels: triaged, module: fx

### 🚀 The feature, motivation and pitch

Torch.fx is a very useful tool capable of analysing and understanding deep models, but from an analysis perspective it would be interesting to know the dimensions of the tensors that have passed through the nodes. In order to understand where there might be memory space problems, for example.

What I'd like to see is for each node to have two additional arguments, input_size and output_size so that this information can be easily retrieved during an analysis using torch.fx

I've see something in torch.passes.shape_prop that seems to analyze tensors but I don't see anything on how to use it. And same for the proxy, I see that it have some useful information in but no way to use it.

Maybe the solution already exist but I prefer suggest if it doesn't

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @SherlockNoMad @soumith @EikanWang @jgong5 @wenzhe-nrv
torch.dist with minus norm returns tensor(0.), while with -inf can return result
Labels: module: distributions, triaged

### 🐛 Describe the bug

# with minus norm
```
import torch
arg_1_tensor = torch.rand([4], dtype=torch.float32)
arg_1 = arg_1_tensor.clone()
print(arg_1)
arg_2_tensor = torch.rand([4], dtype=torch.float32)
arg_2 = arg_2_tensor.clone()
print(arg_2)
arg_3 = -100
res = torch.dist(input=arg_1,other=arg_2,p=arg_3,)
print(res)
```
```
tensor([0.3692, 0.1006, 0.4169, 0.5297])
tensor([0.4667, 0.3731, 0.2566, 0.8941])
tensor(0.)
```

# with -inf norm
```
import torch
inf = float('inf')
arg_1_tensor = torch.rand([4], dtype=torch.float32)
arg_1 = arg_1_tensor.clone()
print(arg_1)
arg_2_tensor = torch.rand([4], dtype=torch.float32)
arg_2 = arg_2_tensor.clone()
print(arg_2)
arg_3 = -inf
res = torch.dist(input=arg_1,other=arg_2,p=arg_3,)
print(res)
```
```
tensor([0.2863, 0.5415, 0.4990, 0.6137])
tensor([0.1516, 0.4867, 0.1853, 0.8488])
tensor(0.0548)
```

I find this happens when norm is less than -40.

### Versions

pytorch version: 2.0.0
cuda : 118

cc @fritzo @neerajprad @alicanb @nikitaved
TracingContext.get().frame_summary_stack doesn't produce full stack trace
Labels: triaged, module: dynamo

### 🐛 Describe the bug

If you are here because an error message, comment on the issue to describe how you were affected, so we can help prioritize this issue.

TracingContext.get().frame_summary_stack doesn't report full backtraces; you'll only get frames from inside the region of code that dynamo traced through. In principle, we could also collect the regular backtrace from before callback entry and report that too.

@Chillee notes that it is good not to give too much information, and indeed the stack trace before calling into the model is probably not that useful. But if we have graph breaks inside the model, we may have lost useful context. Maybe only want the partial stack trace up to torch.compile? Not going to do it unless someone shouts.

### Versions

master

cc @soumith @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
Request for deterministic support for reflection_pad2d_backward_cuda
Labels: module: cuda, triaged, enhancement, module: padding

### 🚀 The feature, motivation and pitch

Currently, the `reflection_pad2d_backward_cuda` operation used in neural network training cannot be computed deterministically on the GPU. This causes issues for users who have set `torch.use_deterministic_algorithms(True)` and require deterministic behavior in their training process.

I would like to request that deterministic support be added for `reflection_pad2d_backward_cuda` so that users can perform neural network training deterministically on the GPU. Thank you for your consideration.

### Alternatives

_No response_

### Additional context

_No response_

cc @ngimel
Integrate open device privateuse1 customized method registration
Labels: triaged, module: backend

### 🚀 The feature, motivation and pitch

Currently, if user registers the privateuse1 backend, the user may need to perform the following operations:
1. Call `torch._register_device_module` to register the device module.
2. Call `torch.serialization.register_package` to register the `_tag` and `_deserialize` methods customized by privateuse1 backend.
There may be more examples
You need to register multiple places in the patch of privateuse1.

### Alternatives

Unified registration entry. Users only need to implement the corresponding method.
Such as [#98920](https://github.com/pytorch/pytorch/pull/98920)

### Additional context

_No response_
Change module to module_ in torch/csrc/api/include/torch/python.h
Labels: module: build, triaged

### 🚀 The feature, motivation and pitch

Until 2023.04.12 on master branch, 'module' is used as a variable name in torch/csrc/api/include/torch/python.h, however, 'module' has became a keyword since C++20, consider replace module to module_? 

### Alternatives

_No response_

### Additional context

_No response_

cc @malfet @seemethere
Move template code to header
Labels: module: cpp, triaged

### 🐛 Describe the bug

The template code https://github.com/pytorch/pytorch/blob/39fd7f945f292bea7af411946f75417966470359/torch/csrc/api/src/nn/modules/batchnorm.cpp#L17-L34 appears in the `.cpp` file.
It will make extending from `BatchNormImplBase` using libtorch troublesome.
Unless copying the above quoted code into users own source code, the linker will not able to find implementations of `pretty_print`

### Versions

latest master

cc @jbschlosser
Test failure: TestCommonCPU.test_python_ref__refs_abs_cpu_complex32
Labels: module: tests, triaged

### 🐛 Describe the bug

When I execute the following test case on s390x, I got the failure.

```
% python test/test_ops.py TestCommonCPU.test_python_ref__refs_abs_cpu_complex32
...

----------------------------------------------------------------------
Ran 1 test in 3.065s

FAILED (unexpected successes=1)
```

When I executed the same test on x86, it passed.
```
$ python test/test_ops.py TestCommonCPU.test_python_ref__refs_abs_cpu_complex32
...

x
----------------------------------------------------------------------
Ran 1 test in 0.920s

OK (expected failures=1)
```

So, this test suite expects that one of tests is generates different results. I am not sure why this test case expects to generate different results so far.

### Versions

```
Collecting environment information...
PyTorch version: 2.1.0a0+gite3df6a7
Is debug build: True
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (s390x)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-5.4.0-144-generic-s390x-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False

Versions of relevant libraries:
[pip3] numpy==1.24.2
[pip3] torch==2.1.0a0+gite3df6a7
[conda] Could not collect
```

cc @mruberry
Changes to TorchScript autodiff changing default behavior are no longer accepted
Labels: triage review, oncall: jit

TorchScript support is very limited currently, and changes to autodiff can (and did!) lead to very hard to diagnose bugs. To prevent these bugs, and in view of TorchScript not being actively developed, we no longer accept PRs that expand or change functionality of autodiff. If it's absolutely necessary to change autodiff for your backend, the changes should be behind a config flag and not enabled by default. 
To reviewers: please don't accept PRs modifying TS autodiff. 

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Expand component configurable logging system to C++
Labels: module: logging, triaged

The component configurable logging system introduced for #94788 needs to be expanded to control logs generated from C++

Some of the requirements for this are discussed here:
* https://github.com/pytorch/pytorch/issues/94788#issuecomment-1502519872
* https://github.com/pytorch/pytorch/issues/94788#issuecomment-1503989249

List of requirements:

  * Logs generated in C++ should get piped out to the Python logging system.

  * C++ should have access to all the logging component and artifact types
    available in Python.

  * From Python user's perspective, whether a log came from C++ or Python
    should be opaque.
Support SPDA on non-CUDA backends
Labels: oncall: transformer/mha

### 🚀 The feature, motivation and pitch

Currently, [SPDA](https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html#torch.nn.functional.scaled_dot_product_attention) supports  CUDA backend as outlined in the documentation tutorial. Requesting the op to become available for other backends, specifically, `torch_xla`. 

CC @JackCaoG @wconstab 


cc @jbschlosser @bhosmer @cpuhrsch @erichan1
Sharded Grad Scaler Issue Tracker
Labels: oncall: distributed, triaged, module: amp (automated mixed precision), module: fsdp

I recently unintentionally discovered that there exists a [sharded_grad_scaler.py](https://github.com/pytorch/pytorch/blob/4584851da5cad7f2e5f9fd5ed2245f3a06f8359e/torch/distributed/fsdp/sharded_grad_scaler.py#L4) which derives a lot from our [amp/grad_scaler.py](https://github.com/pytorch/pytorch/blob/e64ddd1ab9d46cfc921c19269969ffc5cd7d6f6c/torch/cuda/amp/grad_scaler.py#L195).

- [ ] A lot of the duplication seems unnecessary. ShardedGradScaler should reuse/call as much of GradScaler as possible instead copying/pasting. This way, ShardedGradScaler can be automatically enrolled in GradScaler bug fixes/improvements.
- [ ] It is uncertain whether ShardedGradScaler supports all that current GradScaler supports in a consistent way, such as being able to call unscale separate from step like in the grad clipping use case: https://pytorch.org/docs/stable/amp.html#torch.cuda.amp.GradScaler.unscale_
- [ ] ShardedGradScaler should be documented as a public API once it's ready. It would be good to link to it from the existing GradScaler page.

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @mcarilli @ptrblck @leslie-fang-intel @jgong5
[functorch] vmap_hessian_fc - fails under torch.compile
Labels: triaged, oncall: pt2, module: functorch

### 🐛 Describe the bug

Running vmap_hessian_fc from torchbench/userbench with following patch

```patch
diff --git a/userbenchmark/functorch/vmap_hessian_fc.py b/userbenchmark/functorch/vmap_hessian_fc.py
index ebfe7cf3..6ae546c0 100644
--- a/userbenchmark/functorch/vmap_hessian_fc.py
+++ b/userbenchmark/functorch/vmap_hessian_fc.py
@@ -1,6 +1,6 @@
 import torch
 import torch.nn as nn
 from functorch import vmap, jacfwd, jacrev
 from .util import BenchmarkCase
 
 # batched hessians of fully connected layers is a popular quantity
@@ -8,12 +8,25 @@ from .util import BenchmarkCase
 # This test case is from https://github.com/pytorch/functorch/issues/989
 # We haven't been able to get the full model yet, so, this test case
 # is going into the functorch userbenchmark instead of torchbenchmark.
+
+from torch._dynamo import allow_in_graph
+from functools import wraps
+
+def traceable(f):
+    f = allow_in_graph(f)
+
+    @wraps(f)
+    def wrapper(*args, **kwargs):
+        return f(*args, **kwargs)
+
+    return wrapper
+
 class VmapHessianFC(BenchmarkCase):
     def __init__(self):
-        device = 'cuda'
+        device = 'cpu'
         D1 = 2  # x, y
         D2 = 3  # u, v, p
-        B = 10000
+        B = 10
         x = torch.randn(B, D1).to(device)
 
         model = nn.Sequential(
@@ -43,9 +56,12 @@ class VmapHessianFC(BenchmarkCase):
             out = self.model(x)
             return out, out
 
-        hessian, pred = vmap(
+        fn = vmap(
             jacfwd(jacrev(predict, argnums=0, has_aux=True), argnums=0, has_aux=True),
             in_dims=0,
-        )(
+        )
+
+        fn = torch.compile(traceable(fn))
+        hessian, pred = fn(
             self.x
         )
```

Leads to failure:
```
RuntimeError: Failed running call_function <function VmapHessianFC.run.<locals>.predict at 0x7fa75ef8f040>(*(FakeTensor(FakeTensor(..., device='meta', size=(10, 2)), cpu),), **{}):
InferenceMode::is_enabled() && self.is_inference() INTERNAL ASSERT FAILED at "/home/kshiteej/Pytorch/pytorch_functorch/aten/src/ATen/native/VariableMethodStubs.cpp":67, please report a bug to PyTorch. Expected this method to only be reached in inference mode and when all the inputs are inference tensors. You should NOT call this method directly as native::_fw_primal. Please use the dispatcher, i.e., at::_fw_primal. Please file an issue if you come across this error otherwise.
(scroll up for backtrace)
```

### Versions

master

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @zou3519 @Chillee @samdow @janeyx99
[functorch] functorch_maml_omniglot - fails under torch.compile
Labels: triaged, oncall: pt2, module: functorch

### 🐛 Describe the bug

Running `functorch_maml_omniglot` from `torchbench` with following patch
```patch
diff --git a/torchbenchmark/models/functorch_maml_omniglot/__init__.py b/torchbenchmark/models/functorch_maml_omniglot/__init__.py
index faf16d73..430eaadf 100644
--- a/torchbenchmark/models/functorch_maml_omniglot/__init__.py
+++ b/torchbenchmark/models/functorch_maml_omniglot/__init__.py
@@ -10,6 +10,17 @@ from typing import Tuple
 from ...util.model import BenchmarkModel
 from torchbenchmark.tasks import OTHER
 
+from torch._dynamo import allow_in_graph
+from functools import wraps
+
+def traceable(f):
+    f = allow_in_graph(f)
+
+    @wraps(f)
+    def wrapper(*args, **kwargs):
+        return f(*args, **kwargs)
+
+    return wrapper
 
 def loss_for_task(net, n_inner_iter, x_spt, y_spt, x_qry, y_qry):
     params, buffers, fnet = net
@@ -66,7 +77,7 @@ class Model(BenchmarkModel):
         self.model = net
 
         root = str(Path(__file__).parent.parent)
-        self.meta_inputs = torch.load(f'{root}/maml_omniglot/batch.pt')
+        self.meta_inputs = torch.load(f'{root}/functorch_maml_omniglot/batch.pt')
         self.meta_inputs = tuple([torch.from_numpy(i).to(self.device) for i in self.meta_inputs])
         self.example_inputs = (self.meta_inputs[0][0],)
 
@@ -90,7 +101,9 @@ class Model(BenchmarkModel):
         # In parallel, trains one model per task. There is a support (x, y)
         # for each task and a query (x, y) for each task.
         compute_loss_for_task = functools.partial(loss_for_task, net, n_inner_iter)
-        qry_losses, qry_accs = vmap(compute_loss_for_task)(x_spt, y_spt, x_qry, y_qry)
+        fn = vmap(compute_loss_for_task)
+        fn = torch.compile(traceable(fn))
+        qry_losses, qry_accs = fn(x_spt, y_spt, x_qry, y_qry)
 
         # Compute the maml loss by summing together the returned losses.
         qry_losses.sum().backward()
```

Leads to failure:
```
Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten.convolution.default(*(FakeTensor(FakeTensor(..., device='meta', size=(160, 1, 28, 28)), cpu), 
```

### Versions

master

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @zou3519 @Chillee @samdow @janeyx99
[functorch] torch.compile - functorch transforms Interaction
Labels: triaged, oncall: pt2, module: functorch

### 🐛 Describe the bug

This is an umbrella issue for all the issues related to functorch - torch.compile interaction.

Currently, functorch transforms can be compiled under torch.compile with undocumented API. However, there are still a few issues which require resolution.

Example of compiling transform
```python
import torch
from torch._dynamo import allow_in_graph
from functools import wraps

def traceable(f):
    f = allow_in_graph(f)

    @wraps(f)
    def wrapper(*args, **kwargs):
        return f(*args, **kwargs)

    return wrapper

def fn(x):
    return vmap(torch.sin)(x)

opt_fn = torch.compile(traceable(fn))
opt_fn(torch.randn(3, 3))

```

torchbench issues:

- [ ] https://github.com/pytorch/pytorch/issues/98825
- [ ] https://github.com/pytorch/pytorch/issues/98827

user reported issues:
- [x] https://github.com/pytorch/pytorch/issues/97425
- [ ] https://github.com/pytorch/pytorch/issues/100105
- [ ] https://github.com/pytorch/pytorch/issues/100075

### Versions

master

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @zou3519 @Chillee @samdow @janeyx99
[FSDP] summon_full_params with_grad=True CPU offload can crash
Labels: oncall: distributed, triaged, module: fsdp

### 🐛 Describe the bug

It can crash if the grads are not None (i.e. optim.zero_grad(set_to_none=False) is called, or grad is not zeroed at all):

```
527     work = group._allgather_base(output_tensor, input_tensor)
528 RuntimeError: Tensors must be CUDA and dense
```

### Versions

main

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
File-level retry enhancements
Labels: triaged, module: devx

### 🐛 Describe the bug

Here is the list of several potential enhancements on how file-level retry https://github.com/pytorch/pytorch/pull/97506 works.  The goal is to start the discussion to see which makes sense and keep track of their progress:

* [x] Correctly resume from the last running test when timed out.  Here is an example of `distributed/_tensor/test_dtensor_ops` failure at 
https://github.com/pytorch/pytorch/actions/runs/4662747010/jobs/8253593870.  The test timed out flakily at the first run and was retried accordingly `Command took >30min, retrying (retries left=1)`.  However, when the retry kicked in, it started again from the beginning because the test timed out, not failing `stepwise: no previously failed tests, not skipping`.  Thus, the second try also timed out.  It makes sense to start from the last running test here instead.
* [ ] Ensure that each flaky test is retried once.  Take an example when a test fails, the retry logic will run the test again starting at the failed test.  The number of remaining retry would decrease from 1 to 0 (no more retry).  Assuming that the failed test is indeed flaky and it now passes, the test file will continue.  However, the edge case here is that if there is yet another flaky test further down the list, there is no retry left to handle it.  A possible solution is to only decrease the number of retries if the same flaky test fails again.  Here is an example https://github.com/pytorch/pytorch/actions/runs/4660804309/jobs/8249750328 in which the first flaky test `TestForeachCUDA.test_binary_op__foreach_clamp_max_is_fastpath_True_cuda_float32` was retried successfully while the second flaky test `TestForeachCUDA.test_binary_op__foreach_mul_is_fastpath_True_cuda_bfloat16` was out of luck.

Other issues:
* [ ] Integrate with ONNX tests https://github.com/pytorch/pytorch/issues/98626

### Versions

PyTorch CI

cc @ZainRizvi @kit1980 @clee2000
[FSDP] move up the first all gather
Labels: oncall: distributed, triaged, module: fsdp

### 🚀 The feature, motivation and pitch

The first allgather in FSDP is currently launched before computation for layer 1 needs to begin, but it can actually begin much sooner:

1. Potentially overlap with `_to_kwargs` data movement
2. API for advanced users to kick off this all gather even outside of model forward pass, to overlap with other work in their training loop.

The API could look as follows:

```
def gather_first_fsdp_layer_params(self: FullyShardedDataParallel):
    handle = self._root_handle
    handle.unshard() # kicks off the allgather on the unshard stream, but does not block
```

And can be used as follows in an example training loop:

```
while has_next(dataloader):
    batch = next(dataloader)
    # kick off FSPD allgather
    model.dense.gather_first_fsdp_layer_params()
    # overlap with next data transfer + sparse part of model
    dataloader.prefetch()
    model.sparse()
    model.dense()
```

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Discrepancy of supported Python versions between Get Started page and index of pre-built binaries for PIP installation
Labels: module: docs, triaged

### 📚 The doc issue

Sorry if this is not directly under contents under https://pytorch.org/docs/stable/index.html but for the page for "Get Started" https://pytorch.org/get-started/locally/

In the page description of required Python versions, it shows "Currently, PyTorch on Windows only supports Python 3.7-3.9; Python 2.x is not supported."

However, if we choose pip as the package manager to install Pytorch, the index of pre-built binaries of package `torch` is actually not containing `py37`. For example, under https://download.pytorch.org/whl/nightly/torch/, there is no item contains field `py37`. By the way, Python 3.10 and 3.11 seems actually supported.

If we try the command suggested by the page with Python 3.7, in my case `pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu118`, the installation is not going to work. The error message is:
```
ERROR: Could not find a version that satisfies the requirement torch (from versions: none)
ERROR: No matching distribution found for torch
```

### Suggest a potential alternative/fix

We may change the page description to match the actual support. Maybe the supported versions should be 3.8-3.11.

cc @svekars @carljparker
[SPMD] DistCompiler graph optimization improvement
Labels: oncall: distributed, triaged

### 🚀 The feature, motivation and pitch

The graph optimization for DistCompiler is enabled via the stack of https://github.com/pytorch/pytorch/pull/98182. Many feedbacks and experiences are provided during the development and code review of the stack. This issue is used to track the issues and TODOs.

### List of TODOs
1. Instead of having another layer of Module, IterGraphModule should serve the purpose of lowering Inductor. Context: https://github.com/pytorch/pytorch/pull/98182/files#r1158786964
2. `graph_optimization_pass` should support `run_before` argument.
3. `graph_optimization_pass` should support support multiple runs of graph optimization.
4.  Ensure that `_optimized_func` of `graph_optimization_pass` does not have conflict.
5. Graph optimization passes should support symbolic shape. Context: https://github.com/pytorch/pytorch/pull/98285/files#r1161194034



cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Linking ResNeXt PyTorch Hub in Pipeline docs
Labels: oncall: distributed

### 📚 The doc issue

Mention ResNeXt model at the first without link reference to PyTorch Hub.

see https://pytorch.org/docs/stable/pipeline.html#skip-connections

The main reason to show differentiation between ResNet and ResNeXt model. Apart from ResNet topic is excluded in docs of Pipeline. Since ResNeXt is next generation of well known more popular ResNet.

Introducing ResNeXt at first is a necessity.

### Suggest a potential alternative/fix

Provide link to PyTorch Hub ResNeXt in Pipeline docs.

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
DISABLED test_gradgrad_nn_GroupNorm_cuda_float64 (__main__.TestModuleCUDA)
Labels: module: nn, triaged, skipped

Platforms: linux

This test was disabled because it is failing on master ([recent examples](http://torch-ci.com/failure/test_modules.py%3A%3ATestModuleCUDA%3A%3Atest_gradgrad_nn_GroupNorm_cuda_float64)).

This test is failing on slow gradcheck, pending a fix https://github.com/pytorch/pytorch/pull/98424#issuecomment-1499858018

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
DISABLED test_grad_nn_GroupNorm_cuda_float64 (__main__.TestModuleCUDA)
Labels: module: nn, triaged, skipped

Platforms: linux

This test was disabled because it is failing on master ([recent examples](http://torch-ci.com/failure/test_modules.py%3A%3ATestModuleCUDA%3A%3Atest_grad_nn_GroupNorm_cuda_float64)).

This test is failing on slow gradcheck, pending a fix https://github.com/pytorch/pytorch/pull/98424#issuecomment-1499858018

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki
[ux] Non-blocking tensor constructors
Labels: triaged, enhancement, has workaround, module: tensor creation

### 🚀 The feature, motivation and pitch

It appears that regular `torch.tensor`/`torch.as_tensor` constructors incur blocking behavior when placing the result in CUDA memory: https://github.com/pytorch/vision/issues/7504 https://github.com/pytorch/vision/pull/7506

So @nlgranger had to replace `torch.tensor(..., device = my_cuda_device)` by `torch.tensor(...).to(device = my_cuda_device, non_blocking = True)`

I think, it would be more idiomatic/clean to allow `non_blocking` argument directly on torch.tensor/torch.as_tensor

### Alternatives

_No response_

### Additional context

_No response_

cc @gchanan @mruberry
DISABLED test_transpose_with_norm (__main__.CPUReproTests)
Labels: triaged, skipped, module: inductor

Platforms: linux

This test was disabled because it is failing on master ([recent examples](http://torch-ci.com/failure/inductor%2Ftest_cpu_repro.py%3A%3ACPUReproTests%3A%3Atest_transpose_with_norm)).

After https://github.com/pytorch/pytorch/pull/97841 to re-enable it, this test has been failing consistently, i.e. https://hud.pytorch.org/pytorch/pytorch/commit/46d765c15e702e2e2bc64b2948fba1f8845c4cda

cc @soumith @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire
Add test/distributed/test_c10d_mpi.py
Labels: oncall: distributed, triaged

>  We have one config where MPI is available and we build the wheels with MPI support.

https://github.com/pytorch/pytorch/pull/98545#issuecomment-1500440581

We don't have any tests for MPI as we do for gloo (test_c10d_gloo) or nccl (test_c10d_nccl). As @malfet mentions since we have support in CI for mpi, we should at least create a basic test_c10d_mpi to perform basic distributed operations (e.g. init_process_group) to prevent any future regressions.

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @kwen2501 @awgu
Wrong illustration in README.md
Labels: module: docs, triaged

### 📚 The doc issue

The last frame of the illustration is wrong:

![wrong illustration](https://github.com/pytorch/pytorch/raw/390c51bf87a5e9958f0190c0226493dc4dcdd61e/docs/source/_static/img/dynamic_graph.gif)


### Suggest a potential alternative/fix

The correct illustration should look like this:

![correct illustration](https://user-images.githubusercontent.com/8487114/230635413-925f0858-7166-4557-bb46-484be4b8f1f2.png)


cc @svekars @carljparker
Cannot use AT_CUDA_DRIVER_CHECK from user code
Labels: module: build, module: cpp-extensions, module: internals, module: cuda, triaged

### 🐛 Describe the bug

In my C++ extension that links against PyTorch I cannot use the AT_CUDA_DRIVER_CHECK macro to check the return code of a call to the CUDA driver API, because that macro makes use of the at::cuda::NVRTC struct which is defined in the ATen/cuda/nvrtc_stub/ATenNVRTC.h header, which doesn't get installed/shipped by PyTorch!

### Versions

I'm using PyTorch 2.0.0 installed from the official conda channels (build py3.9_cuda11.7_cudnn8.5.0_0).

cc @malfet @seemethere @zou3519 @ezyang @bhosmer @smessmer @ljk53 @bdhirsh @ngimel
torch.jit.script codegen warning with cuda and vmap
Labels: oncall: jit, triaged, module: functorch

### 🐛 Describe the bug

I'm getting the following warning which hints at suboptimal speed, and doesn't look like it should happen at any point.

```... site-packages\torch\_functorch\vmap.py:619: UserWarning: FALLBACK path has been taken inside: torch::jit::fuser::cuda::runCudaFusionGroup. This is an indication that codegen Failed for some reason.
To debug try disable codegen fallback path via setting the env variable `export PYTORCH_NVFUSER_DISABLE=fallback`
 (Triggered internally at C:\cb\pytorch_1000000000000\work\third_party\nvfuser\csrc\manager.cpp:340.)
  batched_outputs = func(*batched_inputs, **kwargs)```

This can be reproduced with the following code:

```import torch
from torch import vmap
import torch.jit

@torch.jit.script
def test(params):
    x0 = params[0]
    y = torch.arange(0, 64, dtype=torch.float32, device=params.device)
    return torch.cos(x0)*y

params = torch.zeros((200, 4), dtype=torch.float32, device='cuda')
torch.vmap(test, chunk_size=100)(params)
```

It seems to occur only when erf is passed an array instead of scalar.
I tested this both on pytorch 2.0 and on the nightly build (version is collected below), and on windows and ubuntu (see collected environments)

### Versions

[windows_env.txt](https://github.com/pytorch/pytorch/files/11175006/windows_env.txt)
[linux_env.txt](https://github.com/pytorch/pytorch/files/11175007/linux_env.txt)


cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel @zou3519 @Chillee @samdow @soumith @kshitij12345 @janeyx99
Memory corruption using torch.ops.* to access re-registered operator
Labels: module: internals, triaged

### 🐛 Describe the bug

We register an operator (foo::sum), call torch.ops.foo.sum, de-register it (by deleting the Library object), re-register it, and then call torch.ops.foo.sum again. This causes memory corruption (and leads to segfaults).

Repro: Run the following under an asan build.
```python
class TestTesting(TestCase):
    def test_AAA(self) -> None:
        from torch.library import Library

        my_lib1 = Library("foo", "DEF")
        my_lib1.define("sum(Tensor self) -> Tensor")

        @torch.library.impl(my_lib1, "sum", "CPU")
        def my_sum(*args, **kwargs):
            return args[0]

        x = torch.tensor([1, 2])
        self.assertEqual(torch.ops.foo.sum(x), x)

        import sys
        assert sys.getrefcount(my_lib1) == 2
        del my_lib1

        my_lib1 = Library("foo", "DEF")
        my_lib1.define("sum(Tensor self) -> Tensor")

        @torch.library.impl(my_lib1, "sum", "CPU")
        def my_sum(*args, **kwargs):
            return args[0]

        x = torch.tensor([1, 2])
        self.assertEqual(torch.ops.foo.sum(x), x)
```
Likely related to https://github.com/pytorch/pytorch/issues/98537, unsure if same exact issue.

### Versions

master

cc @ezyang @bhosmer @smessmer @ljk53 @bdhirsh
Segfault when using torch.ops.* to access de-registered op
Labels: module: crash, triaged, module: dispatch, module: library

### 🐛 Describe the bug

We register an operator (foo::sum), call torch.ops.foo.sum, de-register it (by deleting the Library object), and then call torch.ops.foo.sum again. This segfaults.

```python
import torch
from torch.library import Library

my_lib1 = Library("foo", "DEF")
my_lib1.define("sum(Tensor self) -> Tensor")

@torch.library.impl(my_lib1, "sum", "CPU")
def my_sum(*args, **kwargs):
    return args[0] * 2

x = torch.tensor([1, 2])
torch.ops.foo.sum(x)

import sys
assert sys.getrefcount(my_lib1) == 2
del my_lib1

x = torch.tensor([1, 2])
torch.ops.foo.sum(x)
```

It should not segfault

### Versions

master

cc @anjali411
torch.cond should work with expressions involving SymInt
Labels: triaged, module: functorch

### 🐛 Describe the bug

If x.size(0) is an unbacked symint (because x came from nonzero), it is useful to say `torch.cond(x.size(0) > 5, do_true, do_false)`.

I believe @tugsbayasgalan will indirectly finish this off with https://github.com/pytorch/pytorch/pull/98453 because with symbool, we can then do `scalar_to_tensor(x.size(0) > 5)` and then pass that as a tensor argument to torch.cond.

### Versions

master

cc @zou3519 @Chillee @samdow @soumith @kshitij12345 @janeyx99
Power VSX vectorization support disabled
Labels: module: build, triaged, module: regression, module: POWER

### 🐛 Describe the bug

During cmake cleanup VSX vectorization support for power was disabled as well:

https://github.com/pytorch/pytorch/commit/847dbb8684f9d9bbf59cae629d07bff3ede0c4a2#diff-12e8125164bbfc7556b1781a8ed516e333cc0bf058acb7197f7415be44606c72L1729

ZVECTOR support for s390x was removed as well, but it's not part of this issue, it's already being worked on separately.
For ZVECTOR it takes more than just 1 cmake line to properly restore vectorization support, thus I assume it might be also the case for VSX.

### Versions

pytorch master branch

cc @malfet @seemethere
Higher order derivatives not working when setting compute device to `torch.device("mps")`
Labels: module: autograd, triaged, module: mps

### 🐛 Describe the bug

I have a code example that performs fast updates, similar to what we have in MAML, to a perform inner updates and then computes the meta-loss to backpropagate through the optimization trajectory. When computing gradients in the fast updates, we can set `create_graph=True` to enable second-order derivatives when call backward on the meta-loss. When using `torch.device("mps")`, it throws an error that `derivative for aten:linear is not implemented`. It works fine when you set `create_graph=False` in the inner updates but then it wouldn't compute the higher order derivatives. I don't get the error when using `torch.device("cpu")` and `torch.device("cuda")`. Here is the code to reproduce the error:

```python

device = torch.device("mps")
model = nn.Sequential(nn.Linear(10, 5), nn.ReLU(), nn.Linear(5, 1))
model.to(device)

# Initial fast parameters
fast_params_0 = {n: deepcopy(p) for (n, p) in model.named_parameters()}

# First inner update
x = torch.randn(10, 10, device=device)
y = torch.randn(10, 1, device=device)
logits_0 = torch.func.functional_call(model, fast_params_0, x)
loss = nn.MSELoss()(logits_0, y)
grads_0 = torch.autograd.grad(loss, fast_params_0.values(),
                              create_graph=True,
                              retain_graph=True)

# Compute fast parameters after the first inner update
fast_params_1 = {n: p - 0.1 * g for ((n, p), g) in zip(fast_params_0.items(), list(grads_0))}

# Compute meta-loss and backprop through the optimization trajectory
x = torch.randn(10, 10, device=device)
y = torch.randn(10, 1, device=device)
logits_1 = torch.func.functional_call(model, fast_params_1, x)
met_loss = nn.MSELoss()(logits_1, y)
met_loss.backward()

```

And, the error I get:
```
RuntimeError: derivative for aten::linear_backward is not implemented
```

*I get the same error for any layer type. 

### Versions

```
[pip3] numpy==1.23.5
[pip3] pytorchcv==0.0.67
[pip3] torch==2.0.0
[pip3] torchaudio==2.0.0
[pip3] torchmetrics==0.11.4
[pip3] torchvision==0.15.0
[conda] numpy                     1.23.5           py39h1398885_0  
[conda] numpy-base                1.23.5           py39h90707a3_0  
[conda] pytorch                   2.0.0                   py3.9_0    pytorch
[conda] pytorchcv                 0.0.67                   pypi_0    pypi
[conda] torchaudio                2.0.0                  py39_cpu    pytorch
[conda] torchmetrics              0.11.4                   pypi_0    pypi
[conda] torchvision               0.15.0                 py39_cpu    pytorch
```

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @kulinseth @malfet @DenisVieriu97 @razarmehr @abhudev
Strided to batch BSR/BSC conversion fails when the number of zeros per block varies while the number of blocks per patch is constant
Labels: module: sparse, triaged

## Issue description

As in the title.

## Code example

```python
>>> torch.tensor([[[1, 2]], [[3, 4]]]).to_sparse_bsr((1, 1))
tensor(crow_indices=tensor([[0, 2],
                            [0, 2]]),
       col_indices=tensor([[0, 1],
                           [0, 1]]),
       values=tensor([[[[1]],

                       [[2]]],


                      [[[3]],

                       [[4]]]]), size=(2, 1, 2), nnz=2,
       layout=torch.sparse_bsr)
>>> torch.tensor([[[1, 2]], [[0, 4]]]).to_sparse_bsr((1, 1))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: Expect the same number of specified elements per batch.
>>> torch.tensor([[[1, 0]], [[0, 4]]]).to_sparse_bsr((1, 1))
tensor(crow_indices=tensor([[0, 1],
                            [0, 1]]),
       col_indices=tensor([[0],
                           [1]]),
       values=tensor([[[[1]]],


                      [[[4]]]]), size=(2, 1, 2), nnz=1,
       layout=torch.sparse_bsr)
```

Notice that in the failing conversion example, the number of zeros in the first block is 0 and in the second block it is 1.

Apparently, the check logic in
https://github.com/pytorch/pytorch/blob/ccc27bc361f2fa5043534b8f898922ffd0ca9340/aten/src/ATen/native/TensorConversions.cpp#L95-L98
is flawed for BSR and BSC conversion cases.

## System Info

- PyTorch version: master

cc @alexsamardzic @nikitaved @cpuhrsch @amjames @bhosmer
torch.fx.GraphModule inside custom backend has `training` attribute always set to `True` regardless of the user settings
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

Calling `eval()` and `train()` on either the original `torch.nn.Module` or `OptimizedModule` returned by `torch.compile` has no effect on the `training` attribute of `torch.fx.GraphModule` that is passed to custom backend function.

### Error logs

_No response_

### Minified repro

```
import torch

def my_custom_backend(gm, example_inputs):
    print(gm.training)
    return gm.forward

class MockModule(torch.nn.Module):
    def __init__(self):
        super().__init__()
    def forward(self, x):
        if self.training:
            return x + 2
        else:
            return x + 3

mod = MockModule()

optimized_mod = torch.compile(mod, backend=my_custom_backend)

mod.eval()
optimized_mod.eval()
print(optimized_mod(torch.zeros(10)))
print(optimized_mod(torch.zeros(10)))

mod.train()
optimized_mod.train()
print(optimized_mod(torch.zeros(10)))
print(optimized_mod(torch.zeros(10)))

mod.eval()
optimized_mod.eval()
print(optimized_mod(torch.zeros(10)))
print(optimized_mod(torch.zeros(10)))
```

Result:

```
True
tensor([3., 3., 3., 3., 3., 3., 3., 3., 3., 3.])
tensor([3., 3., 3., 3., 3., 3., 3., 3., 3., 3.])
True
tensor([2., 2., 2., 2., 2., 2., 2., 2., 2., 2.])
tensor([2., 2., 2., 2., 2., 2., 2., 2., 2., 2.])
tensor([3., 3., 3., 3., 3., 3., 3., 3., 3., 3.])
tensor([3., 3., 3., 3., 3., 3., 3., 3., 3., 3.])
```

### Versions

Torch 2.0

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
Options are not forwarded to the custom backend
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

Based on the description inside the [torch. compile](https://pytorch.org/docs/stable/generated/torch.compile.html) options are passed to the backend. Unfortunately, they are only passed to the inductor backend. Currently, the backend function has the following contract `(gm: torch.fx.GraphModule, example_inputs: List[torch.Tensor]) -> Callable` and there is no way to get options from the registered backend function. 

Am I missing something? Is there a way to get options from the custom backend function? If not, is it possible to add that possibility or update the description in the documentation?

### Error logs

_No response_

### Minified repro

#torch version 2.0

### Versions

#torch version 2.0

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
Improvements to FSDP debugability
Labels: oncall: distributed, triaged, module: fsdp

### 🚀 The feature, motivation and pitch

There are a couple pain points which make FSDP harder to debug: 
- In some cases, post backward hooks don't fire resulting in layers not getting gradients, resulting in training convergence issues. We could add logging in a debug mode for this, but need to think of a more comprehensive solution to identify this issue. 
- On lazy init, we should iterate through the original params and check that shared params are in the same FSDP instance. 


### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Bring CudaPluggableAllocator to feature parity with the Native Allocator
Labels: module: internals, module: cuda, triaged, module: CUDACachingAllocator

### 🚀 The feature, motivation and pitch

I have tried a few times to add Unified Memory support to Pytorch, so as to leverage as many resources of my computer as possible while running training and inference alike, but to no avail; so I abandoned my fork somewhat. After I hear about the pluggable Allocator mechanism, I tried it and [RAPIDS rmm](https://github.com/AUTOMATIC1111/stable-diffusion-webui)  with [stable diffusion WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui)  , but it gave errors such as "CudaPluggableAllocator does not yet support CacheInfo", and thus prohibiting my ability to run operations requiring more than 5.5GB of memory efficiently; thus, I would like to request that CudaPluggableAllocator get all features which the native allocator has. 

### Alternatives

Implement Cuda and ROCm Unified Memory support, and provide users an easy way to use it, like how one can switch between the native allocator and Cudamallocasync with an environment variable.

### Additional context

_No response_

cc @ezyang @bhosmer @smessmer @ljk53 @bdhirsh @ngimel
tacotron2 times out
Labels: triaged, oncall: pt2, module: inductor

Repro:
```
python benchmarks/dynamo/torchbench.py --accuracy --inference --amp --backend inductor --disable-cudagraphs --device cuda --only tacotron2
```

ctrl+c gives this stack information, which looks like a problem in the fuser heuristic,
```
  File "/fsx/users/binbao/conda/envs/release/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 636, in __init__
    self.fuse_nodes()
  File "/fsx/users/binbao/conda/envs/release/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 817, in fuse_nodes
    self.fuse_nodes_once()
  File "/fsx/users/binbao/conda/envs/release/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 833, in fuse_nodes_once
    if self.can_fuse(node1, node2) and not self.will_fusion_create_cycle(
  File "/fsx/users/binbao/conda/envs/release/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 907, in will_fusion_create_cycle
    return any(check(self.name_to_fused_node[n]) for n in combined_predecessors)
  File "/fsx/users/binbao/conda/envs/release/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 907, in <genexpr>
    return any(check(self.name_to_fused_node[n]) for n in combined_predecessors)
  File "/fsx/users/binbao/conda/envs/release/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 896, in check
    return bool(combined_names & node.recursive_predecessors) or any(
KeyboardInterrupt
```

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10
Need better error message when a merge cancelled because of timeout
Labels: module: ci, triaged

As suggested by @malfet, the error message needs improvements.

See this as an example: https://github.com/pytorch/pytorch/pull/98201#issuecomment-1497216298

The merge was cancelled because MacOS jobs time-outed - see https://github.com/pytorch/pytorch/issues/98362
The error message on the PR is just "The merge job was canceled. If you believe this is a mistake,then you can re trigger it through pytorch-bot", which is not very informative.

Should be something like this instead:

{timeout//60} hours have passed, but {len(jobs_pending} job(s) are still running, first few of them are ...)


cc @seemethere @malfet @pytorch/pytorch-dev-infra
README could use link to governance
Labels: high priority, module: docs, triaged

### 📚 The doc issue

README.md does not have link to governance https://github.com/pytorch/pytorch/blob/master/docs/source/community/governance.rst

~~indicate different governance structure, different maintainers, etc..~~

~~### Suggest a potential alternative/fix~~

### Suggest updating the readme to point to the governance documentation ~~.. modify the current readme contents to be a thank you for Emeritus efforts section.~~

updated issue to reflect the readme maintainer list is now in sync .. and could benefit from a link to governance structure docs

cc @ezyang @gchanan @zou3519 @svekars @carljparker
Torch Compile is slightly slower than eager mode.
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

When running some models on Torch, I have noticed that the torch.compile mode is slightly slower than the eager mode. 

It may or may not be related to this issue : https://github.com/pytorch/pytorch/issues/98102

one example is : microsoft-deberta-base

To reproduce:

go to this folder transformers/examples/pytorch/language-modeling/  and run:

eager mode:
`python run_mlm.py  --model_name_or_path microsoft/deberta-v3-base --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1  --num_train_epochs 1 --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --do_train --do_eval --overwrite_output_dir --output_dir ./outputs/ --seed 1137 --fp16 --report_to none --max_train_samples 1000 `

torch.compile:
`python run_mlm.py  --model_name_or_path microsoft/deberta-v3-base --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1  --num_train_epochs 1 --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --do_train --do_eval --overwrite_output_dir --output_dir ./outputs/ --seed 1137 --fp16 --report_to none --max_train_samples 1000 --torch_compile`

results :
<html>
<body>
<!--StartFragment-->

Metric | Eager | TorchCompile
-- | -- | --
Avg of 2nd half | 72.44162 ms | 102.73143 ms
Train loss | 5.995 | 5.9397
Train runtime | 0:03:09.17 | 0:04:38.75
Train samples | 1000 | 1000
Train samples per second | 5.286 | 3.587
Train steps per second | 5.286 | 3.587
Eval accuracy | 0.3637 | 0.3657
Eval loss | 4.8822 | 4.8525
Eval runtime | 0:00:10.11 | 0:00:32.71
Eval samples | 230 | 230
Eval samples per second | 22.746 | 7.031
Eval steps per second | 22.746 | 7.031
Perplexity | 131.92 | 128.0628

<!--EndFragment-->
</body>
</html>

Ran on a Single Tesla V100 16GB GPU.






### Versions

[conda] numpy                     1.24.1                   pypi_0    pypi
[conda] pytorch-triton            2.1.0+46672772b4          pypi_0    pypi
[conda] torch                     2.1.0.dev20230404+cu117          pypi_0    pypi
[conda] torch-ort                 1.14.0                   pypi_0    pypi
[conda] torchaudio                2.0.0.dev20230313+cu117          pypi_0    pypi
[conda] torchvision               0.15.0.dev20230313+cu117          pypi_0    pypi
[conda] triton                    2.0.0                    pypi_0    pypi

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh