gopikrsmscs/torch-issues huggingface dataset:
dynamo.optimizations.training.aot_autograd does not trace correct overload
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

In the following Python program, `x + 2` gets traced as `torch.ops.aten.add.Tensor` instead of `torch.ops.aten.add.Scalar` which would be more technically correct (and is the op that torch.jit.script gives). I noticed this because in Torch-MLIR we verify that the operands match the schema exactly. Is it possible to have aot_autograd trace the more technically correct overload? Or can you provide guidance about how to correctly emulate the promotion semantics in appropriate generality?

Also, to add onto that, `torch.ops.aten.add.Tensor` is missing an argument. I need special-case code in my importer to handle this (by querying the schema for the default value). One possibility could be that PyTorch provides a utility that normalizes the graph of aten ops to "typecheck" correctly against the schemas of the constituent ops including reifying all default values.

```python
from typing import List

import torch
import torch._dynamo as dynamo
from torch._dynamo.optimizations.training import aot_autograd


def my_backend(gm: torch.fx.GraphModule, example_inputs: List[torch.Tensor]):
    gm.print_readable()
    return gm


@dynamo.optimize(aot_autograd(fw_compiler=my_backend))
def f(x):
    return x + 2


example_inputs = (torch.randn(3, 4),)
f(*example_inputs)
```

Output:
```
class <lambda>(torch.nn.Module):
    def forward(self, arg0_1: f32[3, 4]):
        # File: /tmp/repro2.py:15, code: return x + 2
        add: f32[3, 4] = torch.ops.aten.add.Tensor(arg0_1, 2);  arg0_1 = None
        return (add,)     
```

### Versions

PyTorch version: 2.0.0.dev20221213+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux rodete (x86_64)
GCC version: (Debian 12.2.0-3) 12.2.0
Clang version: 14.0.6-2
CMake version: version 3.25.0
Libc version: glibc-2.35

Python version: 3.10.8 (main, Nov  3 2022, 15:17:13) [GCC 12.2.0] (64-bit runtime)
Python platform: Linux-5.19.11-1rodete1-amd64-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.24.0rc2
[pip3] torch==2.0.0.dev20221213+cpu
[pip3] torchvision==0.15.0.dev20221213+cpu
[conda] Could not collect

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
Support for Transformer Models on Android with Vulkan Backend
Labels: triaged, module: vulkan

### 🚀 The feature, motivation and pitch

Hello

We are currently using a number of different transformer models (plain BERT encoders with attached classification head) on Android. In order to increase the performance of the inference on mobile devices, we're currently evaluating the Pytorch Vulkan backend. We therefore compiled Pytorch from source for desktop to export our model for mobile and specifically for the Vulkan backend. Afterwards, we compiled Pytorch with Vulkan backend for Android, included the resulting aars in our Testing-App and tried running the model. The following error occured:

```
com.facebook.jni.CppException: Vulkan vk_format(): no corresponding format for dtype
```

After some digging around in the source code, we found that the dtype for which there is no corresponding format is `Long`. We assume that the model requires a `Long` as index to the embedding bag. To make sure that nothing was wrong with our custom build, we also tried to run a basic mobilenet v2 model on Android with Vulkan backend and it seemed to work as expected. We therefore assume that this is an unsupported feature. 

Just out of curiosity we also tried passing `Int` and `Float` values to the model, just to see what would happen. With `Int` we received the same error whereas with `Float` the following error occured:
```
com.facebook.jni.CppException: Could not run 'aten::as_strided' with arguments from the 'Vulkan' backend. This could be because the operator doesn't exist for this backend
```

Can you please confirm that this is in fact an unsupported feature? Is there currently some kind of workaround for this? Is there a plan to support this in the future?

Thank you in advance for your reply and have a great day!

### Alternatives

_No response_

### Additional context

_No response_
Torch SummaryWriter import fails with torch 2.0 with an error on numpy.object
Labels: module: tensorboard, oncall: visualization

### 🐛 Describe the bug

On pytorch 2.0 the code line 
'from torch.utils.tensorboard import SummaryWriter'
produces the following error.
![Cattura](https://user-images.githubusercontent.com/16682919/207862762-2a57956f-4c93-419f-b6f5-8be5bba40447.PNG)


### Versions

pytorch 2.0
numpy 1.24.0rc2
Error in guard code crashes process NULL ERROR: /Users/ezyang/Dev/pytorch-metal/torch/csrc/dynamo/eval_frame.c:251
Labels: triaged, bug, oncall: pt2

### 🐛 Describe the bug

Crashing processes makes your SREs unhappy. Don't crash; propagate the exception back to Python.

This is annoying to do as lookup() function in eval_frame.c already uses null return to mean something different.

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh
Retrieve Tensor from Tensor.data_ptr()
Labels: triaged, core issue

### 🚀 The feature, motivation and pitch

We are working on a DSL implementation which would use the PyTorch to manage the allocation of tensors and the kernel would accept the Tensor.data_ptr() and other attributes, e.g. stride, type, shape as arguments. However we need assemble these info automatically to a tensor-like object now, but it difficult to correspond these info scattering in arguments. 


### Alternatives

There are 4 choices:
1. accept torch tensor as input
2. encapsulate torch tensor to hide implementation or encapsulate info from tensor to one new type
3. assemble these info automatically as stated before  - difficult to correspond arguments and seems prone to errors
4. recover the tensor from the data_ptr(), but other arguments are useless except explicit reference 

We will try the 1st and 2nd options and also seize the opportunity of 4th option

### Additional context

_No response_
ModuleNotFoundError: No module named 'torch._C._distributed_c10d'; 'torch._C' is not a package
Labels: module: build, triaged

### 🐛 Describe the bug

When I build PyTorch on OS X, and then attempt to import torch.distributed, I get this error:

```
$ python -c "import torch.distributed.distributed_c10d"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/ezyang/Dev/pytorch-cpu/torch/distributed/distributed_c10d.py", line 16, in <module>
    from torch._C._distributed_c10d import (
ModuleNotFoundError: No module named 'torch._C._distributed_c10d'; 'torch._C' is not a package
```

OS X build command:

```
MACOSX_DEPLOYMENT_TARGET=10.11 CC=clang CXX=clang++ python setup.py develop
```

Cmake says

```
-- Building with distributed package: 
  -- USE_TENSORPIPE=False
  -- USE_GLOO=False
  -- USE_MPI=False
--   USE_DISTRIBUTED       : OFF
```

but I would still expect the import to gracefully fail lazily

### Versions

master

cc @malfet @seemethere
DISABLED test_numpy_ref_mps_nn_functional_group_norm_mps_float32 (__main__.TestCommonMPS)
Labels: triaged, module: flaky-tests, skipped, module: mps

Platforms: mac, macos

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/failure/test_numpy_ref_mps_nn_functional_group_norm_mps_float32) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/10102049593).

Over the past 72 hours, it has flakily failed in 2 workflow(s).

**Debugging instructions (after clicking on the recent samples link):**
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Grep for `test_numpy_ref_mps_nn_functional_group_norm_mps_float32`


cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
[inductor] Add more matmul configurations to `TORCHINDUCTOR_MAX_AUTOTUNE=1` mode
Labels: triaged, enhancement, oncall: pt2

https://github.com/pytorch/pytorch/pull/90738 contains 12 configurations for mm/addmm/bmm/etc:

https://github.com/pytorch/pytorch/blob/2b04caf71f09728cb655ccd9c7569d04ecb8b883/torch/_inductor/ops/mm_common.py#L22-L57

While writing this PR I observed speedups by adding/tweaking configurations, and I still think there is room for even more improvements.

When run with `TORCHINDUCTOR_MAX_AUTOTUNE=1`,
those 12 configurations are benchmarked, and the fastest configuration is chosen (at compile time) here:
https://github.com/pytorch/pytorch/blob/2b04caf71f09728cb655ccd9c7569d04ecb8b883/torch/_inductor/select_algorithm.py#L576

Upstream Triton has some similar configurations:
https://github.com/openai/triton/blob/8650b4d1cbc750d659156e2c17a058736614827b/python/triton/ops/matmul.py#L30-L51
plus 100+ more that are generated in `get_configs_io_bound()`.

It then prunes those configurations via a performance model:
https://github.com/openai/triton/blob/8650b4d1cbc750d659156e2c17a058736614827b/python/triton/ops/matmul_perf_model.py#L33

We might be able to use a similar perf model + pruning approach, though we should spend time benchmarking  -- espeically for the mm variants like bmm that upstream Triton doesn't have ops for.

Note we will need to filter out `SPLIT_K>1` configurations, since our template doesn't allow that in order to do epilogue fusions.

We could also try making other tweaks to the templates, such as different load orders (see the GROUP_M reordering).  For small sizes we could also try alternate approaches such as removing the loop and loading things into a single Triton block.



cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
Exporter for ONNX GroupNormalization operator
Labels: module: onnx, triaged

### 🚀 The feature, motivation and pitch

GroupNormalization has been introduced into ONNX opset 18. https://github.com/onnx/onnx/blob/main/docs/Operators.md#GroupNormalization

It would be more efficient if the torch-to-ONNX converter could directly export to this new op instead of using a combination of primitive ops.

### Alternatives

_No response_

### Additional context

_No response_
Umbrella issue for weakref related Dynamo PyTorch test suite failures
Labels: triaged, bug, oncall: pt2

### 🐛 Describe the bug

collected from stack at https://github.com/pytorch/pytorch/pull/90825

### Error logs

_No response_

### Minified repro

_No response_

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh
Umbrella issue for only populate real_value_cache in export test suite fails
Labels: triaged, bug, oncall: pt2

### 🐛 Describe the bug

See https://github.com/pytorch/pytorch/pull/90468 for the list of introduced skips

### Error logs

_No response_

### Minified repro

_No response_

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh
Support arbitrary masks for _nested_tensor_from_mask in nn.TransformerEncoder
Labels: triaged, module: nestedtensor, oncall: transformer/mha

### 🚀 The feature, motivation and pitch

I am working on transformers and am using the Better Transformer. To enjoy the speed improvement, we need to pass a src_key_padding_mask to the nn.TransformerEncoder.

The problem is that, the src_key_padding_mask has to be left-aligned, which means all True values must be in front of all False values. That is the common case in NLP as sentences have different lengths, but in computer vision, the True values in a mask are usually arbitrarily dispersed. I am wondering if you could support transforming a tensor with an arbitrary mask to a nested tensor, and also support transforming the nested tensor back to a normal tensor with 0 paddings according to the same mask.

This feature would be very useful for vision transformers when people want to use masks and enjoy the inference speed improvement for whatever their tasks, for example, dynamic pruning, etc.

### Alternatives

_No response_

### Additional context

_No response_

cc @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are @cpuhrsch @bhosmer @drisspg @mikaylagawarecki @erichan1
[FSDP] Prepare to deprecate `FullyShardedDataParallel.<attrs>`
Labels: oncall: distributed, triaged, module: fsdp

Attributes including `mixed_precision`, `sharding_strategy`, and `cpu_offload` should be applied per `FlatParamHandle` / `FlatParameter`, not at the module level (i.e. `FullyShardedDataParallel` or plain module for composable path).

We should prepare to deprecate using those attributes directly and instead lower the logic to the `FlatParamHandle`. This issue is used as a unified way to track to-dos that indicate a usage that will need to be migrated.

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501
[FSDP] Investigate the need for public `check_is_root()` method
Labels: oncall: distributed, triaged, module: fsdp

Should we expose this method publicly for stable release?

It is currently used for `fsdp_modules()` and `register_comm_hook()`.
https://github.com/pytorch/pytorch/blob/0cd69d7cda15634cb9a86aa18ca759a4ff16b4a0/torch/distributed/fsdp/fully_sharded_data_parallel.py#L473
https://github.com/pytorch/pytorch/blob/0cd69d7cda15634cb9a86aa18ca759a4ff16b4a0/torch/distributed/fsdp/fully_sharded_data_parallel.py#L1744

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501
AOT Autograd doesn't respect no_grad() during input mutations
Labels: triaged, module: aotdispatch

min repro:
```
 @torch.compile(backend='aot_eager')
 def f(x):
     with torch.no_grad():
         x.add_(1)

 x = torch.zeros(2, requires_grad=True)
 # Fails with "RuntimeError: a leaf Variable that requires grad is being used in an in-place operation"
 out = f(x)
```

You can see the problem when I print out:

(1) the graph that dynamo traced
(2) the functionalized, lowered-to-aten, forward graph that AOT Autograd created

(1):
```
class GraphModule(torch.nn.Module):
    def forward(self, x : torch.Tensor):
        # No stacktrace found for following nodes
        _set_grad_enabled = torch._C._set_grad_enabled(False)

        # File: tmp3.py:7, code: x.add_(1)
        add_ = x.add_(1);  x = None

        # No stacktrace found for following nodes
        _set_grad_enabled_1 = torch._C._set_grad_enabled(True)
        return ()
```

(2):
```
class GraphModule(torch.nn.Module):
    def forward(self, primals_1: f32[2]):
        # No stacktrace found for following nodes
        clone: f32[2] = torch.ops.aten.clone.default(primals_1);  primals_1 = None

        # File: tmp3.py:7, code: x.add_(1)
        add: f32[2] = torch.ops.aten.add.Tensor(clone, 1);  clone = None
        return [add]
```

Dynamo adds `set_grad_enabled()` calls into the torch graph, but they disappear (and get executed) when aot autograd retraces the graph and lowers to ATen.

In this case, the last API to be called was `_set_grad_enabled_1 = torch._C._set_grad_enabled(True)` when aot autograd traced the dynamo graph, and so we're stuck with grad being enabled: at runtime when we execute the compiled graph + input mutations, `torch.is_grad_enabled() == True`, and doesn't reflect the fact that the input mutation happened inside of a no_grad() block.

I think the (easy) fix is that when we have input mutations, we should always execute them in `torch.no_grad()` mode. One thing we should double check to be careful about is that we expect errors that occurred in eager mode to still happen in torch.compile() mode. I.e. if a user tries to mutate a leaf that requires grad, we should confirm that this fails at compilation time.
Inference Mode docs
Labels: module: docs, triaged, inference mode

### 🐛 Describe the bug

A part of the explanation on inference mode [docs](https://pytorch.org/cppdocs/notes/inference_mode.html) says:

> A non-view tensor is an inference tensor if and only if it was allocated inside InferenceMode. A view tensor is an inference tensor if and only if the tensor it is a view of is an inference tensor.

where `A view tensor is an inference tensor if and only if the tensor it is a view of is an inference tensor.` is a bit ambiguous to me, does it mean: `A view tensor is an inference tensor if and only if the tensor is a view of an inference tensor.`?

### Versions

master

cc @svekars @carljparker
Support for Pylint
Labels: module: lint, triaged

### 🚀 The feature, motivation and pitch

Using linters is crucial for code safety and clarity. One of the commonly used linters Pylint (e.g. default for Visual Studio Code), fails to find members in the `torch` module as these are supposedly generated (see https://github.com/PyCQA/pylint/issues/7937).  

### Alternatives

Pylint supports external C libraries through the use of the `--extension-pkg-allow-list` option, and this works well e.g. for NumPy, but not for PyTorch.
The common solution people use is to completely ignore `torch` members by using `--generated-members=numpy.*,torch.*`, which is like using no linting for PyTorch at all.

### Additional context

This issue has been repeatedly brought up with no solution, see https://github.com/pytorch/pytorch/issues/24807, https://github.com/pytorch/pytorch/issues/1942, https://github.com/pytorch/pytorch/issues/701. PyLint devs claim they are unable to receive member information from PyTorch, see https://github.com/PyCQA/pylint/issues/7937.
Support `divmod` for tensors
Labels: triaged, module: numpy

### 🚀 The feature, motivation and pitch

Python has a nice builtin function called `divmod` defined as `divmod(x, y) = (x // y, x % y)`. There are some previous issues about supporting it in pytorch https://github.com/pytorch/pytorch/pull/20979 and https://github.com/pytorch/pytorch/issues/18627 but they appear to only work on integers and not tensors.

For example, in pytorch (1.11) I get the following error:
```
>>> ti = torch.randint(10, size=(3,3))
>>> divmod(i, 3)
TypeError: unsupported operand type(s) for divmod(): 'Tensor' and 'int'
```

whereas in numpy I can do this without issue:
```
>>> ni = np.random.randint(10, size=(3,3))
>>> divmod(ni,3)
(array([[2, 3, 0],
        [2, 3, 1],
        [2, 1, 0]]),
 array([[2, 0, 1],
        [2, 0, 1],
        [0, 1, 1]]))
```



### Alternatives

Alternatively pytorch could support a `tensor.divmod` method, but I think it makes more sense to just follow the standard Python API.

### Additional context

One may have to think about how broadcasting would work if you called `divmod(x, y)` where both `x` and `y` were tensors. However, in general it should probably just follow the equation above `divmod(x, y) = (x // y, x % y)`.

cc @mruberry @rgommers
[Composable API] Add `fully_shard` state dict unit test after manual "wrapping" is supported
Labels: oncall: distributed, triaged, module: fsdp

We should remember to add `fully_shard` state dict tests after manual "wrapping" is supported.

https://github.com/pytorch/pytorch/pull/90767#discussion_r1047482214

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501
[FSDP] Investigate `test_fsdp_pure_fp16.py` inaccuracy
Labels: oncall: distributed, triaged, module: fsdp

`test_fsdp_pure_fp16.py` has been mostly flaky or broken. For now, we are disabling the parameter check after the training step. However, this means that the backward pass could be incorrect for pure FP16 models using FSDP.



cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501
Extend "torch.utils.cpp_extension.load" for both lib64 and **lib**
Labels: module: cpp-extensions, triaged

### 🚀 The feature, motivation and pitch

Currently, `torch.utils.cpp_extension.load()` has an assumption that CUDA lib directory is named "lib64" under `${CUDA_HOME}`. (Doc: https://pytorch.org/docs/stable/cpp_extension.html#torch.utils.cpp_extension.load)
However, when using conda to install cuda, it is actually under conda env's "lib" but not "lib64".
Can we extend `torch.utils.cpp_extension.load()` to also consider this case?
Otherwise users can only hack this by linking "lib" to "lib64".

### Alternatives

_No response_

### Additional context

_No response_

cc @malfet @zou3519
Pytorch 2.0 document issue
Labels: triaged

### 📚 The doc issue

Hello, according to the brand new pytorch 2.0 introduction document, it seems that ddp mode is only compatible with
static_graph=True and find_unused_parameters=True for now.
![image](https://user-images.githubusercontent.com/15059072/207267397-c03a1562-808f-44a8-8531-42289b99b855.png)

However, in the detailed post, DDP needs to be run with static_graph=False.
![image](https://user-images.githubusercontent.com/15059072/207267791-8853531e-b998-4ea0-b495-8c5e036e4978.png)

So I suppose the right one is DDP needs to be run with static_graph=False, as said in the detailed post? Could you please check this, thanks.

Doc addr:
https://pytorch.org/get-started/pytorch-2.0/#torchinductor-fast-codegen-using-a-define-by-run-ir

### Suggest a potential alternative/fix

_No response_
Adam (fused=True) issues
Labels: module: performance, module: optimizer, module: cuda, triaged

### 🐛 Describe the bug

I am having serious doubt on the 1.13 Adam(fused=true) implementation.
First it should be faster than fused=False, in our case it is not, but also the numerical seems off.

Adam(fused=False)
[2022-12-09 12:21:37,894 INFO] Start training loop and validate every 10000 steps...
[2022-12-09 12:23:49,851 INFO] Step 100/100000; acc: 16.1; ppl: 6103.2; xent: 8.7; lr: 0.00002; sents: 169920; bsz: 8605/10659/340; 32607/40388 tok/s; 132 sec;
[2022-12-09 12:25:22,693 INFO] Step 200/100000; acc: 21.4; ppl: 957.8; xent: 6.9; lr: 0.00005; sents: 153056; bsz: 8588/10584/306; 46250/57000 tok/s; 225 sec;
[2022-12-09 12:26:55,347 INFO] Step 300/100000; acc: 25.8; ppl: 414.5; xent: 6.0; lr: 0.00007; sents: 139424; bsz: 8796/10695/279; 47467/57717 tok/s; 317 sec;

Adam(fused=True)
[2022-12-09 12:29:46,843 INFO] Step 100/100000; acc: 15.9; ppl: 7363.9; xent: 8.9; lr: 0.00002; sents: 169920; bsz: 8605/10659/340; 32465/40212 tok/s; 133 sec;
[2022-12-09 12:31:19,458 INFO] Step 200/100000; acc: 18.1; ppl: 1606.6; xent: 7.4; lr: 0.00005; sents: 153056; bsz: 8588/10584/306; 46363/57140 tok/s; 225 sec;
[2022-12-09 12:32:53,634 INFO] Step 300/100000; acc: 22.1; ppl: 702.1; xent: 6.6; lr: 0.00007; sents: 139424; bsz: 8796/10695/279; 46700/56784 tok/s; 319 sec;

Fusedadam apex O2
[2022-12-09 12:35:51,087 INFO] Step 100/100000; acc: 16.1; ppl: 6136.2; xent: 8.7; lr: 0.00002; sents: 169920; bsz: 8605/10659/340; 36301/44964 tok/s; 119 sec;
[2022-12-09 12:37:11,869 INFO] Step 200/100000; acc: 21.4; ppl: 959.9; xent: 6.9; lr: 0.00005; sents: 153056; bsz: 8588/10584/306; 53155/65510 tok/s; 199 sec;
[2022-12-09 12:38:33,601 INFO] Step 300/100000; acc: 25.8; ppl: 415.1; xent: 6.0; lr: 0.00007; sents: 139424; bsz: 8796/10695/279; 53810/65430 tok/s; 281 sec;

Fusedadam apex old legacy code
[2022-12-09 12:42:04,245 INFO] Step 100/100000; acc: 16.1; ppl: 6136.2; xent: 8.7; lr: 0.00002; sents: 169920; bsz: 8605/10659/340; 37141/46004 tok/s; 116 sec;
[2022-12-09 12:43:21,080 INFO] Step 200/100000; acc: 21.4; ppl: 959.9; xent: 6.9; lr: 0.00005; sents: 153056; bsz: 8588/10584/306; 55886/68875 tok/s; 193 sec;
[2022-12-09 12:44:41,898 INFO] Step 300/100000; acc: 25.8; ppl: 415.1; xent: 6.0; lr: 0.00007; sents: 139424; bsz: 8796/10695/279; 54419/66169 tok/s; 274 sec;

As you can see the second one (Adam(fused=True) is not faster, clearly much slower than the Apex implementations, and accuracy, ppl, loss are off compared to the 3 others.





### Versions


pytorch 1.13
Ubuntu 20.04
OpentNMT-py 3.0.1

cc @ngimel @vincentqb @jbschlosser @albanD @janeyx99
Importing torch causes segfault when using python installed with conda
Labels: high priority, triage review, needs reproduction, oncall: binaries, module: crash

### 🐛 Describe the bug

Cross-posted to: https://discuss.pytorch.org/t/importing-torch-causes-segfault-when-using-python-installed-with-conda/168212

I create a conda environment with: `conda create -y -n dev python=3.7`.
In install torch with:
```
conda run -n dev pip install torch==1.14.0.dev20221027+cu116 --pre --extra-index-url https://download.pytorch.org/whl/nightly/cu116
```

running: `python3 -c import torch` gives a segfault.
Here's the gdb backtrace:

```
-c (gdb) r -c 'import torch'
Starting program: /opt/conda/envs/dev/bin/python3 -c 'import torch'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after fork from child process 31265]

Program received signal SIGSEGV, Segmentation fault.
0x000055555564d27c in type_name (context=<optimized out>, type=0x555558aa7630)
    at /home/conda/feedstock_root/build_artifacts/python_1635226063427/work/Objects/typeobject.c:433
433	/home/conda/feedstock_root/build_artifacts/python_1635226063427/work/Objects/typeobject.c: No such file or directory.
```

### Versions

The script segfaults.

cc @ezyang @gchanan @zou3519 @seemethere @malfet
Adopt full_backward_pre_hook in DDP
Labels: oncall: distributed, triaged, module: ddp

### 🚀 The feature, motivation and pitch

Since https://github.com/pytorch/pytorch/pull/86700 has landed supporting the full backward pre hook, we should enable this with DDP for a true module level pre-backward hook and eliminate things such as _DDPSink.

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
test_copy_broadcast
Labels: triaged

Repro:
```
PYTORCH_TEST_WITH_INDUCTOR=1 python test/test_torch.py -k test_copy_broadcast
```
```
FAIL: test_copy_broadcast (__main__.TestTorch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/scratch/binbao/work/pytorch/test/test_torch.py", line 7650, in test_copy_broadcast
    self.assertRaises(RuntimeError, lambda: torch.zeros(5, 6).copy_(torch.zeros(30)))
  File "/scratch/binbao/work/pytorch/torch/testing/_internal/common_utils.py", line 2927, in assertRaises
    return super().assertRaises(expected_exception, *args, **kwargs)
AssertionError: RuntimeError not raised by <lambda>
```

Note: there are two tests with this name, we can rename it.
PixelShuffle/Unshuffle Channels Last Support on GPU
Labels: triaged, module: memory format

### 🚀 The feature, motivation and pitch

According to https://github.com/pytorch/pytorch/wiki/Operators-with-Channels-Last-support pixel_shuffle & pixel_unshuffle have channels last support on CPU, but not GPU. 

This appears to be the case based on anecdotal training performance for networks using PixelShuffle/Unshuffle & channels last (sorry no hard numbers on hand right now). 

It is a very useful op that can really make a difference in high-fidelity pixel-level reconstruction / segmentation / salient object detection type tasks, and would be greatly appreciated! 

### Alternatives

One could probably write a hand-crafted python nn.Module to detect if the incoming tensor is channels last & then do the reshuffling "manually"? 

### Additional context

@bdhirsh in https://github.com/pytorch/pytorch/pull/86608 "Is pixel_shuffle/unshuffle commonly used with cuda? Maybe we should just add a fast cuda kernel for it if that's the case." - yes please! 

cc @jamesr66a
`torch.empty` produces incorrect tensors with `layout=sparse_csr|sparse_csc` on the CPU
Labels: module: sparse, module: docs, module: cpu, triaged

### 🐛 Describe the bug

I would argue that this issue is potentially rather critical given how many sparse compressed algorithms rely on it.
A potential cause of https://github.com/pytorch/pytorch/issues/90693.

This also raises a question of the quality of the corresponding tests which probably do not test extreme cases.

```python
In [1]: import torch

In [2]: torch.empty(2, 2, layout=torch.sparse_csr)
<ipython-input-2-76b85c15c1f2>:1: UserWarning: Sparse CSR tensor support is in beta state. If you miss a functionality in the sparse tensor support, please submit a feature request to https://github.com/pytorch/pytorch/issues. (Triggered internally at /home/nik/git/Quansight/pytorch/aten/src/ATen/SparseCsrTensorImpl.cpp:54.)
  torch.empty(2, 2, layout=torch.sparse_csr)
Out[2]: 
tensor(crow_indices=tensor([139874393660768, 139874393660768, 139870036047680]),
       col_indices=tensor([], size=(0,)),
       values=tensor([], size=(0,)), size=(2, 2), nnz=0,
       layout=torch.sparse_csr)

In [3]: torch.empty(2, 2, layout=torch.sparse_csc)
Out[3]: 
tensor(ccol_indices=tensor([2314885530818453536, 8319099727289262112,
                            7591715963103638117]),
       row_indices=tensor([], size=(0,)),
       values=tensor([], size=(0,)), size=(2, 2), nnz=0,
       layout=torch.sparse_csc)

In [4]: torch.empty(2, 2, layout=torch.sparse_csr, device='cuda')
Out[4]: 
tensor(crow_indices=tensor([0, 0, 0]),
       col_indices=tensor([], size=(0,)),
       values=tensor([], size=(0,)), device='cuda:0', size=(2, 2), nnz=0,
       layout=torch.sparse_csr)

In [5]: torch.empty(2, 2, layout=torch.sparse_csc, device='cuda')
Out[5]: 
tensor(ccol_indices=tensor([0, 0, 0]),
       row_indices=tensor([], size=(0,)),
       values=tensor([], size=(0,)), device='cuda:0', size=(2, 2), nnz=0,
       layout=torch.sparse_csc)

```

### Versions

Current master.

cc @pearu @cpuhrsch @amjames @bhosmer @svekars @carljparker @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
[FSDP][BE] `test_fsdp_comm_hooks.py` cleanup
Labels: oncall: distributed, triaged, module: fsdp

`test_fsdp_comm_hooks.py` takes too long to run (374 sec / 6:14 min on AWS cluster -- for perspective, `test_fsdp_mixed_precision.py` takes 524 sec / 8:44 min). This issue outlines some ways to make it run faster.

1. We should only test the FSDP model with nested wrapping since nested wrapping is orthogonal to the communication hook. This involves removing the `has_wrapping` argument from `Net` and each unit test in favor of only the `has_wrapping=True` path.
2. We should subtest some of the configurations. For example, `FULL_SHARD` and `SHARD_GRAD_OP` take exact same code path (both now and in the future) since they both require a reduce-scatter. These do not need to run in new processes.
3. We should consider if we want to run separate tests for both the FP16 and BF16 compression hooks. Their logic is identical except the dtype.

---

Some other miscellaneous items:
- This comment does not seem to align with the `assertEqual` since it checks the parameter, not its gradient:
https://github.com/pytorch/pytorch/blob/c7d2fb7f8649b798d805c18f3c1d21abd0bdb2dd/test/distributed/fsdp/test_fsdp_comm_hooks.py#L399-L401


cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501
`torch.inverse` multi-threading RuntimeError: lazy wrapper should be called at most once
Labels: triaged, module: multithreading, module: linear algebra

### 🐛 Describe the bug

`torch.inverse` and the related modules won't work with multi-threading:

```py
import concurrent.futures
import torch

def run_task(req):
    out = torch.inverse(torch.eye(4, device="cuda:0"))
    return req, out


infer_tasks = [1, 2, 3, 4, 5, 6, 7, 8]

with concurrent.futures.ThreadPoolExecutor(4, "BatchTask") as executor:
    futures = {executor.submit(run_task, t) for t in infer_tasks}
    for fut in concurrent.futures.as_completed(futures):
        print("outcome", fut.result())

```

run this script will raise a runtime error:
```
Traceback (most recent call last):
  File "my_test.py", line 15, in <module>
    print("outcome", fut.result())
  File "/home/anaconda3/envs/py38/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/home/anaconda3/envs/py38/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/home/anaconda3/envs/py38/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "my_test.py", line 5, in run_task
    out = torch.inverse(torch.eye(4, device="cuda:0"))
RuntimeError: lazy wrapper should be called at most once
```

### Versions

Collecting environment information...
PyTorch version: 1.13.0+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2
Libc version: glibc-2.27

Python version: 3.8.13 (default, Mar 28 2022, 11:38:47)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-135-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.4.48
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA TITAN Xp COLLECTORS EDITION
GPU 1: NVIDIA TITAN Xp COLLECTORS EDITION


(monai ref: https://github.com/Project-MONAI/MONAI/issues/5696)

cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano
Operator overload priority should not rely on static initialization order
Labels: oncall: jit

### 🐛 Describe the bug

In our internal build, calling `torch.ops.aten.add()` on two scalar float tensors will incorrectly return a Python native integer (instead of a float tensor).

### Root Cause:
I found this is because the operator overloads for `aten::add` looks like this:
```
print(torch._C._jit_get_operation('aten::add'))

(<built-in method add of PyCapsule object at 0x7fb54234aae0>, ['t', 'str', 'int', 'complex', 'float', 'int_complex', 'complex_int', 'float_complex', 'complex_float', 'int_float', 'float_int', '', 'Tensor', 'out', 'Scalar', 'Scalar_out'])
```

while in a correct build, the output should look like this:
```
(<built-in method add of PyCapsule object at 0x7fb231070930>, ['Tensor', 'Scalar', 'out', 'Scalar_out', 't', 'str', 'int', 'complex', 'float', 'int_complex', 'complex_int', 'float_complex', 'complex_float', 'int_float', 'float_int', ''])
```
i.e., the `Tensor` overload must come before the "int" overload.

If I'm not mistaken, the operator overload priority is defined by the order they are registered - which is defined by the ambiguous static initialization order. In particular, the `Tensor` overload for `aten::add` is registered in https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/templates/RegisterSchema.cpp while the `int` overload is registered in https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/runtime/register_prim_ops.cpp.


### Versions

I'm building pytorch 1.13.1-rc1

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
AttributeError: 'tuple' object has no attribute 'grad'
Labels: triaged

```
PYTORCH_TEST_WITH_DYNAMO=1 python test/test_nn.py -k test_AdaptiveAvgPool1d_cuda --verbose
```
passes with graph break;

```
PYTORCH_TEST_WITH_DYNAMO=1 python test/test_nn.py -k test_Adaptive --verbose
```
fails with
```
test_AdaptiveAvgPool1d_cuda (__main__.TestNN) ... Traceback (most recent call last):
  File "<string>", line 2, in <lambda>
AttributeError: 'tuple' object has no attribute 'grad'
ERROR RUNNING GUARDS <graph break in _backward> /scratch/binbao/work/pytorch/test/test_nn.py:90
  ___guarded_code.valid and
  ___check_type_id(input.grad, 109098144) and
  ___check_tensors(input, input.grad)
NULL ERROR: /scratch/binbao/work/pytorch/torch/csrc/dynamo/eval_frame.c:251
```
Multiprocessing "Error Propagation" doesn't work for FullyShardedDataParallelism.
Labels: oncall: distributed, module: fsdp

https://github.com/pytorch/pytorch/blob/b7dfbf876f640591c399de2c92f523432b3455a1/torch/multiprocessing/spawn.py#L131

The multiprocessing spawn error propagation relies on sending a "process.terminate" call to request that each process should stop executing. Terminate sends a _request_ that should be honored by the receiving process - but in FSDP, that request is not honored. If one process OOMs, but the others do not, that process will die and the terminate call will be sent out while the others simply end up waiting indefinitely. Due to how FSDP works, I'm assuming they're hanging waiting for the weight all-gather phase, and aren't willing to accept the terminate signal. This is a serious blocking point for FSDP users, since the typical setting (large-model training) will often lead to CUDA OOMs that cannot be handled or propagated correctly.

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Bfloat16 tensor .numpy() support 
Labels: triaged, module: numpy, module: bfloat16

### 🚀 The feature, motivation and pitch

Numpy doesn't support bfloat16, and doesn't plan to do so. The effect of this is that code that makes any `tensor.numpy()` call breaks when you make it use bfloat16. I was thinking that bfloat16 getting outputted to `np.float32` would make sense, as it just keeps the exponent and ads a few mantissa bits. This must be very quick. This would make all code that is supported with float32 or float16 be compatible with bfloat16 out of the box, and feels like reasonable behavior to me.

### Additional context

The `to_numpy` function seems to be here https://github.com/pytorch/pytorch/blob/master/torch/csrc/utils/tensor_numpy.cpp#L159

and the function that decides the output `np.dtype` seems to be here:
https://github.com/pytorch/pytorch/blob/master/torch/csrc/utils/tensor_numpy.cpp#L267


cc @mruberry @rgommers
[discussion, idea] Batched, vectorized base64 decoding / encoding + maybe RLE decoding / encoding
Labels: feature, triaged, module: vision, module: nestedtensor

### 🚀 The feature, motivation and pitch

Discussed in context of scriptable base64 decoding here: https://github.com/pytorch/vision/issues/6878#issuecomment-1343439120, http://www.alfredklomp.com/programming/sse-base64/, https://r-libre.teluq.ca/1362/1/base64.pdf 

But then I thought that base64 can be implemented in a vectorized way - and probably some design could probably do batching/parallelization (especially if inputs/outputs of same/padded size or nestedtensor/tensorlist)

The usecases are probably limited but could include some visualization utils that need to encode many images / audio files to base64 for further burning into html as data-uris for self-containedness

### Alternatives

_No response_

### Additional context

_No response_

cc @datumbox @vfdev-5 @pmeier @cpuhrsch @jbschlosser @bhosmer @drisspg @mikaylagawarecki
[RFC] Add torch.backends.tbb.is_available()
Labels: triaged, enhancement, module: tbb

### 🚀 The feature, motivation and pitch

Add a similar API to `torch.backends.openmp.is_available()` for TBB, `torch.backends.tbb.is_available()`. 

### Usage
The expected usage of `torch.backends.tbb.is_available()` is of similar to that of `torch.backends.openmp.is_available()`. I assume users use `torch.backends.openmp.is_available()` to verify their pytorch config, and `torch.backends.tbb.is_available()` can be of similar usage to that - users can verify whether TBB is enabled on their PyTorch or not with this API rather than using `TBB_VERSION=1`, `MKL_VERBOSE=1`, etc. 

```python
import torch
torch.backends.tbb.is_available() # True if TBB enabled, False otherwise
```

### Alternatives

_No response_

### Additional context

_No response_
Embedding dynamic quantization is not documented and hard to use
Labels: oncall: quantization, triaged

### 🐛 Describe the bug

PyTorch quantization supports dynamic quantization of embeddings.  For example, here is a working code snippet from `test/quantization/eager/test_quantize_eager_ptq.py`:

```
        class EmbeddingBagWithLinear(torch.nn.Module):
            def __init__(self):
                super().__init__()
                self.emb = torch.nn.EmbeddingBag(num_embeddings=10, embedding_dim=12,
                                                 include_last_offset=True, scale_grad_by_freq=False, mode='sum')
                self.fc = torch.nn.Linear(5, 5)

            def forward(self, indices, offsets, linear_in):
                return self.emb(indices, offsets), self.fc(linear_in)
        model = EmbeddingBagWithLinear().eval()

        qconfig_dict = {
            torch.nn.EmbeddingBag : float_qparams_weight_only_qconfig,
            torch.nn.Linear: default_dynamic_qconfig
        }
        indices = torch.tensor([9, 6, 5, 7, 8, 8, 9, 2, 8, 6, 6, 9, 1, 6, 8, 8, 3, 2, 3, 6, 3, 6, 5, 7, 0, 8, 4, 6, 5, 8, 2, 3])
        offsets = torch.tensor([0, 19, 20, 28, 28, 32])
        q_model = quantize_dynamic(model, qconfig_dict)
```

It's currently impossible for the user to figure this out without reading the source code.  We should make it easy for the user to read https://pytorch.org/docs/master/quantization.html and https://pytorch.org/docs/master/generated/torch.quantization.quantize_dynamic.html#torch.quantization.quantize_dynamic and know how to call the `quantize_dynamic` API with the correct syntax for dynamically quantizing embeddings.

### Versions

master

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @jgong5 @Xia-Weiwen @leslie-fang-intel
AOT Autograd should differentiate intermediate leaves.
Labels: triaged, oncall: pt2, module: aotdispatch, module: dynamo

This failure was first noticed by @desertfire in https://github.com/pytorch/pytorch/issues/93589.

A minimum repro looks like this:
```
@torch.compile(...)
def f(x):
    leaf = torch.ones(2, requires_grad=True)
    return leaf, leaf * 2

leaf, out = f(torch.ones(2, requires_grad=True))
out.backward()

# This incorrectly prints None. autograd-ing through "out" should have populated `leaf.grad`.
print(leaf.grad)
```

This case (hopefully) is pretty uncommon: Most autograd leaves in user code come from model parameters, which are always guaranteed to be function inputs. The bad situation happens when you create a differentiable leaf tensor through a factory function, inside of a graph (so it's not a graph input), and return both (leaf, tensor-computation-on-leaf) as outputs.

The problem is that when AOT Autograd traces a fwd + bwd graph, it only differentiates w.r.t graph inputs. One solution (from talking with @ezyang) would be to detect this situation, and "lift" leaves into graph inputs inside of AOT Autograd.

Ideally, we should do this in a way that doesn't add too much overhead to compilation (i.e. it would be nice handling this situation didn't require running the forward() an extra time during compilation).

cc @ezyang @soumith @msaroufim @wconstab @ngimel @mlazos @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
Could not run 'aten::as_strided' with arguments from the 'Metal' backend.
Labels: triaged, module: ios

### 🚀 The feature, motivation and pitch

I would like to run Yolov7 (which includes keypoint detection) in iOS using GPU acceleration (with the metal framework)
[see github repo here](https://github.com/WongKinYiu/yolov7)

However, just subbing in the [model](https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7-w6-pose.pt) into the Hello-world Metal example

https://github.com/pytorch/ios-demo-app/tree/master/HelloWorld-Metal

I get the following error

`HelloWorld-Metal[67812:4283712] Could not run 'aten::as_strided' with arguments from the 'Metal' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process `



### Alternatives

I have tried converting to coreml, which also fails.

### Additional context

_No response_
Unable to link LibTorch against CUDA and CUDNN statically
Labels: triaged, module: static linking

### 🐛 Describe the bug

Hey,

So far I have managed to link against IntelMKL statically, now I am fighting with CUDA and CUDNN. While with CUDNN the issue is obvious (libcudnn_static.a got split into several other archives, https://github.com/pytorch/pytorch/issues/81692), I still struggle with CUDA itself.
Reading CMake logic in the repository, one can see that PyTorch will prefer static libs over dynamic ones: https://github.com/pytorch/pytorch/blob/c371542efc31b1abfe6f388042aa3ab0cef935f2/cmake/Modules_CUDA_fix/upstream/FindCUDA.cmake#L882 .

However in my case, always dynamic ones are picked.

Using PyTorch 1.13 sources, I call:
```
export BUILD_DOCS=OFF
export BUILD_TEST=OFF
export BLAS=MKL
export USE_CUDA=ON
export USE_DISTRIBUTED=OFF
export USE_FBGEMM=OFF
export USE_MKLDNN=OFF
export USE_NINJA=ON
export USE_PYTORCH_QNNPACK=OFF
export USE_QNNPACK=OFF
export USE_SYSTEM_NCCL=ON
export USE_XNNPACK=OFF
export TORCH_CUDA_ARCH_LIST='8.6'

# Static linkage
export USE_STATIC_MKL=1
export CAFFE2_STATIC_LINK_CUDA=ON
export CUDA_USE_STATIC_CUDA_RUNTIME=ON
#export USE_STATIC_CUDNN=1 # TODO: doesn't work, drops CUDNN completely
                          # https://github.com/pytorch/pytorch/issues/81692
    
python3 tools/build_libtorch.py
```

### Versions

1.13
[Dispatchable Collectives] Follow up tasks
Labels: oncall: distributed, triaged

Creating this issue to aggregate a list of follow up tasks for https://github.com/pytorch/pytorch/pull/88330 based on BE changes identified while developing the feature as well as follow up work items for the feature itself based on review feedback (thanks @kwen2501!). The list of follow up tasks can be farmed out as separate GH issues.

BE changes:
- [ ] Bug with `@require_backends_available` test decorator. It requires that ALL backends (GLOO, NCCL, UCC) in that list be available or the test skips. This causes issues since some jobs do not have UCC built by default (also local development UCC is not on by default) so the test is skipped for that job
- [ ] [linux-bionic-cuda11.6-py3.10-gcc7-bazel-test / build-and-test](https://github.com/pytorch/pytorch/actions/runs/3635142627/jobs/6133965664#logs) does not build with GLOO enabled. It should be enabled by default.
- [x] Remove `Backend.TCP`, there is no longer a TCP backend.
- [ ] Investigate removing `create_device` from ProcessGroupGloo as there is no documented use for it and we don't want users to create python ProcessGroupGloo instances. https://github.com/pytorch/pytorch/pull/88330#discussion_r1044744269
- [x] Figure out why NCCL_DESYNC_DEBUG environment variable is getting set in our tests
- [ ] Fix destruction fiasco in `ProcessGroupNCCL` https://github.com/pytorch/pytorch/issues/90848


Feature related follow up tasks:
- [x] Update ProcessGroup.hpp to call Ops.cpp directly, remove the pybinded function definitions for ProcessGroup https://github.com/pytorch/pytorch/issues/90932
- [ ] Lazy initialization of backend
- [x] Remove Backend from the pybind definition (only expose ProcessGroup)
- [x] Handle case of PythonProcessGroup, alert extension writers to migration for C++ PG extensions in order to use the dispatcher
- [ ] Update C++ classes from ProcessGroupGloo, ProcesGroupNCCL, etc into the respective BackendGloo, BackendNCCL... 
- [x] https://github.com/pytorch/pytorch/issues/90659
- [ ] Update tensor to be a static member of ProcessGroup for `barrier` https://github.com/pytorch/pytorch/pull/88330#discussion_r1043939495
- [ ] Update https://pytorch.org/tutorials/intermediate/process_group_cpp_extension_tutorial.html

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @kwen2501 @awgu