gopikrsmscs/torch-issues huggingface dataset:
[feature request] Specialized memory layouts and wide blocked/tiled dtypes for cublasLt/onednn: e.g. torch.float16x32 / torch.int8x32 (akin to torch.quint2x4)
Labels: feature, triaged, module: memory format

### 🚀 The feature, motivation and pitch

I found a blog post explaining how to get speedups by using int8 gemm kernels on CUDA: https://www.speechmatics.com/company/articles-and-news/fast-and-accurate-gpu-quantization-for-transformers

It mentions several specialized memory layouts to maximize perf of cublas gemm ops (found them documented in https://docs.nvidia.com/cuda/cublas/#cublasltorder-t):
- `CUBLASLT_ORDER_COL32` (mentioned as the most performant memory layout for cublasLt int8 gemms)
- `CUBLASLT_ORDER_COL4_4R2_8C`
- `CUBLASLT_ORDER_COL32_2R_4R4`

These memory formats/layouts seem important for max-perf cublasLt for int8, as evidenced by this blog post and FasterTransformer kernels. @vkuzo are these formats supported by `_int_mm` in https://github.com/pytorch/pytorch/pull/96685?

Is `CUBLASLT_ORDER_COL32` logically representable in PyTorch? Other formats?

For int8, I would understand this means as if there was dtype holding 32-sized tiles/tuples of `torch.int8`  (32 bytes = 256 bits), currently the widest dtype is `complex128` (supposed to hold a tuple of two float64 i.e. 16bytes = 128 bits). (and then these tuples are stored in column-major)

So IMO a minimal way to support it would be:
- introducing uninterpreted (except torch.to/torch.view/print) dtypes `torch.int8x32`, maybe `torch.float16x32` and so forth
  - https://github.com/pytorch/pytorch/pull/94992 introduced `torch.bits2x4` (there also exists torch.quint2x4`) and `torch.bits16`, so maybe at least `torch.bits2048` could be added?
- implementing int8_tensor.to(dtype = torch.int8x32) and maybe tensor.view(torch.int8x32)
- an alternative way might be to introduce separate `memory_format=` supporting conversions to these layouts via this avenue (especially for `CUBLASLT_ORDER_COL32_2R_4R4`?)

In that blog post they are fusing the quantization + conversion into a LayerNorm kernel, but I guess this can be introduced later (if needed at all).

Also, FasterTransformer has many COL32 kernels: https://github.com/NVIDIA/FasterTransformer/blob/main/src/fastertransformer/kernels/unfused_attention_int8_kernels.cu

It also appears that similar tiled/blocked memory formats dtypes are used for MKLDNN: https://oneapi-src.github.io/oneDNN/v1.0/dev_guide_understanding_memory_formats.html and are probably already supported by calling `.to_mkldnn()` / weight prepacking. I wonder if supporting such tupled-dtypes could be good and unify the layouts for cublasLt and MKLDNN (cc @jamesr66a @mingfeima)

For onednn it is:
- nChw16c
- nChw8c

@jerryzh168 @vkuzo @ngimel this is probably also related to int8 gemm code paths as we discussed in https://github.com/pytorch/pytorch/issues/69364

(256 bit AVX2's equivalent is float32x8, and for AVX512 it's float32x16
DISABLED test_cuda_memory_leak_detection (__main__.TestCudaMultiGPU)
Labels: module: cuda, triaged, module: flaky-tests, skipped

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_cuda_memory_leak_detection&suite=TestCudaMultiGPU) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/14816707226).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 2 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_cuda_memory_leak_detection`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_cuda_multigpu.py`

cc @ptrblck
[compile] DDPOptimizer + activation checkpointing not supported
Labels: module: checkpoint, triaged, module: ddp, oncall: pt2

Activation checkpointing is supported via higher order operators. However, DDP optimizer backend in Dynamo does not work with higher order operators yet. The workaround is to disable DDP optimizer using `torch._dynamo.config.optimize_ddp = False`. However, the tradeoff is bad performance because there will be just one bucket for the entire Dynamo graph.

No plan to support it yet. We will revisit if this is a common ask. 

cc @ezyang @msaroufim @wconstab @bdhirsh
Extend ATen op benchmarks
Labels: module: cpu, triaged, open source, module: amp (automated mixed precision), release notes: benchmark, ciflow/mps, ciflow/inductor

## Summary
Extended ATen op benchmarking coverage:
1. Enabled benchmarking of more ATen ops
2. Enabled benchmarking with BF16 dtype as well, if `AVX512_BF16` ISA is supported (on only Linux).
3. Enabled benchmarking of inplace variants, wherever possible.
4. Enabled benchmarking with channels-last memory layout, wherever applicable

## Verification
Currently, no CI job runs this code, so maybe we can create a new weekly CI job on TorchBench.



cc @jgong5 @mingfeima @XiaobingSuper @ashokei @jingxu10 @mcarilli @ptrblck @leslie-fang-intel @jiayisunx @peterbell10 
cc @albanD @ezyang
I propose a new overview section in the documentation
Labels: triaged, topic: docs

### 📚 The doc issue

<<Editted - 11/07/23 I found a lot of this material already existed so have reworded this issue >>

I am new to PyTorch and what would have been useful to me was an overview of the project, at level higher than the code.
I think that this would be an 'and' not an 'or' with the existing docs and may help users who like this sort of conceptual view over the project. 

An overview of a typical PyTorch application, main and training data, model creation, loss and optimizer , training loop and test. I know these are also standard ML concepts, but I think it would help as they tie to specific PyTorch named concept


### Suggest a potential alternative/fix

_No response_
Failure in optimize_for_mobile when using conv1d(..., padding='same')
Labels: triaged, oncall: mobile

### 🐛 Describe the bug

If a module uses `torch.nn.functional.conv1d` with the argument `padding='same'`, `optimize_for_mobile` fails. If an integral padding argument is provided, it works as expected.

**Example:**
```python
import torch
from torch.utils.mobile_optimizer import optimize_for_mobile


class TestModule(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, input: torch.Tensor, weight: torch.Tensor) -> torch.Tensor:
        return torch.nn.functional.conv1d(input, weight, padding='same')


module = TestModule()
module.eval()
scripted_module = torch.jit.script(module)
mobile_module = optimize_for_mobile(scripted_module)
```

**Output:**
```
Traceback (most recent call last):
  File "/Users/gtyukasz/TorchFail/torch_fail.py", line 16, in <module>
    mobile_module = optimize_for_mobile(scripted_module)
  File "/Users/gtyukasz/TorchFail/venv/lib/python3.10/site-packages/torch/utils/mobile_optimizer.py", line 62, in optimize_for_mobile
    optimized_cpp_module = torch._C._jit_pass_optimize_for_mobile(
RuntimeError: isList() INTERNAL ASSERT FAILED at "/Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/core/ivalue_inl.h":2034, please report a bug to PyTorch. Expected GenericList but got String
```

### Versions

```
PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.4.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.12 (main, Jun 20 2023, 19:43:52) [Clang 14.0.3 (clang-1403.0.22.14.1)] (64-bit runtime)
Python platform: macOS-13.4.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] torch==2.0.1
[conda] Could not collect
```
F.adaptive_avg_pool3d(input, 1) returns infinity in half precision
Labels: module: numerical-stability, module: nn, module: cpu, triaged

### 🐛 Describe the bug

When training a network using half precision, I experience a nan values.
When looking into the network I saw that F.adaptive_avg_pool3d(input, 1) outputs infinity on some inputs.
[bad_input.zip](https://github.com/pytorch/pytorch/files/11950096/bad_input.zip)

It reproduced using different pytorch versions - 1.10-1.12
Also reproduced in different cuda versions - 10-11.7.


code reproduce:
```
import torch
import torch.nn.functional as F

input = torch.load("bad_input.pt").half()
result = F.adaptive_avg_pool3d(bad_input, 1)
print(torch.max(result)) 
```
The print will output infinity.

Possible duplicate issue in https://github.com/pytorch/pytorch/issues/52719.
However, the workadound of x.mean(dim={2,3,4}) does not reproduce the same output as adaptive_avg_pool3d.



### Versions

pip3] numpy==1.22.1
[pip3] pytorch-ranger==0.1.1
[pip3] pytorchvideo==0.1.5
[pip3] torch==1.10.1+cu113
[pip3] torch-optimizer==0.3.0
[pip3] torchmetrics==0.9.3
[pip3] torchvision==0.11.2+cu113

And also here:
pip3] numpy==1.21.5
[pip3] pytorch-ranger==0.1.1
[pip3] pytorchvideo==0.1.5
[pip3] torch==1.13.1+cu117
[pip3] torch-optimizer==0.3.0
[pip3] torchaudio==0.13.1+cu117
[pip3] torchelastic==0.2.0
[pip3] torchmetrics==0.9.3
[pip3] torchtext==0.13.0
[pip3] torchvision==0.14.1+cu117


cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
version libcudnn_ops_infer.so.8 not defined in file libcudnn_ops_infer.so.8 with link time reference
Labels: module: cuda, triaged


## Issue description

when I load one torch model ,it works,but when I load two torch models,it shows：
```
Could not load library libcudnn_cnn_infer.so.8. Error: /usr/local/cuda/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8: symbol _ZN5cudnn3ops18GetInternalStreamsEP12cudnnContextiPP11CUstream_st, version libcudnn_ops_infer.so.8 not defined in file libcudnn_ops_infer.so.8 with link time reference
Please make sure libcudnn_cnn_infer.so.8 is in your library path!
```
but I can found libcudnn_cnn_infer.so.8 file in this path:/usr/local/cuda/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8

## Code example
```python
from TTS.api import TTS
from faster_whisper import WhisperModel
tts = TTS(model_name="tts_models/multilingual/multi-dataset/your_tts", progress_bar=True, gpu=True)
tts_clone = TTS(model_name="voice_conversion_models/multilingual/vctk/freevc24", progress_bar=True, gpu=True)
whispter_model = WhisperModel(WHISPER_MODEL_PATH, device="cuda", compute_type="float16")
segments, info = whispter_model.transcribe(audio_file_path, beam_size=5, word_timestamps=True)
```
error information
```
Could not load library libcudnn_cnn_infer.so.8. Error: /usr/local/cuda/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8: symbol _ZN5cudnn3ops18GetInternalStreamsEP12cudnnContextiPP11CUstream_st, version libcudnn_ops_infer.so.8 not defined in file libcudnn_ops_infer.so.8 with link time reference
Please make sure libcudnn_cnn_infer.so.8 is in your library path!
```
then I just load one tts model,it works
```python
from TTS.api import TTS
from faster_whisper import WhisperModel
tts = TTS(model_name="tts_models/multilingual/multi-dataset/your_tts", progress_bar=True, gpu=True)
#tts_clone = TTS(model_name="voice_conversion_models/multilingual/vctk/freevc24", progress_bar=True, gpu=True)
whispter_model = WhisperModel(WHISPER_MODEL_PATH, device="cuda", compute_type="float16")
segments, info = whispter_model.transcribe(audio_file_path, beam_size=5, word_timestamps=True)
```

## System Info
![image](https://github.com/pytorch/pytorch/assets/22809208/8404d6e8-55db-4745-b475-7c76a87a2b97)


cc @ptrblck
cpu reduce: output accumulate type when input is bfloat16 or float16
Labels: module: cpu, open source, Stale

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #104586
* #104585
* #104584
* #104583
* #104239
* #104238



cc @jgong5 @XiaobingSuper @sanchitintel @ashokei @jingxu10
Remove cpp_custom_type_hack
Labels: module: cpu, open source, topic: not user facing

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #104569
* #72303
* #104535

Closes #72263

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
Remove deprecated fbgemm operators
Labels: open source, ciflow/trunk, release notes: quantization, topic: bc breaking, merging

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #104569
* #72303
* __->__ #104535

These operators are not used and have been deprecated since #72690 (Feb 2022). Additionally, the `torch.jit.quantized` interface has been deprecated since #40102 (June 2020).
DISABLED test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
Labels: oncall: distributed, module: flaky-tests, skipped

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_sparse_all_reduce_sum_cuda&suite=TestDistBackendWithSpawn) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/14731815606).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 12 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_sparse_all_reduce_sum_cuda`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `distributed/test_distributed_spawn.py`

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Add inverse gamma distribution and fix `sign` bug in `PowerTransform`.
Labels: module: distributions, triaged, open source, ciflow/trunk

This PR comprises a few small contributions:

1. `PowerTransform` returned a sign of `+1` irrespective of exponent. However, it should return the sign of the exponent because the gradient has the same sign as the exponent. That issue has been fixed.
2. Added tests to catch errors akin to 1. in the future.
3. Added an `InverseGamma` distribution as a `TransformedDistribution` with `PowerTransform(-1)` and `Gamma` base distribution. The `InverseGamma` is often used as a prior for the length scale of Gaussian processes to aggressively suppress short length scales (see [here](https://betanalpha.github.io/assets/case_studies/gaussian_processes.html#323_Informative_Prior_Model) for a discussion).

Note: I added a `positive` constraint for the support of the inverse gamma distribution because the `PowerTransform(-1)` can fail for `nonnegative` constraints if the random variable is zero. 

```python
>>> torch.distributions.InverseGamma(0.5, 1.0).log_prob(torch.zeros(1))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-8-758aa22deacd> in <module>
----> 1 torch.distributions.InverseGamma(0.5, 1.0).log_prob(torch.zeros(1))

~/git/pytorch/torch/distributions/transformed_distribution.py in log_prob(self, value)
    140         """
    141         if self._validate_args:
--> 142             self._validate_sample(value)
    143         event_dim = len(self.event_shape)
    144         log_prob = 0.0

~/git/pytorch/torch/distributions/distribution.py in _validate_sample(self, value)
    298         valid = support.check(value)
    299         if not valid.all():
--> 300             raise ValueError(
    301                 "Expected value argument "
    302                 f"({type(value).__name__} of shape {tuple(value.shape)}) "

ValueError: Expected value argument (Tensor of shape (1,)) to be within the support (GreaterThan(lower_bound=0.0)) of the distribution InverseGamma(), but found invalid values:
tensor([0.])
```

This differs from the scipy implementation.

```python
>>> scipy.stats.invgamma(0.5).pdf(0)
0.0
```

cc @fritzo @neerajprad @alicanb @nikitaved @lezcano
[NCCL][CUDA][CUDA Graphs] Flush enqueued work before starting a graph capture
Labels: module: cuda, triaged, module: nccl, open source, Merged, Reverted, module: cuda graphs, ciflow/trunk, topic: not user facing, ciflow/periodic

#103503 addresses the situation where additional work is enqueued for the NCCL watchdog to poll during a graph capture---something we want to avoid as the subsequent polling will query an event and crash the capture.

However, there is currently no check that there was not work _already_ enqueued for the watchdog to poll. If there was already work that was enqueued and not cleaned up before the start of a graph capture, then we run into a similar problem where the event query will crash the capture. We've observed this causing crashes on several workloads, although it is somewhat flaky (if the watchdog happens to poll just before the graph capture and cleanup, then we dodge the crash).

This is a bit of a tricky issue as it involves making sure that no process group has enqueued work at the start of a capture, and as such the simplest solution is to add a bit of global state to track all process groups. This PR forces the start of the graph capture to wait until all enqueued work is completed and cleaned up or times out.

I did consider the alternative of simply having the watchdog skip cleanup if we detect that we are in the middle of a graph capture, but I think deferring the cleanup until later could result in false positive timeouts (e.g., if we defer cleanup on work that has completed long ago, checking timers after the graph capture could yield a "timeout").

CC @Aidyn-A 
@bottler @kwen2501 @ptrblck 

cc @ptrblck @mcarilli @ezyang
FSDP Optimizer Overlap - follow ups
Labels: oncall: distributed, triaged, module: fsdp

### 🚀 The feature, motivation and pitch

FSDP optimizer overlap is in https://github.com/pytorch/pytorch/pull/98667 needs some follow up work:

- We reallocate the _cpu_grad for CPU offload every iteration. This is potentially wasteful and we should run some profiling to understand the memory / allocation overhead tradeoff.
- We wait on optimizer step completion in post-backward, when it could be moved to the next pre-forward
- For CPU offload, we should explore running the step() on the GPU, right now it runs on CPU

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
`torch.view_as_real(tensor)` should return `nn.identity(tensor)` if its not complex instead of raising an error
Labels: triaged, module: complex

### 🚀 The feature, motivation and pitch

I think it would be easier if `torch.view_as_real(tensor)` just return the original tensor (maybe with a warning) instead of raising an error. Would simplify usage from something like
```
import torchaudio
import torch
spectrogram = torchaudio.transforms.Spectrogram()(audio)
spectrogram = torch.view_as_real(spectrogram) if torch.is_complex(spectrogram) else spectrogram 
```

to 

```
import torchaudio
import torch
spectrogram = torchaudio.transforms.Spectrogram()(audio)
spectrogram = torch.view_as_real(spectrogram) #handles any case
```

### Alternatives

_No response_

### Additional context

_No response_

cc @ezyang @anjali411 @dylanbespalko @mruberry @Lezcano @nikitaved
[Feature Request] Add a new overload of torch::jit::load to restore traced shape and type 
Labels: oncall: jit

### 🚀 The feature, motivation and pitch

I have a inference system, which consist of offline part and runtime part. In the offline phase, we will need to convert torchscript to our model format. So I'd like to be able to access the shape and type information of torch script inputs. Luckily, someone has done similar thing. PR: https://github.com/pytorch/pytorch/pull/89541

I only need to add a new api:

TORCH_API Module load(
    const std::string& filename, bool restore_shapes,
    c10::optional<c10::Device> device = c10::nullopt,
    bool load_debug_files = true);

### Alternatives

_No response_

### Additional context

Usage: 

auto module = torch::jit::load(model_path, true);

After that, you can get the input shapes and types.

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Deprecated the device usage without device_type
Labels: triaged, open source, module: deprecation, Stale, topic: deprecation, topic: not user facing

Fixes #ISSUE_NUMBER
1. Now for the operators or APIs, we can call it by `device=index` without `device_type`, for example, `torch.device(0)` will get `cuda:0`, and `torch.rand(2, device=0)` also will get `cuda:0`, but it is not friendly for other device type, so I add the deprecated warning info to suggest to use `device="cuda:0"`.
DISABLED test_nnc_correctness_frac_cpu_bfloat16 (__main__.TestNNCOpInfoCPU)
Labels: skipped, NNC

Platforms: asan

This test was disabled because it is failing on master ([recent examples](http://torch-ci.com/failure/test_jit_fuser_te.py%3A%3ATestNNCOpInfoCPU%3A%3Atest_nnc_correctness_frac_cpu_bfloat16)).

Disable this XFAIL while we try to figure out why this successes only on ASAN.  This is related to the change in https://github.com/pytorch/pytorch/pull/103647.  This test was set as XFAIL in https://github.com/pytorch/pytorch/issues/76047

cc @EikanWang @jgong5 @jeanschmidt @cyyever @davidberard98
[ONNX][TypePromo] Automate codegen type promotion rules
Labels: module: onnx, triaged

              Maybe we should have an exclusive file with the auto generated content? That would make future update easier, maybe even automated without having to parse stuff like "# DO NOT EDIT MANUALLY !!!"

_Originally posted by @thiagocrepaldi in https://github.com/pytorch/pytorch/pull/104063#discussion_r1247218389_
Functorch scan
Labels: triaged, open source, Stale, no-stale

This pull request contains the dense and autograd implementation of the functorch scan operator, as well as some unit tests.

The unit tests can be run with:
```
python test/functorch/test_control_flow.py -k TestControlFlow.test_scan
```
from the repo root directory.

Some details:

1. The scan operator interface adhere to JAX scan operator found in https://jax.readthedocs.io/en/latest/_autosummary/jax.lax.scan.html
2. lengths, reverse and unroll are not yet supported
3. You may find that there is a *reverse* keyword argument in the op implementation, but not exposed. The reverse argument is there so that we could reuse scan_dense for backward computation. In the future, we can expose the *reverse* kw option at the scan operator top level for a reversed forward computation
Silent incorrect result for addmm for noncontiguous input
Labels: high priority, triaged, module: mkldnn, module: correctness (silent), module: intel

### 🐛 Describe the bug

```python
import torch

def f_non_contiguous(x, y, z):
    z_t = torch.ops.aten.t.default(z)
    return torch.ops.aten.addmm.default(x, y, z_t)

def f_contiguous(x, y, z):
    z_t = torch.ops.aten.t_copy.default(z)
    return torch.ops.aten.addmm.default(x, y, z_t)

x = torch.randn(256)
y = torch.randn(6, 960)
z = torch.randn(256, 960)

contiguous_result = f_contiguous(x, y, z)
non_contiguous_result = f_non_contiguous(x, y, z)

print(contiguous_result.is_contiguous())  # prints True 
print(non_contiguous_result.is_contiguous()) # print True 

print(torch.allclose(contiguous_result, non_contiguous_result)) # prints False
print(torch.allclose(contiguous_result.contiguous(), non_contiguous_result.contiguous())) # prints False
```



### Versions

latest main

cc @ezyang @gchanan @zou3519 @gujinghui @PenghuiCheng @XiaobingSuper @jianyuh @jgong5 @mingfeima @sanchitintel @ashokei @jingxu10 @min-jean-cho @yanbing-j @Guobing-Chen @Xia-Weiwen @frank-wei
Make decomps opt-in for upsample_nearest 1D / 2D / 3D
Labels: Stale, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #104432

Fixes the decomps for `upsample_nearest_{1d, 2d, 3d}` so they are opt-in rather than always run. This keeps `upsample_nearest_*d.default` in the graph during normal tracing.
Support CUDA 12.2 
Labels: module: cuda, triaged

### 🚀 The feature, motivation and pitch

Interesting feature:
This release introduces Heterogeneous Memory Management (HMM), allowing seamless sharing of data between host memory and accelerator devices. HMM is supported on Linux only and requires a recent kernel (6.1.24+ or 6.2.11+).

### Alternatives

_No response_

### Additional context

_No response_

cc @ptrblck
distributed hooks want to support custom device
Labels: oncall: distributed, triaged

### 🐛 Describe the bug

for distributed hooks, there are some hard-codes, for cutsom device (privateuse1 backend, cuda-like device), we also want to support these hooks.

### Versions

main

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
FakeTensor can't handle meta impls that perform device conversion
Labels: triaged, module: fakeTensor

Minified after finding an internal example from @suo.

```
import torch
from torch.fx.experimental.proxy_tensor import make_fx

def foo_meta(x):
    return x.to(device='cpu')

lib = torch.library.Library("test", "FRAGMENT")
lib.define("foo(Tensor x) -> Tensor")
lib.impl("test::foo", foo_meta, "Meta")

def f(x):
    return torch.ops.test.foo.default(x)

x = torch.ones(4, device='cuda')
fx_g = make_fx(f, tracing_mode='symbolic')(x)
print(fx_g.code)
```

Fails with:
```
  File "/scratch/hirsheybar1/work/pytorch/tmp3.py", line 6, in foo_meta
    return x.to(device='cpu')
  File "/scratch/hirsheybar1/work/pytorch/torch/_decomp/decompositions.py", line 1646, in _to_copy
    x = torch._prims.device_put(x, device)
  File "/scratch/hirsheybar1/work/pytorch/torch/_ops.py", line 429, in __call__
    return self._op(*args, **kwargs or {})
  File "/scratch/hirsheybar1/work/pytorch/torch/_prims/__init__.py", line 2069, in _device_put_meta
    return TensorMeta(a, device=utils.canonicalize_device(device))
  File "/scratch/hirsheybar1/work/pytorch/torch/_prims/__init__.py", line 258, in TensorMeta
    return torch.empty_strided(shape, strides, dtype=dtype, device=device)
RuntimeError: /scratch/hirsheybar1/work/pytorch/build/aten/src/ATen/RegisterCPU.cpp:5200: SymIntArrayRef expected to contain only concrete integers
```

FakeTensor treads a fine line by trying to ensure that any operators that run underneath that dispatch to meta. It looks like that doesn't work properly when performing device conversion ops inside of your meta function, like `.to()`.
DISABLED test_conv3d_64bit_indexing_cuda (__main__.TestConvolutionNNDeviceTypeCUDA)
Labels: module: rocm, triaged, skipped

Platforms: rocm

This test was disabled because it is failing on ROCm5.5 CI upgrade. This test is also being skipped in ROCm5.4.2 upstream CI.

Copied from: https://github.com/pytorch/pytorch/issues/104348

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
[testing only] Enable inlining modules by default
Labels: Stale, release notes: quantization, module: inductor, module: dynamo, ciflow/inductor, keep-going

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #104360
* #103676
* #103987



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler @ngimel @anijain2305
Segmentation error while using F.cross_entropy with mps(for code that works fine with device= "cpu")
Labels: triaged, module: mps

### 🐛 Describe the bug

import torch
import torch.nn as nn
from torch.nn import functional as F
batch_size = 32 
block_size = 8 
device = "mps"

vocab_size = 1000
X = torch.randint(0, vocab_size, (32, 8), device = device)
Y = torch.randint(0, vocab_size, (32, 8), device = device)
token_embedding_table = nn.Embedding(vocab_size, vocab_size, device=device)
logits = token_embedding_table(X) 
B, T, C = logits.shape
logits = logits.view(BT, C)
targets = Y.view(BT)
loss = F.cross_entropy(logits, targets)


### Versions

vrindavan@Vrindavans-MBP Code % python3 collect_env.py
Collecting environment information...
PyTorch version: 2.0.0.dev20230208
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.0 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: version 3.25.2
Libc version: N/A

Python version: 3.9.6 (default, May  7 2023, 23:32:44)  [Clang 14.0.3 (clang-1403.0.22.14.1)] (64-bit runtime)
Python platform: macOS-14.0-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M2

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==2.0.0.dev20230208
[pip3] torchaudio==2.0.0.dev20230207
[pip3] torchvision==0.15.0.dev20230207
[conda] Could not collect
vrindavan@Vrindavans-MBP Code % 

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
DISABLED test_backward_ddp_inside (__main__.TensorPipeDdpUnderDistAutogradTest)
Labels: oncall: distributed, module: flaky-tests, skipped

Platforms: linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_backward_ddp_inside&suite=TensorPipeDdpUnderDistAutogradTest) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/14609542854).

Over the past 3 hours, it has been determined flaky in 2 workflow(s) with 2 failures and 2 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_backward_ddp_inside`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `distributed/rpc/test_tensorpipe_agent.py`

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Torch randperm with device mps does not sample exactly uniformly from all possible permutations
Labels: triaged, module: random, module: mps

### 🐛 Describe the bug

I don't think this bug is very urgent but I just wanted to capture a note stemming from discussion in #104171 so it wasn't forgotten.

Current the mps fused(?) kernel for randperm does works by generating random values without replacement from `min_int64` to `max_int64` and then for each index and sorting the indexes by these values in ascending order. Unfortunately due to collisions the resulting distribution over permutations of the indexes will not be perfectly uniform (see [this comment](https://github.com/pytorch/pytorch/pull/104171#issuecomment-1608754004) for a more in depth explanation). 

The cuda kernel solves this by taking doing an extra step after the sort to shuffle all "islands" of consecutive indexes with the same value. See the [algorithm description](https://github.com/pytorch/pytorch/blob/05ebd538d4b064b19ba960c2b198bf991a49ca89/aten/src/ATen/native/cuda/Randperm.cu#L230) for a more detailed view.

This extra step would likely have to be implemented as a custom metal kernel but could probably just follow the structure of the [cuda implementation](https://github.com/pytorch/pytorch/blob/05ebd538d4b064b19ba960c2b198bf991a49ca89/aten/src/ATen/native/cuda/Randperm.cuh#L13).

### Versions

```
Collecting environment information...
PyTorch version: 2.1.0a0+git2967116
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.3.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.202)
CMake version: version 3.26.4
Libc version: N/A

Python version: 3.11.3 (main, May 15 2023, 18:01:31) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-13.3.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==1.25.0
[pip3] torch==2.1.0a0+git2967116
[conda] numpy                     1.25.0                   pypi_0    pypi
[conda] torch                     2.1.0a0+git2967116           dev_0    <develop>
```

cc @pbelevich @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
affine_grid and grid_sample operators merge/accelleration
Labels: module: performance, feature, triaged

### 🚀 The feature, motivation and pitch

Hi, 

To warp some data according to a (batch) of affine transformations, two functions called sequentially need to be used:

1.  [affine_grid](https://pytorch.org/docs/stable/generated/torch.nn.functional.affine_grid.html) to calculate the transformed coordinates followed by
2.  [warp_sample](https://pytorch.org/docs/stable/generated/torch.nn.functional.grid_sample.html#torch.nn.functional.grid_sample) to do the actual warping.

Would it make sense to have also the option of a function that does the two operation in a single pass rather than having to call the two sequentially?
The advantage would be only for speed: warping according to an affinity transformation would avoid to store the result of affine_grid (shape (N,H,W,2) for a spatial warp), but the transformed coordinates could be calculated locally and then used immediately to warp the input data.

I am exploring the possibility of working on the topic and wanted to know if a contribution in this direction could be useful. 
Thanks in advance for your feedback



### Alternatives

_No response_

### Additional context

_No response_
getattr on `__slots__` object potentially suspicious
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

@jansel pointed out that https://github.com/pytorch/pytorch/blob/afc788a99c1853e815283854ef6f168b250eaf2c/torch/_dynamo/variables/user_defined.py#L367 looked suspicious. The check should be something like ("if the object has slots AND no custom getattr, then we know how to handle it. Otherwise, the getattr may be unsafe").



### Error logs

_No response_

### Minified repro

_No response_

### Versions

main

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
`F.conv1d` and `F.conv2d` propagate `nan`'s incorrectly when minibatch > 15
Labels: needs reproduction, module: nn, triaged, module: macos, module: NaNs and Infs, module: arm

### 🐛 Describe the bug

Both `F.conv1d` and `F.conv2d` propagate `nan`'s incorrectly when minibatch > 15.
The test function below checks if the number of `nan`'s is the same for a minibatch of 15 and a minibatch of 16.

```python
import torch

def test_conv(in_channels, out_channels, kernel_size, input_size):
    weight = torch.randn(out_channels, in_channels, *kernel_size)
    input = torch.randn(16, in_channels, *input_size)
    input[...,-1:] = float("nan")

    assert len(kernel_size) == len(input_size)
    if len(kernel_size) == 1:
        conv_fun = torch.nn.functional.conv1d
    elif len(kernel_size) == 2:
        conv_fun = torch.nn.functional.conv2d
    elif len(kernel_size) == 3:
        conv_fun = torch.nn.functional.conv3d

    output1 = conv_fun(input[:15], weight)
    output2 = conv_fun(input, weight)
    assert output1[:15].isnan().sum() == output2[:15].isnan().sum()
```

Some examples for `F.conv1d` and `F.conv2d` that fail are:
```python
test_conv(8, 6, (4,), (10,))
test_conv(4, 5, (4, 8), (6, 16))
```

However, I have not been able to replicate this with `F.conv3d`.

### Versions

PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.4.1 (arm64)
GCC version: Could not collect
Clang version: 16.0.6
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.6 (main, Aug 29 2022, 10:06:59) [Clang 13.1.6 (clang-1316.0.21.2.5)] (64-bit runtime)
Python platform: macOS-13.4.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Max

Versions of relevant libraries:
[pip3] mypy==1.3.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.24.2
[pip3] torch==2.0.1
[pip3] torchaudio==2.0.0
[pip3] torchtext==0.13.1
[pip3] torchvision==0.15.0
[conda] Could not collect

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @malfet
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
Labels: needs reproduction, oncall: binaries, module: cuda, triaged

### 🐛 Describe the bug

import torch，then:
import torch，then:

### Versions

import torch，then:
![image](https://github.com/pytorch/pytorch/assets/39661319/561c5a45-8396-4377-ae8e-2b483825bc34)


cc @seemethere @malfet @ptrblck
[FSDP] `ignored_states` is broken with auto wrap
Labels: triaged, module: fsdp

Passing ignored parameters to `ignored_states` with auto wrap does not propagate them to the nested FSDP instances.

My current thinking is that, for a quick fix, we can add `"ignored_states": self._ignored_params`  to these `fsdp_kwargs`:
https://github.com/pytorch/pytorch/blob/18dacf7e793ec3ca3fb5c3d1e85485068e004d6e/torch/distributed/fsdp/fully_sharded_data_parallel.py#L417-L429
Then, we need to add some unit tests.

However, I am thinking about what the right design should be with `ignored_modules` / `ignored_states` more broadly. Passing ignored parameters to `ignored_states` _does not_ match the semantics of passing ignored modules to `ignored_modules` even excluding partially ignored modules. Via `ignored_states`, FSDP will still construct a `FullyShardedDataParallel` instance that just manages 0 parameters, which is undesirable.




cc @zhaojuanmao @mrshenli @rohan-varma @fegin
[RFC] Make `_HYBRID_SHARD_ZERO2` public as `HYBRID_SHARD_GRAD_OP`
Labels: triaged, module: fsdp

This is for consistency with our existing `SHARD_GRAD_OP` sharding strategy and to avoid mentioning ZeRO-2 in our public API (given that (1) we should not expect all FSDP users to know about ZeRO and (2) there are technical differences between our `SHARD_GRAD_OP` and ZeRO-2).
https://github.com/pytorch/pytorch/blob/18dacf7e793ec3ca3fb5c3d1e85485068e004d6e/torch/distributed/fsdp/api.py#L69C5-L69C24

cc @zhaojuanmao @mrshenli @rohan-varma @fegin
[inductor] Updated upsample_bicubic2d decomposition
Labels: open source, module: inductor, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #104248
* #104182
* #104181

- fixed dispatch for upsample_bicubic2d_vec and use inductor lowering
- added support for uint8
- updated tests

Performance benchmarks results

```
[-------------------------- Interpolate bicubic, AA=false, cpu -------------------------]
                                                                   |  Eager   |  Compiled
1 threads: ------------------------------------------------------------------------------
- main/nightly:
      Input (3, 345, 456), torch.uint8, torch.contiguous_format    |   647.2  |   2828.6
      Input (3, 345, 456), torch.float32, torch.contiguous_format  |  2163.4  |   2076.7
      Input (3, 345, 456), torch.uint8, torch.channels_last        |   263.5  |   3284.6
      Input (3, 345, 456), torch.float32, torch.channels_last      |  3585.1  |   2400.3
- PR:
      Input (3, 345, 456), torch.uint8, torch.contiguous_format    |   634.5  |   3579.6   <--- worse than in main
      Input (3, 345, 456), torch.float32, torch.contiguous_format  |  2143.6  |   2237.8   <--- worse than in main
      Input (3, 345, 456), torch.uint8, torch.channels_last        |   261.2  |   2883.2 
      Input (3, 345, 456), torch.float32, torch.channels_last      |  3544.3  |   1814.4


Times are in microseconds (us).

[------------------------ Interpolate bicubic, AA=false, cuda -------------------------]
                                                                   |  Eager  |  Compiled
1 threads: -----------------------------------------------------------------------------
- main/nightly:
      Input (3, 345, 456), torch.float32, torch.contiguous_format  |   17.2  |   352.0
      Input (3, 345, 456), torch.float32, torch.channels_last      |   17.5  |   357.4
- PR:
      Input (3, 345, 456), torch.float32, torch.contiguous_format  |   14.3  |    43.2
      Input (3, 345, 456), torch.float32, torch.channels_last      |   14.4  |    40.1

Times are in microseconds (us).
```



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[proposal] "Name" string attribute for modules, parameters, buffers, tensors for more pleasant debugging (especially for graph printouts / export / studying compiled generated code)
Labels: feature, triaged, needs design

### 🚀 The feature, motivation and pitch

This is useful for debugging complex tree-structured models to be able to `.name` and understand where a module is found within the whole model tree

This idea is currently used only for parameters within `state_dict()` serialization. I suggest that enabling `.name` attribute/property would be useful for debugging and various formatting/debug-printing in the general context. Such `.name` could be used in `__str__` implementations of modules and tensors, could be used for creating easier-visualizable ONNX graphs. Access to module name might also be useful for various finer-grained logic hacks in module hooks (although maybe not the best coding practice in all cases - but for hacks might be okay).

I propose to:
1. introduce a `.name` property (backed by `._name` attribute which may not exist if we wrongly torch.load and old model file) or just a `.name` attribute if the deserializing-old-module-objects-without-some-attributes is not a problem. It should be an empty string by default.
2. introduce an instance `.set_names_recursively()` (modulo bikeshedding) method on `torch.nn.Module` which would go around and produce names similar to what's now found in `state_dict()` keys formatting

I also propose to support setting/getting such attribute for any tensor. It's understood that most tensors won't have it assigned and the user would need to set it manually for any usefulness, but it's still good (e.g. can be set for tensors-to-be-returned-from-a-function). Propagation of such tensor names is a more complex task, and I propose that it's out of scope as the feature is already useful if only manual tensor names are supported (for the cases useful for debugging). Alternatively a `tensor.name()` / `tensor.name(value)` could be maybe used instead of the attribute so that the setter would `return self` for fluency and convenience, so that one can write `return (x + 1).name("mytensorplus1")` - although it would mean that the `name(...)` method would return either a string or a Tensor/sth else, depending on its argument.

Another consideration is that currently ONNX exporter already produces some automatic module/op names, and some code might have taken some dependencies on this. So the ONNX naming behavior should be unchanged unless explicitly `.set_names_recursively()` was called (so empty names would be overridden by existing automatic names)

Also probably if `.name`s are set, state_dict() should use them. Another consideration is that some people are probably monkey-patching `.name` attributes on tensors/modules themselves, so if state_dict starts using them, it might be surprising, so maybe some attribute name bikeshedding is required

e.g. https://github.com/facebookresearch/fvcore/blob/main/fvcore/nn/jit_analysis.py is doing sth similar wrt to module names

cc @ezyang @albanD
DISABLED test_mem_get_info (__main__.TestCudaMultiGPU)
Labels: module: cuda, triaged, module: flaky-tests, skipped

Platforms: linux, win

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_mem_get_info&suite=TestCudaMultiGPU) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/14578954385).

Over the past 3 hours, it has been determined flaky in 4 workflow(s) with 4 failures and 4 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_mem_get_info`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_cuda_multigpu.py`

cc @ptrblck
[ONNX] Investigate `nn.functional.nll_loss` skip/xfail reason
Labels: module: onnx, triaged

              After type promotion this test starts to fail a few subtests with unexpected pass, which appears to be incosistent with the taggex xfail reason.

_Originally posted by @BowenBao in https://github.com/pytorch/pytorch/pull/104064#discussion_r1243093115_