gopikrsmscs/torch-issues huggingface dataset:
dist.barrier() should be able to go through custom backend
Labels: oncall: distributed

### 🚀 The feature, motivation and pitch

With https://github.com/pytorch/pytorch/pull/95072, users can now create a gloo backend for cpu tensors and a custom backend for cuda tensors. However, in such a use case, `dist.barrier()` will always go to gloo due to the following code block in https://github.com/pytorch/pytorch/blob/master/torch/csrc/distributed/c10d/ProcessGroup.cpp: 
```
virtual c10::intrusive_ptr<Work> barrier(
      const BarrierOptions& opts = BarrierOptions()) {
    static at::Tensor tensor;
    // TODO: if nccl was specified then use it
    if (backendType_ == c10d::ProcessGroup::BackendType::NCCL) {
      // set cuda tensor
      tensor = at::empty(
          {1},
          at::TensorOptions().device(at::DeviceType::CUDA).dtype(at::kByte));
    } else {
      // Default to using cpu implementation
      tensor = at::empty(
          {1},
          at::TensorOptions().device(at::DeviceType::CPU).dtype(at::kByte));
    }
```
Custom backends such as SMDDP (https://docs.aws.amazon.com/sagemaker/latest/dg/data-parallel-modify-sdp-pt.html) require `dist.barrier()` to go through SMDDP barrier() because of certain cuda stream semantics -- when a barrier is called, all previous collective communication operations must finish, and frameworks such as DeepSpeed rely on this semantic.

We should allow a way for `dist.barrier()` to go through custom backends. Possibilities are:
1. Add a `device` option in `dist.barrier()`. E.g. `dist.barrier(device='cuda')`
2. When we register custom backends, add a parameter there to signal that the custom backend supports cuda tensor, so the check `backendType_ == c10d::ProcessGroup::BackendType::NCCL` can become `is_cuda_backend(backendType_)`
3. There might be more solutions

Urgency wise, we need a solution before the 2.0 release.

P.S. We don't have the same problem when mapping a custom backend, say SMDDP, to both cpu and cuda devices.

Tagging @H-Huang for visibility and technical inputs. 
Also tagging custom backend developers @indhub @0x6b64 @Arjunbala 

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
[BE] [cuDNN] Always build assuming cuDNN >= 8.0
Labels: module: cpu, triaged, open source, Stale, ciflow/trunk, release notes: quantization, topic: not user facing, ciflow/periodic, ciflow/inductor

<!--
copilot:summary
-->
### <samp>🤖 Generated by Copilot at 27084ed</samp>

This pull request simplifies and cleans up the code that uses the cuDNN library for convolution, batch normalization, CTC loss, and quantized operations. It removes the unnecessary checks and conditions for older cuDNN versions and the experimental cuDNN v8 API, and replaces them with the stable `cudnn_frontend` API that requires cuDNN v8 or higher. It also adds the dependency and configuration for the `cudnn_frontend` library in the cmake and bazel files.

This is a re-land of https://github.com/pytorch/pytorch/pull/91527

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
Multiheadattention module doesn't implement the function about kdim and vdim
Labels: triaged, module: multi-headed-attention

### 🐛 Describe the bug

![a](https://user-images.githubusercontent.com/104755879/221919054-ff045a9b-7937-407a-831c-2767f39534bb.png)
The function just implements kdim=vdim=embed_dim.

### Versions

The code on GitHub
`copy.deepcopy` does not copy gradients of nn.Parameter
Labels: module: bc-breaking, module: autograd, module: nn, triaged, topic: bc breaking

Deep copy now copies the .grad field for tensors, but it does not work for Parameter

```
import torch
import copy

c = torch.nn.Linear(1, 10)

c.weight.grad = torch.rand((10, 1))

print(c.weight.grad)
d = copy.deepcopy(c)
print(d.weight.grad)
```
Tested with version 1.13.1. Running the code from @albanD gives the expected result. It would be great if we could find a way to copy an entire (larger) model including its gradients without having to copy every one by hand.

_Originally posted by @jhuebotter in https://github.com/pytorch/pytorch/issues/3307#issuecomment-1425753484_
            

cc @ezyang @gchanan @albanD @zou3519 @gqchen @pearu @nikitaved @Lezcano @Varal7 @mruberry @jbschlosser @walterddr @mikaylagawarecki @saketh-are
torch.compile compilation time on huggingface regressed ~11%
Labels: triaged, oncall: pt2

From looking at the perf [dashboard](https://github.com/pytorch/pytorch/issues/93794), https://github.com/pytorch/pytorch/pull/95134 regressed inductor compilation time by about ~11% across the huggingface benchmark suite, from a mean compilation time of `65.56s` to `72.57s`.


I tested locally with `MBartForConditionalGenerationq`.

Before: 82.49s
After: 91.90s

We can close this issue if we think this isn't worth digging into further (another option is to look into inductor compile time more holistically instead of looking at regressions in this specific PR).

cc @ezyang @soumith @msaroufim @wconstab @ngimel
torch needs to SHOW that it support sm_89 even if functionally the same as sm_86
Labels: module: cuda, triaged

### 🐛 Describe the bug

torch needs to SHOW that it supports sm_89 even if functionally the same as sm_86.
While it may be true that optimizing specifically for an sm_89 provides no benefits beyond what optimizing for an sm_86 does, by NOT listing sm_89 in either what torch._C._cuda_getArchFlags() nor torch.cuda.get_arch_list() outputs, code which checks if the actual GPU(in my case (8,9)) is in the list of supported GPU's can lead to "generic" optimization which are far from optimal.

If the left hand of pytorch say you have a sm_89 and the right hand say it supports "sm_37 sm_50 sm_60 sm_70 sm_75 sm_80 sm_86 sm_90" with sm_89 being missing from the list this is a problem and I have seen code where this kind of lookup was done.  If you want to save code space and, as part of the build, not generate sm_89 code which is identical to the sm_86 then use "if" tests to redirect sm_89 to the sm_86 code paths to keep end user code from needing to be aware of this quirk. But you need to list sm_89 as a supported architecture.

NOTE: If someone does a single TORCH_CUDA_ARCH_LIST build of sm_86 OR sm_89 both should return "sm_86 sm_89" and ['sm_86', 'sm_89'] from the two functions shown above.  However, I won't push on that.  Anyone doing their own personal build of pytorch probably doesn't need this.  It is the end user or basic app coders that are using the official FULL build of pytorch that needs ALL supported GPU's to be listed because "support for a GPU" is not the same as "does a GPU have any unique performance features another GPU earlier in the list has".

```
torch._C._cuda_getArchFlags() = sm_37 sm_50 sm_60 sm_70 sm_75 sm_80 sm_86 sm_90
torch.cuda.get_arch_list() = ['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90']
```

### Versions

Not needed

cc @ngimel
Create a new Docker image with all inductor benchmarks and pre-trained models downloaded
Labels: module: ci, triaged, module: devx

All these models are currently downloaded as part of the CI and the process is subjected to network flakiness.  Adding them all into a Docker image reduces this flakiness, thus improves reliability.

* All models under `benchmarks/dynamo/*.txt` would need to be added
* The docker image pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7 is currently used for the job.  If the size of all models are significant, we would need to create a new docker image on top of that instead of adding them into the same image to keep pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7 small for other use cases besides benchmarking

Also do we need the feature to redownload these models when they are updated upstream?

### Versions

Inductor workflow

cc @seemethere @malfet @pytorch/pytorch-dev-infra @ZainRizvi @kit1980 @clee2000 @desertfire
Pytorch Home Page does not specify which version of python it requires
Labels: module: docs, triaged

### 🐛 Describe the bug

When using the pip command to install pytorch suggested by the page https://pytorch.org/  , which is 
pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116
pip returns 'no matching distribution found". I guess this is because I have installed python 3.11. If I go back to python 3.10, pytorch installs correctly. I suggest that the pytorch home page clearly state that pytorch is not supported under windows/pip for python 3.11. If this is not possible, how can I know when pytorch will be supported for windows under python 3.11?
I am running Windows version 10.

![image](https://user-images.githubusercontent.com/7177684/221648620-41e61a9a-563a-45a6-992f-ff685ef202d7.png)


### Versions

C:\Users\fredm>wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
'wget' is not recognized as an internal or external command,
operable program or batch file.

So, that command does not work under Windows. How about one that does?

cc @svekars @carljparker
[onnx] sort / argsort with `stable` argument specified cannot be exported to onnx
Labels: module: onnx, triaged

### 🐛 Describe the bug

When using `torch.sort(x)`, onnx export is ok. 
But with `stable` flag, onnx export fails, saying "OnnxExporterError: Unsupported: ONNX export of operator Sort, Out parameter is not supported.". Here `out` argument is actually not provided.
Similar result occurs with `torch.argsort()`.

```python
import torch
from torch import nn
import numpy as np

class Demo(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        v, inds = torch.sort(x, stable=False)
        # v, inds = torch.sort(x, stable=True)
        # inds = torch.argsort(x, stable=False)
        # inds = torch.argsort(x, stable=True)
        return inds

if __name__ == "__main__":
    input_tensor = torch.range(20, 80)
    demo = Demo()
    out = demo(input_tensor)
    torch.onnx.export(demo, input_tensor, "/tmp/debug.onnx", verbose=True,
                        input_names=['data'],
                        opset_version=11,
                        do_constant_folding=True,
                        dynamic_axes={'data':{0:'batch'}})
```

### Versions

[pip3] torch==1.13.1
[Reproducibility]replication_pad2d_backward_cuda does not have a deterministic implementation
Labels: triaged, module: determinism, module: padding, oncall: pt2

### 🚀 The feature, motivation and pitch

Hi, I am following the instruction from [here](https://pytorch.org/docs/stable/notes/randomness.html#cuda-convolution-determinism) to set torch.use_deterministic_algorithms(True)

However, i encounter an error as shown in title.

### Alternatives

I have tried:
torch.use_deterministic_algorithms(True, warn_only=True)
and it works for me.

### Additional context

I not sure why its happening. But my implementation does involves padding. The main architecture contains convolution (Unet-encoder), a transformer encoder, and a Unet-decoder with upsampling.

cc @mruberry @kurtamohler @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
Support datatype argument for torch.distributed.all_gather() (And the whole distributed module)
Labels: feature, triaged, module: nccl, module: c10d

### 🚀 The feature, motivation and pitch

When I try to train with customized data loader, getting ILSVRC2012's sample index with torch.distributed.all_gather() from multiple GPU, I find it uses 16-bit as default when using backend nccl, and torch API doesn't support specifying a datatype. Therefore indexes larger than 65535 will only get lower half digits depending on the default casting.

My workaround is to separate the int32 into high and low 16 bits and then recover from it. 

I checked nccl document, their all_gather supports different datatypes, including their int32 type. I suggest adding a feature of choosing datatype (especially how many bits to use) for all_gather and other distributed operations, so that customized code can easily implement some desired function without increasing unnecessary overhead.

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Tensor.all() fails on MPS for tensors with more than 4 dimensions
Labels: triaged, module: reductions, module: mps

### 🐛 Describe the bug

`Tensor.all()` fails on MPS for tensors with more than 4 dimensions

```python
import torch

x = torch.zeros(1, 1, 1, 1, 1, device="mps")
x.all()       # crash
x.all(dim=0)  # crash
x.all(dim=1)  # no crash
x.all(dim=2)  # no crash
x.all(dim=3)  # no crash
x.all(dim=4)  # no crash
```

Output

```bash
Assertion failed: (0 <= mpsAxis && mpsAxis < 4 && "Runtime canonicalization must simplify reduction axes to minor 4 dimensions."), function encodeNDArrayOp, file GPUReductionOps.mm, line 76.
[1]    77353 abort      python debug.py
```

I believe the MPS implementation is here: https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/mps/operations/ReduceOps.mm

Surprisingly, most ops don't have this problem, including `torch.any()`. I'm guessing this is an undocumented limitation by MPS op `reductionAndWithTensor`.

### Versions

Collecting environment information...
PyTorch version: 2.0.0.dev20230224
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.0 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.102)
CMake version: version 3.24.0
Libc version: N/A

Python version: 3.10.9 | packaged by conda-forge | (main, Feb  2 2023, 20:26:08) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-13.0-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1

Versions of relevant libraries:
[pip3] mypy==1.0.1
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.24.2
[pip3] torch==2.0.0.dev20230224
[conda] numpy                     1.24.2                   pypi_0    pypi
[conda] pytorch                   2.0.0.dev20230224        py3.10_0    pytorch-nightly

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev
dynamo+aot improperly handles dupe args via *args
Labels: triaged, bug, oncall: pt2, module: dynamo

### 🐛 Describe the bug

A simple repro is a new test case for `pytorch/test/dynamo/test_aot_autograd.py` based on the existing test `test_arg_dupe_via_dynamo_recompiles_many_args_param`.

Basically, if a user module accepts *args format to forward, we don't deal with duplicate args correctly.

This shows up in real life once we make dynamo call `__call__` instead of .forward(), since __call__ (or rather, NNModule._call_impl) accepts *args rather than the typical usercode .forward which accepts an explicit list of args.

Dynamo+AOT may need to handle 'LocalInputSource' a little differently, to account for this.

```

    @patch("torch._functorch.config.debug_assert", True)
    def test_arg_dupe_via_dynamo_recompiles_star_args(self):
        class F(torch.nn.Module):
            def __init__(self):
                super().__init__()
                self.mean = torch.nn.Parameter(torch.randn(3, 3))

            def forward(self, *args):
                a, b, c, d = args
                a.t_()
                b.t_()
                c.t_()
                d.t_()
                return a + b + c + d + self.mean

        a = torch.randn(3, 3, requires_grad=True)
        b = torch.randn(3, 3, requires_grad=True)
        a1, a2, a3, a4 = a.clone(), a.clone(), a.clone(), a.clone()
        b1, b2, b3, b4 = b.clone(), b.clone(), b.clone(), b.clone()

        failure_reason = None

        def guard_fail_fn(failure):
            nonlocal failure_reason
            failure_reason = failure[0]

        self.assertTrue(failure_reason is None)

        cc = torch._dynamo.testing.CompileCounterWithBackend("aot_eager")

        f = torch._dynamo.optimize(cc, guard_fail_fn=guard_fail_fn)(F())
        f(a1, a1, a1, a1)

        # This raises the AssertionError pasted below
        f(a2, b2, b2, b2)

```

AssertionError that is raised:
```
 # AssertionError: At compilation time, graph 0 was compiled under the assumption that input 1 would be a duplicate of input 0, but at runtime this was not the case.  This indicates a guard bug in AOTAutograd or Dynamo, please file a bug to PyTorch.
```

Expected behavior: dynamo guard failure triggers recompile

### Versions

pytorch @ bc438af6fed4fe1fd0ed80e6d5f5ea17c3ca30bb

cc @ezyang @soumith @msaroufim @ngimel @bdhirsh @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire @Chillee
Import parameters from jit
Labels: oncall: jit

### 🚀 The feature, motivation and pitch

Hi, it would be great to have the possibility to import parameters from a jit module to the same network implemented as Torch::Module. This will allow performing fine-tuning on the platform.

### Alternatives

_No response_

### Additional context

_No response_

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Enrich shape operations with nested tensors
Labels: triaged, module: nestedtensor

### 🚀 The feature, motivation and pitch

[TorchRL](https://github.com/pytorch/rl/) and [tensordict](https://github.com/pytorch-labs/tensordict) could use nested tensors extensively, but we're facing some blockers:
- It often happens that we stack tensordicts, and then stack further stacks of tensordicts.
  Say that instead of a pure stack we create a nested tensor:
  ```python
  nt0 = torch.nested.nested_tensor((tensor0, tensor1))
  nt1 = torch.nested.nested_tensor((tensor2, tensor3))
  ```
  there is currently no way to create a nested tensor out of `nt0` and `nt1`.
  In RL this is a common operation: the first nesting would represent a time dimension, the second a batch dimension.
- It would also be nice to be able to control along which dimension the nesting happens, ie not always have the leading dimension. The usage would be similar to what I mentioned earlier, that given two batches of data we would like to nest them along the first dimension (eg the time dimension).
- Representing shapes should be possible, with some placeholder value for the mismatching dim

Thanks for the feature!

### Alternatives

_No response_

### Additional context

_No response_

cc @cpuhrsch @jbschlosser @bhosmer @drisspg @mikaylagawarecki
[BE] Make ActivationWrapper an abstract class
Labels: oncall: distributed, better-engineering

### 🚀 The feature, motivation and pitch

As mentioned in the comments for `ActivationWrapper` in torch.distributed's activation checkpoint code, it is not meant to be instantiated directly, so we should just prevent this via using an abstract class.

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
hf_GPT2_large CPU inference shows random failure on CI
Labels: triaged, oncall: pt2

https://hud.pytorch.org/hud/pytorch/pytorch/master/1?per_page=50&name_filter=inductor_torchbench_cpu

https://github.com/pytorch/pytorch/pull/95473 takes hf_GPT2_large off from the CI test, but it still needs investigation.


cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
`add/add_` for CSC: errors when trying to access non-existent `crow_indices`.
Labels: module: sparse, triaged

### 🐛 Describe the bug

A simple repro:

```python
In [1]: import torch         
                                                                                                                       
In [2]: x = torch.rand(3, 3).to_sparse_csc()                                                                           
<ipython-input-2-063a8a560ed1>:1: UserWarning: Sparse CSC tensor support is in beta state. If you miss a functionality in the sparse tensor support, please submit a feature request to https://github.com/pytorch/pytorch/issues. (Triggered 
internally at /home/nik/git/Quansight/pytorch/aten/src/ATen/SparseCsrTensorImpl.cpp:54.)
  x = torch.rand(3, 3).to_sparse_csc()
                                                                                                                       
In [3]: x + x
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [3], in <cell line: 1>()                                                                                      
----> 1 x + x                    
                                                           
RuntimeError: col_indices expected sparse row compressed tensor layout but got SparseCsc

```

All in all, add is broken in many ways because of missing checks/dispatch manipulations.

### Versions

Current master.

cc @alexsamardzic @pearu @cpuhrsch @amjames @bhosmer
Extend docs - Fixing out of memory with python garbage collection
Labels: module: docs, module: memory usage, triaged

### 📚 The doc issue

I suggest to extend https://pytorch.org/docs/stable/notes/faq.html to include a section about the python garbage collector, which is missing.

To be specific, we've now encountered several scenarios where we hit the following case:

Due to other processing during the training loop (in one case the data loader was loading complex data structures, in the other case the main loop created complex tensorboard logs), the garbage collector is triggered for gen0 and gen1, moving a few of those large tensors to gen2 (only about 10-20 objects, but a few hundred megabytes on GPU). Thus, these large tensors will not be released immediately. After adding a `gc.collect()` every nth iteration (or specifically, after each of these costly iterations), the leak was gone. An alternative/addition could be to call `gc.freeze()` before the loop, which will clear the gen2, thus reducing the number of required objects to perform gc gen2 and also reduce the gen2 collection overhead.

Detailed explanation: Because [gen2 gc will only trigger](https://devguide.python.org/internals/garbage-collector/#gc-oldest-generation) if `long_lived_pending / long_lived_total > 0.25` and there are already a few hundred thousand objects in gen2 after startup, it will not trigger before a lot of these large tensors end up in gen2. As an example with numbers: Assuming there are 350.000 objects in `long_lived_total` (which is approx the number I see when in the inference loop), you'd need 87.500 new objects in gen2 before gen2 gc would be triggered, which would need ~5k iterations. This will yield an OOM before ever being triggered.

### Suggest a potential alternative/fix

The suggested addition:

**Python garbage collection**: If you create many python objects during your loop(s), consider calling the Python garbage collector periodically with `gc.collect()` and/or freeze objects to keep generation 2 small by calling `gc.freeze()` after initialization before entering your loop(s). Read more about python garbage collection [here](https://devguide.python.org/internals/garbage-collector).


Thanks for considering this addition!

cc @svekars @carljparker
PT2 Computes Multi Device Backward in a Single Thread
Labels: feature, low priority, triaged, oncall: pt2, module: inductor

### 🐛 Describe the bug

In PyTorch Eager when the backwards is invoked there is a separate thread per device that invokes nodes in the backward graph. However in PT2, this parallelism is lost as we will compile a backward with multiple devices into one single compiled function and invoked in one thread.



### Versions

master

cc @ezyang @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @soumith
DISABLED test_variant_consistency_jit_linalg_lstsq_cpu_complex64 (__main__.TestJitCPU)
Labels: triaged, module: flaky-tests, skipped, module: unknown

Platforms: win, windows, linux

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_variant_consistency_jit_linalg_lstsq_cpu_complex64&suite=TestJitCPU) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/11557283803).

Over the past 3 hours, it has been determined flaky in 3 workflow(s) with 3 failures and 3 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_variant_consistency_jit_linalg_lstsq_cpu_complex64`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_ops_jit.py`
Parallel Associative Scan
Labels: high priority, triaged, oncall: pt2, module: functorch

### 🚀 The feature, motivation and pitch

It would be great to have a general parallel prefix sum (associative scan) operation in PyTorch, something like [associative_scan](https://jax.readthedocs.io/en/latest/_autosummary/jax.lax.associative_scan.html) in JAX or [scan_associative](https://www.tensorflow.org/probability/api_docs/python/tfp/math/scan_associative) in TensorFlow Probability. This operation is key for the parallelization of some algorithms in CRFs, [filtering/smoothing in state space models](https://github.com/EEA-sensors/sequential-parallelization-examples/blob/main/python/temporal-parallelization-bayes-smoothers/parallel_kalman_jax.ipynb), etc.

### Alternatives

I found [this implementation](https://github.com/lxxue/prefix_sum) but it's only for computing the prefix sum and not for general associative binary operations. It would be great to have native support for arbitrary binary operators. 

### Additional context

_No response_

cc @ezyang @gchanan @zou3519 @soumith @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 @Chillee @samdow @kshitij12345 @janeyx99
test_ddp_apply_optim_in_backward in distributed_test.py fails for gloo backend
Labels: oncall: distributed, triaged

### 🐛 Describe the bug

Command:

`BACKEND=gloo WORLD_SIZE=2 /usr/bin/python distributed/test_distributed_spawn.py -k cuda -k Cuda -k CUDA -k gpu -k nccl -k DistributedDataParallel -k DDP -v TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward  `

`BACKEND=gloo WORLD_SIZE=2 /usr/bin/python distributed/test_distributed_spawn.py -k cuda -k Cuda -k CUDA -k gpu -k nccl -k DistributedDataParallel -k DDP -v TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false`

View mismatch:

Mismatched elements: 57026 / 147456 (38.7%)
Greatest absolute difference: 0.058268263936042786 at index (120, 88, 0, 2) (up to 1e-05 allowed)
Greatest relative difference: 13452.156564975463 at index (101, 97, 1, 1) (up to 1.3e-06 allowed)
Params not equal at iteration 0


### Versions

pytorch commit: bc438af6fed4fe1fd0ed80e6d5f5ea17c3ca30bb

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Add log highlights to Dr. CI's failed jobs
Labels: triaged

When a job fails, the Dr CI message should call out the specific log lines that show the failure, showing the text that HUD shows today:

![Image](https://user-images.githubusercontent.com/4468967/221003507-23c99dd9-4bf7-4c9d-89a5-e6530c14d98d.png)
(The old Dr CI had a similar feature)

Expected benefits:
- Devs can see the exact failure faster
- Devs are less likely to incorrectly assume that a failure is unrelated based just on the job name

Feature inspired by https://github.com/pytorch/pytorch/pull/95232#issuecomment-1442089616
Add oscillating activation functions to PyTorch.
Labels: module: loss, triaged, needs research

### 🚀 The feature, motivation and pitch

Oscillating activation functions enable the neurons to learn the XOR function without manual feature engineering. With this kind of activation function, a single neuron can learn to approximate an XOR-like function. Otherwise, it needs multiple hyperplanes from multiple neurons. These activation functions also reduce training time and allow problems to be solved with smaller networks. The experimental results indicate that these activation functions are computationally cheaper than the state-of-art activation functions.

There are four oscillating activation functions to be added:
*	Growing Cosine Unit (GCU): f(z) = zcos(z)
*	Shifted Quadratic Unit (SQU): f(z) = $z^2$ + z
*	Decaying Sine Unit (DSU): f(z) = $\pi$/2(sinc(z - $\pi$) - sinc(z + $\pi$))
*	Non-Monotonic Cubic Unit (NCU): f(z) = z - $z^3$

### Alternatives

_No response_

### Additional context

## **Prior Art**
*	Noel, Mathew Mithra, Advait Trivedi, and Praneet Dutta. "Growing cosine unit: A novel oscillatory activation function that can speedup training and reduce parameters in convolutional neural networks." arXiv preprint arXiv:2108.12943 (2021).
*	Noel, Matthew Mithra, Shubham Bharadwaj, Venkataraman Muthiah-Nakarajan, Praneet Dutta, and Geraldine Bessie Amali. "Biologically inspired oscillating activation functions can bridge the performance gap between biological and artificial neurons." arXiv preprint arXiv:2111.04020 (2021).
*       https://analyticsindiamag.com/how-oscillatory-activation-function-overcomes-problems-with-gradient-descent-and-xor/
*       https://twitter.com/martin_gorner/status/1463294997436899332?lang=en

The most interesting part is that a single neuron activated by the oscillating activation units can learn to represent the XOR logical function, something that is known to be impossible for other artificial neuron designs (one needs multiple layers with those).

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
Circular padding error for 3D arrays
Labels: triaged, module: padding

### 🐛 Describe the bug

An error is raised when trying to do circular padding for a 3D array
```
import torch
import torch.nn.functional as F
F.pad(torch.arange(9).reshape(1, 3,3), (0,1,1,0), mode='circular')
```
Error:
```
RuntimeError: Invalid padding size, expected 2 but got 4
```

A current workaround is to get a view of the array with an extra dimension:
```
import torch
import torch.nn.functional as F
F.pad(torch.arange(9).reshape(1, 1, 3,3), (0,1,1,0), mode='circular')
```
which runs fine.

### Versions

PyTorch version: 1.13.0+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Enterprise
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.8.15 | packaged by conda-forge | (default, Nov 22 2022, 08:43:00) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19044-SP0
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=1805
DeviceID=CPU0
Family=198
L2CacheSize=5120
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=1805
Name=11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.5
[pip3] torch==1.13.0
[pip3] torch-tb-profiler==0.4.0
[pip3] torchvision==0.14.0
[pip3] opencv==4.6.0
[conda] blas 1.0 mkl
[conda] mkl 2021.4.0 haa95532_640
[conda] mkl-service 2.4.0 py38h2bbff1b_0
[conda] mkl_fft 1.3.1 py38h277e83a_0
[conda] mkl_random 1.2.2 py38hf11a4ad_0
[conda] numpy 1.23.0 pypi_0 pypi
[conda] numpy-base 1.23.5 py38h4da318b_0
[conda] torch 1.13.0 pypi_0 pypi
[conda] torch-tb-profiler 0.4.0 pypi_0 pypi
[conda] torchvision 0.14.0 pypi_0 pypi
`torch.distributed.Store` triggers INTERNAL ASSER FAILED when seting
Labels: oncall: distributed, triaged

### 🐛 Describe the bug

`torch.distributed.Store` triggers INTERNAL ASSER FAILED when seting

```py
import torch

torch.manual_seed(420)

store = torch.distributed.Store()
key = "key"
value = "value"

for i in range(100):
    store.set(key+str(i), value+str(i))
# RuntimeError: fn INTERNAL ASSERT FAILED at 
# "/opt/conda/conda-bld/pytorch_1672906354936/work/torch/csrc/distributed/c10d/init.cpp":138, 
# please report a bug to PyTorch.
```

### Versions

```
PyTorch version: 2.0.0.dev20230105
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.9.15 (main, Nov 24 2022, 14:31:59)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090

Nvidia driver version: 515.86.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==2.0.0.dev20230105
[pip3] torchaudio==2.0.0.dev20230105
[pip3] torchvision==0.15.0.dev20230105
[conda] blas                      1.0                         mkl
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py39h7f8727e_0
[conda] mkl_fft                   1.3.1            py39hd3c417c_0
[conda] mkl_random                1.2.2            py39h51133e4_0
[conda] numpy                     1.23.5           py39h14f4228_0
[conda] numpy-base                1.23.5           py39h31eccc5_0
[conda] pytorch                   2.0.0.dev20230105 py3.9_cuda11.7_cudnn8.5.0_0    pytorch-nightly
[conda] pytorch-cuda              11.7                 h67b0de4_2    pytorch-nightly
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] torchaudio                2.0.0.dev20230105      py39_cu117    pytorch-nightly
[conda] torchtriton               2.0.0+0d7e753227            py39    pytorch-nightly
[conda] torchvision               0.15.0.dev20230105      py39_cu117    pytorch-nightly
```

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
`torch.cartesian_prod` returns inconsistent dimensions with only one input
Labels: triaged, module: linear algebra

### 🐛 Describe the bug

When calling `torch.cartesian_prod` with a single tensor, the result is a 1D tensor. Conversely, when calling it with multiple tensors, the result is 2D. I would expect that `torch.cartesian_prod` always returns a 2D tensor for consistency.

Further, this is what `itertools.product` would do, which this function is modelled on.

## MWE
```python
import itertools
import torch

torch.cartesian_prod(
    torch.tensor([1,2,3])
)
# Output: tensor([1,2,3])
# Expected: tensor([[1], [2], [3]])

list(itertools.product([1, 2, 3]))
# Output: [(1,), (2,), (3,)]
```

### Versions

```
Python version: 3.10.9 (main, Jan 11 2023, 15:21:40) [GCC 11.2.0] (64-bit runtime)

Versions of relevant libraries:
[pip3] botorch==0.8.1
[pip3] gpytorch==1.9.1
[pip3] numpy==1.24.2
[pip3] torch==1.13.1+cpu
[conda] botorch                   0.8.1                    pypi_0    pypi
[conda] gpytorch                  1.9.1                    pypi_0    pypi
[conda] numpy                     1.24.2                   pypi_0    pypi
[conda] torch                     1.13.1+cpu               pypi_0    pypi
```

cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano
Continuous dropout layer
Labels: module: nn, triaged, enhancement

### 🚀 The feature, motivation and pitch

Hello!

I am working on information theory application for neural networks [(here)](https://openreview.net/forum?id=bQB6qozaBw).
With my research I show that for valid mutual information measurements one might apply continuous dropout. While continuous dropout was considered already in the original paper introducing dropout, the implementation of it is not unified and not added to the library. From my perspective it can be a large benefit to add a class for Gaussian dropout for example, or maybe for a dropout with noise sampled from any custom distribution.

Since I implemented it for my project, I am willing to create this contribution if it is considered interesting.

### Alternatives

_No response_

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @saketh-are
Make this ridiculously long error message more user friendly
Labels: triaged, module: infra

Via @janeyx99: "highlighting the longest merge rejection message i've seen https://github.com/pytorch/pytorch/pull/92625#issuecomment-1439109986" 

It's about 3 pages long, listing the full GQL query that failed:
![Image](https://user-images.githubusercontent.com/4468967/220470260-47c50c75-6274-4117-9e91-3e246806f89d.png)
Pytorch profiler stack exporting does not work
Labels: oncall: profiler

### 🐛 Description

While trying to export stacks to profile CUDA kernels runtime performance via exporting stacks directly following [public documentation](https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html#examining-stack-traces) one can easily find empty stacks file which is unexpected. To be more specific, the following code

```
with torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA], with_stack=True) as prof:
      self._training_step(...)

prof.export_stacks("stacks.txt", "self_cuda_time_total")
```

saves nothing 

```
cat stacks.txt | wc -l
0
```

The `training_step` above is an arbitrary's model regular backpropagation step like:

```
...
loss = model(**X, **y).loss
optimizer.zero_grad()
loss.backward()
optimizer.step()
...
```

Notes:
 - Nonetheless profiler does look to be working, e.g. calling `print(prof.key_averages().table(sort_by="self_cuda_time_total",
                                    row_limit=10))` results into:
    <img width="1201" alt="vanilla_attn_bf_16_100fwdbkwd_steps" src="https://user-images.githubusercontent.com/14129555/220458397-51e62a9b-9881-4c6a-8d21-ba58fc9a2380.png">
 - [Similar issue reported just recently.](https://discuss.pytorch.org/t/pytorch-profiler-not-exporting-any-stack-information/167477)

### Versions

- Torch version: 1.13.1
- GPU: A100
- CUDA: 11.7 / Driver Version: 515.65.01
- OS: Ubuntu XX.YY

cc @robieta @chaekit @aaronenyeshi @ngimel @nbcsm @guotuofeng @guyang3532 @gaoteng-git @tiffzhaofb @dzhulgakov @davidberard98
ONNX Exporter for circular padding mode in convolution ops
Labels: module: onnx, triaged

### 🚀 The feature, motivation and pitch

Circular/wrap-around padding has recently been added to the Pad operator in ONNX opset 19:
https://github.com/onnx/onnx/blob/main/docs/Operators.md#Pad

When exporting convolution operators with `padding_mode` set to `circular`, one can directly make use of the new Pad operator instead of having to do concatenation.

### Alternatives

_No response_

### Additional context

_No response_
Remove conda virtualenv from the docker image
Labels: oncall: binaries, triaged, module: docker

### 🚀 The feature, motivation and pitch

Current docker image contains a conda virtualenv (`base`), in which all the libs are installed.

Since a docker image is already a kind a separated environement, I think a conda virtualenv is superfluous. 
Conda could be removed, and pytorch could be installed in the native python installl with `pip`.
This could simplify the docker image, and possibly lighten it.

Unless there is another reason to use conda which I'm not aware of ?



cc @ezyang @seemethere @malfet
High Cuda Memory Consumption for Simple ResNet50 Inference
Labels: oncall: jit

### 🐛 Describe the bug

I run a simple inference of a jit::traced Resnet50 model with the C++ API and record a peak GPU memory consumption of 1.2gb. The inference script is taken from the docs https://pytorch.org/tutorials/advanced/cpp_export.html  and looks as follows:
```
#include <torch/script.h>  // One-stop header.

#include <iostream>
#include <memory>
#include <vector>

int main(int argc, const char* argv[]) {
    if (argc != 2) {
        std::cerr << "usage: example-app <path-to-exported-script-module>\n";
        return -1;
    }

    {
        c10::InferenceMode guard;
        torch::jit::script::Module module;
        try {
            // Deserialize the ScriptModule from a file using
            // torch::jit::load().
            module = torch::jit::load(argv[1]);
            module.to(at::kCUDA);
        } catch (const c10::Error& e) {
            std::cerr << "error loading the model\n";
            return -1;
        }

        std::vector<torch::jit::IValue> inputs;
        at::Tensor rand = torch::rand({1, 3, 224, 224});
        at::Tensor rand2 = at::_cast_Half(rand);
        inputs.push_back(rand2.to(at::kCUDA));
        at::Tensor output = module.forward(inputs).toTensor();

        std::cout << output.slice(/*dim=*/1, /*start=*/0, /*end=*/5) << '\n';
    }

    std::cout << "ok\n";
}
```

When running the script with a traced model input (converted to half), the memory consumption goes to 1.2gb. As a comparison:
- A back-of-the-envelope calculation: The model has about 40m parameters and the input is of size 3x224x224 , so I think a very, very memory-efficient inference should be at ~100-150 mb. 
- TensorRT uses ~260mb for the inference. 

We would like to use the libtorch for inference in production and I would be great if there was a way to reduce the memory consumption to ~500mb. Is there a way to do that?




### Versions

build-version in the libtorch folder shows:
```
1.13.1+cu116
```
Should I collect other relevant info? 

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
DISABLED test_memory_format_nn_ConvTranspose2d_cuda_complex32 (__main__.TestModuleCUDA)
Labels: module: nn, triaged, module: flaky-tests, skipped

Platforms: rocm

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/failure/test_memory_format_nn_ConvTranspose2d_cuda_complex32) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/11473508755).

Over the past 72 hours, it has flakily failed in 2 workflow(s).

**Debugging instructions (after clicking on the recent samples link):**
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Grep for `test_memory_format_nn_ConvTranspose2d_cuda_complex32`


Test file path: `test_modules.py`

cc @albanD @mruberry @jbschlosser @walterddr @saketh-are
COO @ COO tries to allocate way too much memory on CUDA
Labels: module: sparse, module: cuda, triaged, matrix multiplication

### 🐛 Describe the bug

```python
In [1]: import torch

In [2]: x = torch.rand(1000, 1000, device='cuda')

In [3]: %timeit x @ x
19.7 µs ± 10.9 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [4]: %timeit x @ x.to_sparse()
21 ms ± 598 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [5]: %timeit x.to_sparse() @ x
30.9 ms ± 72.5 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [6]: %timeit x.to_sparse() @ x.to_sparse()
---------------------------------------------------------------------------
OutOfMemoryError                          Traceback (most recent call last)
Input In [6], in <cell line: 1>()
----> 1 get_ipython().run_line_magic('timeit', 'x.to_sparse() @ x.to_sparse()')

File ~/.conda/envs/pytorch-cuda-dev-nik/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2305, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth)
   2303     kwargs['local_ns'] = self.get_local_scope(stack_depth)
   2304 with self.builtin_trap:
-> 2305     result = fn(*args, **kwargs)
   2306 return result

File ~/.conda/envs/pytorch-cuda-dev-nik/lib/python3.10/site-packages/IPython/core/magics/execution.py:1162, in ExecutionMagics.timeit(self, line, cell, local_ns)
   1160 for index in range(0, 10):
   1161     number = 10 ** index
-> 1162     time_number = timer.timeit(number)
   1163     if time_number >= 0.2:
   1164         break

File ~/.conda/envs/pytorch-cuda-dev-nik/lib/python3.10/site-packages/IPython/core/magics/execution.py:156, in Timer.timeit(self, number)
    154 gc.disable()
    155 try:
--> 156     timing = self.inner(it, self.timer)
    157 finally:
    158     if gcold:

File <magic-timeit>:1, in inner(_it, _timer)

OutOfMemoryError: CUDA out of memory. Tried to allocate 22.37 GiB (GPU 0; 5.78 GiB total capacity; 71.10 MiB already allocated; 4.99 GiB free; 86.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

```

### Versions

Current master.

cc @alexsamardzic @pearu @cpuhrsch @amjames @bhosmer @ngimel
Reversing along a dimension, similarly to numpy
Labels: feature, triaged, module: numpy, module: advanced indexing

### 🚀 The feature, motivation and pitch

In `numpy` you can reverse an array as `arr[::-1]` along any dimension. The same in `torch` raises an error: `ValueError: step must be greater than zero`. It would be useful if both operated the same way.

### Alternatives

_No response_

### Additional context

_No response_

cc @mruberry @rgommers
Whether to consider native support for intel gpu？
Labels: triaged, module: intel

### 🚀 The feature, motivation and pitch

I hope that the support for intel gpu can be implemented like cuda, rather than in the form of plug-ins。I wonder if you have such plans in the future

### Alternatives

_No response_

### Additional context

_No response_

cc @frank-wei @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10