gopikrsmscs/torch-issues huggingface dataset:
Convention for printing the "internal representation" of compiled functions from inductor/other backends
Labels: triaged, module: inductor

### 🐛 Describe the bug

It doesn't really make sense to print a function wrapped in torch.compile, but as a framework developer sometimes I am dealing with compiled_fn prior to frame installation and it would be nice to be able to debug print them and see what was compiled with them. Typically you don't get anything useful because it's some trampoline function (like `align_inputs`) before eventually getting to some eval'ed code. But maybe keeping the graph around for printing here would be handy? IDK, food for thought.

BTW, printing compiled_fn (via a series of hacks) is how I diagnosed #93308

### Versions

master

cc @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire
Bug in torch.linalg.svd 
Labels: triaged, module: mkl

### 🐛 Describe the bug

`torch.linalg.svd` produces an error for some sufficiently large inputs. As the error message says this is most likely a bug in the implementation calling the backend library. 

**Example (run on CPU):** 
```
import torch 
A = torch.randn(172032,30000)
U,S,V = torch.linalg.svd(A,full_matrices=False)
```
yields 
```
Intel MKL ERROR: Parameter 12 was incorrect on entry to SGESDD.
Traceback (most recent call last):

  File "<my_path>/lib/python3.8/site-packages/spyder_kernels/py3compat.py", line 356, in compat_exec
    exec(code, globals, locals)

  File "<my_home_path>/PythonFiles/pytorch_test.py", line 14, in <module>
    U,S,V = torch.linalg.svd(A,full_matrices=False)

RuntimeError: false INTERNAL ASSERT FAILED at "../aten/src/ATen/native/LinearAlgebraUtils.h":289, please report a bug to PyTorch. linalg.svd: Argument 12 has illegal value. Most certainly there is a bug in the implementation calling the backend library.
```

### Versions

PyTorch version: 1.12.0+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Nov 14 2022, 12:59:47)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: 
GPU models and configuration: GPU 0: NVIDIA RTX A6000
Nvidia driver version: 515.86.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mpi4torch==0.1.0
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.0
[pip3] numpydoc==1.4.0
[pip3] torch==1.12.0
[pip3] torchvision==0.13.0
[conda] No relevant packages
Bad conversion from torch.split(2d_tensor,splitsize_list) to SplitToSequence OP (onnx export)
Labels: module: onnx, triaged

### 🐛 Describe the bug

```python
torch.split(concat_feat_mat,idx_list)
```
When I use the instruction in model define and try to export it as onnx model, some execution errors are found. 

> the onnx converter will construct a SplitToSequence with seq(tensor(int64)) type input, which is not supported.

### Versions

PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 10.15.7 (x86_64)
GCC version: Could not collect
Clang version: 11.0.3 (clang-1103.0.32.62)
CMake version: version 3.23.2
Libc version: N/A

Python version: 3.10.8 (main, Nov 24 2022, 08:09:04) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-10.15.7-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] pytorch-model-summary==0.1.1
[pip3] torch==1.13.1
[pip3] torch-tb-profiler==0.4.1
[pip3] torchaudio==0.13.1
[pip3] torchinfo==1.7.1
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.14.1
[conda] blas                      1.0                         mkl  
[conda] ffmpeg                    4.3                  h0a44026_0    pytorch
[conda] mkl                       2021.4.0           hecd8cb5_637  
[conda] mkl-service               2.4.0           py310hca72f7f_0  
[conda] mkl_fft                   1.3.1           py310hf879493_0  
[conda] mkl_random                1.2.2           py310hc081a56_0  
[conda] numpy                     1.23.5          py310h9638375_0  
[conda] numpy-base                1.23.5          py310ha98c3c9_0  
[conda] pytorch                   1.13.1                 py3.10_0    pytorch
[conda] pytorch-model-summary     0.1.1                      py_0    conda-forge
[conda] torch-tb-profiler         0.4.1                    pypi_0    pypi
[conda] torchaudio                0.13.1                py310_cpu    pytorch
[conda] torchinfo                 1.7.1              pyhd8ed1ab_0    conda-forge
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.14.1                py310_cpu    pytorch
Well known way to request user backtrace when inside Dynamo
Labels: triaged, module: dynamo

### 🐛 Describe the bug

A nice debugging tool is to save tracebacks from library code for later. However, if you are symbolically evaluating a Dynamo frame, a user is typically interested in the USER frame, not the framework frame (which is a bunch of impenetrable gobbeldygook.) Need some global function that makes it easy to get what the current user frame (as tracked by Dynamo) is.

### Versions

master

cc @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
Minifier produces minifier script that doesn't fail accuracy on Background_Matting (dynamic shapes, inductor, inference)
Labels: triaged, module: fx, module: functorch

### 🐛 Describe the bug

Original command: `TORCHDYNAMO_REPRO_AFTER="dynamo" TORCHDYNAMO_REPRO_LEVEL=4 python benchmarks/dynamo/torchbench.py --accuracy --ci --inductor --explain --only Background_Matting --dynamic-shapes`

Produced script: https://gist.github.com/aaeef0a8803356d35c5b3fcbc535a506 (after fixing bilinear problems)

```
$ python /data/users/ezyang/b/pytorch/torch_compile_debug/run_2023_01_29_10_36_06_349645/minifier/minifier_launcher.py
/data/users/ezyang/b/pytorch/torch/_inductor/compile_fx.py:89: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
  warnings.warn(
[2023-01-29 10:37:31,679] torch._dynamo.debug_utils: [ERROR] Input graph does not fail accuracy testing
```

### Versions

master 7e449e8ba701046bde5512aad60403a2b22e98a9

cc @SherlockNoMad @soumith @EikanWang @jgong5 @wenzhe-nrv @zou3519 @Chillee @samdow @kshitij12345 @janeyx99
Minifier does not run on LearningToPaint (dynamic shapes, inductor, inference)
Labels: triaged, module: inductor

### 🐛 Describe the bug

LearningToPaint fails accuracy with inference dynamic shapes inductor. However, the minifier never triggers even when you ask for it:


```
(/home/ezyang/local/b/pytorch-env) [ezyang@devgpu020.ftw1 ~/local/b/pytorch (dbaba22e)]$ TORCHDYNAMO_REPRO_
AFTER="aot" TORCHDYNAMO_REPRO_LEVEL=4 python benchmarks/dynamo/torchbench.py --accuracy --ci --inductor --explain --only LearningToPaint  --dynamic-shapes                                                            cuda eval  LearningToPaint                     [2023-01-29 10:24:26,025] torch._dynamo.utils: [ERROR] RMSE 
(res-fp64): 0.00007, (ref-fp64): 0.00000 and shape=torch.Size([4, 65])                                     
FAIL                                                                                                       
Dynamo produced 1 graph(s) covering 72 ops                                                                 
(/home/ezyang/local/b/pytorch-env) [ezyang@devgpu020.ftw1 ~/local/b/pytorch (dbaba22e)]$ TORCHDYNAMO_REPRO_
AFTER="dynamo" TORCHDYNAMO_REPRO_LEVEL=4 python benchmarks/dynamo/torchbench.py --accuracy --ci --inductor 
--explain --only LearningToPaint  --dynamic-shapes                                                         
cuda eval  LearningToPaint                     [2023-01-29 10:25:25,055] torch._dynamo.utils: [ERROR] RMSE 
(res-fp64): 0.00007, (ref-fp64): 0.00000 and shape=torch.Size([4, 65])                                     
FAIL                                                                                                       Dynamo produced 1 graph(s) covering 72 ops  
```

### Versions

master

cc @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire
Add Support for RockChip NPUs (RKNN(2)) 
Labels: triaged, module: arm

### 🚀 The feature, motivation and pitch

The new generation of RockChip Socs have been released for a long time and rockchip’s Socs, they have a huge share of the arm SBC market and are pretty common in the field of SBC PCs, and they’ve been set up NPUs for some of their mainstream products for example like RK3399Pro and RK3588Family and some more, it means a lot to us and has a huge development potential, and FastDeploy from PaddlePaddle https://github.com/PaddlePaddle/FastDeploy already supported RKNN2 for RK3588Family, we hope to see PyTorch supported as well ! Thanks in advance.
Best Regards


### Alternatives

https://github.com/PaddlePaddle/FastDeploy
The following project have already been supported 
Might be referable 

### Additional context

_No response_

cc @malfet
Support for VeLO optimizer.
Labels: module: optimizer, triaged, needs research

### 🚀 The feature, motivation and pitch

VeLO: Training Versatile Learned Optimizers by Scaling Up  
https://arxiv.org/abs/2211.09760  
the velo optimizer sames very good, is there any plan to support it?

### Alternatives

_No response_

### Additional context

_No response_

cc @vincentqb @jbschlosser @albanD @janeyx99
Dynamo doesn't support dict(list_argument)
Labels: triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

```
import torch._dynamo

def f(x):
    y = dict([('a', x)])
    return y['a']

print(torch._dynamo.export(f, torch.randn(3)))
```

gives

```
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/symbolic_convert.py", line 526, in step
    getattr(self, inst.opname)(inst)
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/symbolic_convert.py", line 332, in wrapper
    return inner_fn(self, inst)
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/symbolic_convert.py", line 999, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/symbolic_convert.py", line 460, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/variables/builtin.py", line 341, in call_function
    return super().call_function(tx, args, kwargs)
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/variables/base.py", line 230, in call_function
    unimplemented(f"call_function {self} {args} {kwargs}")
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/exc.py", line 71, in unimplemented
    raise Unsupported(msg)
torch._dynamo.exc.Unsupported: call_function BuiltinVariable(dict) [ListVariable()] {}

from user code:
   File "/data/users/ezyang/b/pytorch/a.py", line 5, in f
    y = dict([('a', x)])

Set torch._dynamo.config.verbose=True for more information
```

This is extracted from torchvision maskrcnn, but actually it doesn't use dict, it uses OrderedDict (but see https://github.com/pytorch/pytorch/issues/93151 ). But dict ought to work too.

### Versions

master

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh @mlazos @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
Dynamo doesn't support OrderedDict
Labels: triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

```
import torch._dynamo
from collections import OrderedDict

def f(x):
    y = OrderedDict()
    y['a'] = x
    return y['a']

print(torch._dynamo.export(f, torch.randn(3)))
```

gives

```
    out_code = transform_code_object(code, transform)
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/bytecode_transformation.py", line 341, in transform_code_object
    transformations(instructions, code_options)
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/convert_frame.py", line 310, in transform
    tracer.run()
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/symbolic_convert.py", line 1717, in run
    super().run()
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/symbolic_convert.py", line 563, in run
    and self.step()
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/symbolic_convert.py", line 526, in step
    getattr(self, inst.opname)(inst)
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/symbolic_convert.py", line 332, in wrapper
    return inner_fn(self, inst)
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/symbolic_convert.py", line 999, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/symbolic_convert.py", line 460, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/variables/misc.py", line 723, in call_function
    unimplemented("call_function in skip_files " + path)
  File "/data/users/ezyang/b/pytorch/torch/_dynamo/exc.py", line 71, in unimplemented
    raise Unsupported(msg)
torch._dynamo.exc.Unsupported: call_function in skip_files /home/ezyang/local/b/pytorch-env/lib/python3.9/collections/__init__.py

from user code:
   File "/data/users/ezyang/b/pytorch/a.py", line 5, in f
    y = OrderedDict()
```

This code was extracted from torchvision maskrcnn.

### Versions

master

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh @mlazos @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire
[RFC] Flop counters in PyTorch
Labels: feature, triaged, module: python frontend

### 🚀 The feature, motivation and pitch

Lots of people have built flop counters/asked for FLOP counters (or other performance counters) in PyTorch. There's a litany of requests, but some common ones include: 1. coverage (i.e. no restrictions on how the model is written, capturing backwards, etc.), 2. module-level grouping, 3. customization (i.e. being able to modify flop formulas), and 4. Capturing metrics other than FLOPs (like memory bandwidth).

There's another factor that's relevant with `PyTorch 2.0`, which is whether it works in eager-mode or under `torch.compile`. Some of these features are a lot of easier to do in eager mode (coverage/module-level grouping), and others are a lot easier to do in compiled mode (capturing memory bandwidth or non-matmul FLOPs).

I propose we add 2 different flop counters to PyTorch, one that's eager based and one that's graph based.

The eager one (implemented similarly to [this](https://dev-discuss.pytorch.org/t/the-ideal-pytorch-flop-counter-with-torch-dispatch/505/5)).
- Works with all operations in eager mode (forwards, backwards, checkpointing, functorch, w.e.)
- Only counts GMACs (no memory accesses, no non-matmul FLOPs)
- Provides module-level breakdowns
- Provides a context manager API where it counts all flops executed within the context manager.
- Provides an API where you can override formulas for each (aten) op.

Compiled flop counter:
- Works with ... most operations in eager mode (the stuff that torch.compile can compile at least).
- Can count FLOPs (for some formula for it...) and memory accesses.
- Does not provide module-level breakdowns.
- Probably also a context manager API.
- No easy API for overriding formulas for each op (I guess we can provide a way to say, map cos to however many flops you want, but that's less clear).

cc: @Jokeren @rwightman, @ppwwyyxx. @ezyang

### Alternatives

N/A

### Additional context

_No response_

cc @albanD
DISABLED test_inplace_grad_index_put_cuda_float64 (__main__.TestBwdGradientsCUDA)
Labels: module: cuda, triaged, module: flaky-tests, skipped, module: unknown, oncall: pt2

Platforms: inductor

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_inplace_grad_index_put_cuda_float64&suite=TestBwdGradientsCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/10898454314).

Over the past 3 hours, it has been determined flaky in 3 workflow(s) with 3 failures and 3 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_inplace_grad_index_put_cuda_float64`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


cc @ptrblck @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @ngimel
DISABLED test_forward_mode_AD_linalg_det_singular_cuda_complex128 (__main__.TestFwdGradientsCUDA)
Labels: module: autograd, module: rocm, triaged, module: flaky-tests, skipped

Platforms: rocm

This test was disabled because it is failing on master ([recent examples](http://torch-ci.com/failure/FAILED%20test_ops_fwd_gradients.py%3A%3ATestFwdGradientsCUDA%3A%3Atest_forward_mode_AD_linalg_det_singular_cuda_complex128%20-%20torch.autograd.gradcheck.GradcheckError%3A%20While%20considering%20the%20imaginary%20part%20of%20complex%20inputs%20only%2C%20Jacobian%20computed%20with%20forward%20mode%20mismatch%20for%20output%200%20with%20respect%20to%20input%200%2C)).

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport
numpy v1.24 does not work with `writer.add_histogram`
Labels: triaged, module: tensorboard

### 🐛 Describe the bug

https://github.com/pytorch/pytorch/blob/664058fa83f1d8eede5d66418abff6e20bd76ca8/torch/utils/tensorboard/summary.py#L380

There is a problem when running this using the latest numpy v1.24.

This is an example of the type of error I get:
```
>>> import numpy as np
>>> a = np.random.randint(0, 100, 10)
>>> np.greater(a, 0, dtype=np.int64)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: No loop matching the specified signature and casting was found for ufunc greater
```
When using any version of numpy 1.23 or earlier, this error does not come up.

### Versions

```
Collecting environment information...
PyTorch version: 1.12.1
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.1 LTS (x86_64)
GCC version: (Ubuntu 8.4.0-1ubuntu1~18.04) 8.4.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.25.2
Libc version: glibc-2.27

Python version: 3.8.15 | packaged by conda-forge | (default, Nov 22 2022, 08:49:35)  [GCC 10.4.0] (64-bit runtime)
Python platform: Linux-4.15.0-136-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.1.74
CUDA_MODULE_LOADING set to: 
GPU models and configuration: 
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti
GPU 2: TITAN V
GPU 3: TITAN V
GPU 4: TITAN V
GPU 5: Tesla V100-PCIE-16GB
GPU 6: Quadro GP100
GPU 7: Quadro GP100

Nvidia driver version: 455.23.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] torch==1.12.1
[pip3] torchaudio==0.12.1
[pip3] torchvision==0.13.1
[conda] blas                      2.116                       mkl    conda-forge
[conda] blas-devel                3.9.0            16_linux64_mkl    conda-forge
[conda] cudatoolkit               11.3.1              h9edb442_11    conda-forge
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] libblas                   3.9.0            16_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            16_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            16_linux64_mkl    conda-forge
[conda] liblapacke                3.9.0            16_linux64_mkl    conda-forge
[conda] mkl                       2022.1.0           h84fe81f_915    conda-forge
[conda] mkl-devel                 2022.1.0           ha770c72_916    conda-forge
[conda] mkl-include               2022.1.0           h84fe81f_915    conda-forge
[conda] numpy                     1.24.1           py38hab0fcb9_0    conda-forge
[conda] pytorch                   1.12.1          py3.8_cuda11.3_cudnn8.3.2_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.12.1               py38_cu113    pytorch
[conda] torchvision               0.13.1               py38_cu113    pytorch
```
ptxas segfault with PT2
Labels: triaged, oncall: pt2, module: inductor

### 🐛 Describe the bug

Sometimes ptxas segfaults when you trigger it via Triton/Inductor/PT2. Apparently the new Triton MLIR rewrite is supposed to fix it.

It looks something like this

```
(/home/ezyang/local/a/pytorch-env) [ezyang@devgpu020.ftw1 ~/local/a/pytorch (20bf77f9)]$ python benchmark
s/dynamo/torchbench.py --ci --amp  --inductor --accuracy --training --only dlrm                          
cuda train dlrm                               sh: line 1: 3723415 Segmentation fault      (core dumped) p
txas -v --gpu-name=sm_80 /tmp/file1TgA5S -o /tmp/file1TgA5S.o 2> /tmp/filebVYSuU                         
ERROR:common:Internal Triton PTX codegen error:                                                          
concurrent.futures.process._RemoteTraceback:                                                             
"""                                         
```

### Versions

master

cc @soumith @msaroufim @wconstab @ngimel @bdhirsh @mlazos @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire
[BE] Improve FSDP <> AC Unit Tests
Labels: oncall: distributed, triaged, module: fsdp

Composing PyTorch Fully Sharded Data Parallel (FSDP) and activation checkpointing (AC) is tested in [`test_fsdp_checkpoint.py`](https://github.com/pytorch/pytorch/blob/master/test/distributed/fsdp/test_fsdp_checkpoint.py). While we have used FSDP and AC together in several real workloads, our unit tests warrant a revamp. This issue tracks some questions and work items.

**Existing Unit Tests**
- [ ] What is the difference between `test_basic_checkpoint_end_to_end()` and `test_checkpoint_fsdp_wrapping()`? Is one a superset of the other?
    - It looks like `test_basic_checkpoint_end_to_end` does not use any nested application of FSDP or AC, while `test_checkpoint_fsdp_wrapping()` does. In that case, can we consolidate to just one with nested application (which represents real use cases)?
- [ ] `test_basic_checkpoint_end_to_end()` and `test_checkpoint_fsdp_wrapping()` test checkpointing the root module (i.e. entire model). Should we include that in our unit tests?
    - Checkpointing the root module implies re-running the entire forward pass in the backward pass and is not realistic.
    - FSDP <> AC does not preserve inductive tree structure. Even if `Checkpoint(FSDP(module))` works for a root module `module`, that does not imply that using that having `Checkpoint(FSDP(module))` as a submodule in a larger module tree will continue to work.
- [ ] `test_basic_checkpoint_end_to_end()` and `test_checkpoint_fsdp_wrapping()` do not test the non-reentrant checkpointing implementation. However, adding tests for it leads to some errors:
    - In `test_basic_checkpoint_end_to_end()`, `checkpointed_fsdp` fails for non-reentrant.
    - In `test_checkpoint_fsdp_wrapping()`, `ckpt_sequential_wrapped_fsdp` fails for non-reentrant. This one might be fixable by modifying FSDP's `pre_forward` and `post_forward` to skip if the handle training state is `BACKWARD_PRE`.

**Adding Unit Tests**
https://github.com/pytorch/pytorch/pull/92935 is a draft PR adding unit tests for several more ways to compose FSDP with AC. We may choose to land or not depending on feedback.

After resolving these items, we should consider doing the same for `fully_shard` <> `_composable.checkpoint`.

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501
Feature request: access to variable
Labels: oncall: transformer/mha

### 🚀 The feature, motivation and pitch

In `torch.nn.functional.multi_head_attention_forward`, there is a variable, `use_separate_proj_weight` that can not be controled by the users of Transformers. I would have liked to use NON-shared weights.

In `torch.nn.modules.transformers.TransformerEncoderLayer.self_attn`, during the forward, use_separate_proj_weight is set to False only in the case where the dimensions for Q, K, V are not the same.  This is verified during initialisation of self_attn. But user can't control this if dimensions are the same.
(same with the decoder)

Could you add an option to control if weights are shared? Thank you very much!

### Alternatives

_No response_

### Additional context

_No response_

cc @jbschlosser @bhosmer @cpuhrsch @erichan1
Test Failure: TestUpgraders.test_aten_div_scalar_at_3 on a big-endian machine (issue in torch.jit.load())
Labels: oncall: jit

### 🐛 Describe the bug

When I execute the following test case on a big-endian machine, I got the test failure due to the runtime error. We expect that this test should also pass on a big-endian machine.

```python
$ python test/test_jit.py TestUpgraders.test_aten_div_scalar_at_3
CUDA not available, skipping tests
monkeytype is not installed. Skipping tests for Profile-Directed Typing
E
======================================================================
ERROR: test_aten_div_scalar_at_3 (jit.test_upgraders.TestUpgraders)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ishizaki/PyTorch/master/test/jit/test_upgraders.py", line 246, in test_aten_div_scalar_at_3
    loaded_model = torch.jit.load(model_path)
  File "/home/ishizaki/PyTorch/master/torch/jit/_serialization.py", line 162, in load
    cpp_module = torch._C.import_ir_module(cu, str(f), map_location, _extra_files)
RuntimeError: Unexpected end of pickler archive.
```

The minimum program to reproduce this issue is here. So, `torch.jit.load()` does not seem to work correctly on a big-endian platform.   

```python
import torch
loaded_model = torch.jit.load("test/jit/fixtures/test_versioned_div_scalar_float_v3.pt")
```

I believe that `torch.jit.load()` reads the multiple-byte data without swapping on a big-endian machine while pickle serialization store data as mostly little-endian (as shown in #92910).

### Versions

```
PyTorch version: 2.0.0a0+git215f4fc
Is debug build: True
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (s390x)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-5.4.0-125-generic-s390x-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] torch==2.0.0a0+git17c570c
[conda] Could not collect
```

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Inplace fused (leaky)relu+(leaky)dropout for memory savings (I think, can be made fully allocation-less if never fully allocating random mask in FlashAttention style and recover the mask from the output)
Labels: feature, triaged, oncall: pt2, module: aotdispatch

### 🚀 The feature, motivation and pitch

It's possible that such an inplace fusion is already possible with dynamo, but if not, it's quite good to have for saving memory. Also maybe worth introducing some torch.nn.fused namespace which would have torch.nn.fused.ReLU_Dropout, torch.nn.fused.Conv2d_BatchNorm2d_LeakyReLU_Dropout or some other similar modules that in the ultimate case might be defined as `torch.compile(nn.Sequential(nn.ReLU(inplace=True), nn.Dropout(inplace=True)))`, but in the meanwhile can be hand-optimized for these very frequent cases and some documentation can exist which can present some guarantees and tests that these work and are expected to consume this much memory.

My earlier issue in torchvision:
https://github.com/pytorch/vision/issues/4851

### Alternatives

_No response_

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @saketh-are @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
Add Stride Argument For Constructors
Labels: triaged, enhancement, module: tensor creation

### 🚀 The feature, motivation and pitch

randn, full, ones, and zeros dont have a stride argument. The lack of the stride argument makes it difficult to write functional decompositions. For instance, `randn_like` I would like to decompose functionally, but because there is no way to pass along the stride of the input, I need to write it as the following:
```
@register_decomposition(aten.randn_like)
def randn_like(
    a,
    *,
    dtype: Optional[torch.dtype] = None,
    device: Optional[torch.device] = None,
    layout: Optional[torch.layout] = None,
    pin_memory: bool = False,
    memory_format: torch.memory_format = None,
):
    result = torch.empty_like(a, dtype=dtype, device=device, layout=layout, pin_memory=pin_memory, memory_format=memory_format)
    result.random_(0, 1)
    return result

```

### Alternatives

_No response_

### Additional context

_No response_

cc @gchanan @mruberry
functorch.functionalize doesn't error out with logcumsumexp.out
Labels: triaged, module: xla, module: meta tensors, module: functionalization

### 🐛 Describe the bug

logcumsumexp.out will error out if the given output buffer's dtype is different from the input, but somehow that property is violated after functionalization, here is the PoC:
```
import torch
import functorch


def poc6():
  device = 'cpu'
  axis = -1

  # Check input and inplace_output type mismatch
  b = torch.randn(5, 2, device=device, dtype=torch.float64)
  inplace_out = torch.zeros(5, 2, device=device, dtype=torch.float32)
  try:
    torch.logcumsumexp(b, axis, out=inplace_out)
  except Exception as e:
    print(e)



print("CPU")
poc6()
print()

print("CPU Functionalize")
functorch.functionalize(poc6)()
```

Output:
```
ptxla@t1v-n-307ffe96-w-0:/workspaces/work/pytorch/xla$ python poc6.py 
CPU
expected scalar_type Double but found Float

CPU Functionalize
```

### Versions

Nightly.

CC @bdhirsh @wonjoolee95 @JackCaoG 

cc @bdhirsh @ezyang @eellison @soumith
[LibTorch] pickle_save output cannot be reloaded using pickle_load in Windows
Labels: oncall: jit

### 🐛 Describe the bug

For the following code, it works in Mac but crashed in Windows for the last step

```
#include <torch/script.h>

torch::jit::IValue tensor(torch::randn({1, 64}));
std::vector<char> input = torch::jit::pickle_save(tensor);
tensor = torch::jit::pickle_load(input);
```
Error messages:
`Microsoft C++ exception: c10::Error at memory location 0x0000000C8155E460`

### Versions

LibTorch version is latest stable 1.13.1.

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Profiler documentation doesn't mention some exports are mutually exclusive
Labels: oncall: profiler

### 📚 The doc issue

The [profiler documentation](https://pytorch.org/docs/stable/profiler.html) and the tutorials on it don't mention that the tensorboard and chrome trace exports are mutually exclusive. I had setup an initial run to dump out all the supported formats of profiling information as I was unsure which tool would prove most useful, and when exporting a chrome trace after using the tensorboard output hook I get a stack trace like:

```
File "/path/to/venv/lib/python3.8/site-packages/torch/profiler/profiler.py", line 136, in export_chrome_trace
    return self.profiler.export_chrome_trace(path)
  File "/path/to/venv/lib/python3.8/site-packages/torch/autograd/profiler.py", line 278, in export_chrome_trace
    self.kineto_results.save(path)  # type: ignore[union-attr]
RuntimeError: Trace is already saved.
```

This is presumably related to [this line](https://github.com/pytorch/pytorch/blob/master/torch/csrc/profiler/kineto_shim.h#L112) which notes "Kineto's save is destructive" but as far as I can tell this fact isn't documented anywhere, and it probably should be.

### Suggest a potential alternative/fix

Add a note to the profiler docs page saying which export methods are mutually exclusive, and update the tutorials to note that.

cc @robieta @chaekit @aaronenyeshi @ngimel @nbcsm @guotuofeng @guyang3532 @gaoteng-git @tiffzhaofb
Enable OnDemand for Open Source CI 
Labels: triaged

### 🚀 The feature, motivation and pitch

### A Common Scenario
You submit PR to PyTorch and and a test fails on a CI machine that is significantly different from your local coding environment. 

Instead of using `with-ssh` you can quickly fire up a github codespaces or other OnDemand instance that has your code checked out and you are free to debug without the time limit.

### Alternatives

An alternative solution is to tag the failing pull request with the label `with-ssh`. This will allow a user to ssh into the CI machine and gives the user 2 hours (I think more time can be requested) to try and debug the error.  This is far better than nothing but for non vim users and general UX it can be hard to complete the required debug work.

### Additional context

Besides debugging purposes this could also be very helpful for first time contributors. Having a ready made environment that has been streamlined for PyTorch development.
Double free when running torch.linalg.ldl_solve
Labels: triaged, module: edge cases

### 🐛 Describe the bug



```
import torch
import numpy as np
arg_1_tensor = torch.rand([5, 5], dtype=torch.float32)
arg_1 = arg_1_tensor.clone()
arg_2_tensor = torch.randint(-2048,8,[5], dtype=torch.int32)
arg_2 = arg_2_tensor.clone()
arg_3_tensor = torch.rand([5, 1], dtype=torch.float32)
arg_3 = arg_3_tensor.clone()
arg_4 = False
try:
  res = torch.linalg.ldl_solve(arg_1,arg_2,arg_3,hermitian=arg_4,)
except Exception as e:
  print("Error:"+str(e))

```

Log message:

```
double free or corruption (out)
Aborted

```

### Versions

```
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.15 (main, Nov 24 2022, 14:31:59)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1660 Ti
Nvidia driver version: 525.78.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==1.13.1
[pip3] torchaudio==0.13.1
[pip3] torchvision==0.14.1
[conda] blas                      1.0                         mkl  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.23.5           py39h14f4228_0  
[conda] numpy-base                1.23.5           py39h31eccc5_0  
[conda] pytorch                   1.13.1          py3.9_cuda11.6_cudnn8.3.2_0    pytorch
[conda] pytorch-cuda              11.6                 h867d48c_1    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.13.1               py39_cu116    pytorch
[conda] torchvision               0.14.1               py39_cu116    pytorch
```
segfault when running torch.igamma
Labels: triaged

### 🐛 Describe the bug


```
import torch
import numpy as np
arg_1_tensor = torch.randint(0,2,[], dtype=torch.bool)
arg_1 = arg_1_tensor.clone()
arg_2_tensor = torch.rand([3], dtype=torch.float32)
arg_2 = arg_2_tensor.clone()
arg_3_tensor = torch.neg(torch.rand([3, 3, 3], dtype=torch.float16))
arg_3 = arg_3_tensor.clone()
try:
  res = torch.igamma(arg_1,arg_2,out=arg_3,)
except Exception as e:
  print("Error:"+str(e))

```

### Versions

```
PyTorch version: 1.12.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.15 (main, Nov 24 2022, 14:31:59)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: 
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1660 Ti
Nvidia driver version: 525.78.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==1.12.0
[pip3] torchaudio==0.12.0
[pip3] torchvision==0.13.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.23.5           py39h14f4228_0  
[conda] numpy-base                1.23.5           py39h31eccc5_0  
[conda] pytorch                   1.12.0          py3.9_cuda10.2_cudnn7.6.5_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.12.0               py39_cu102    pytorch
[conda] torchvision               0.13.0               py39_cu102    pytorch
```
Ability to manually set the gradient in FSDP while inside `summon_full_params` and make it persistent
Labels: needs reproduction, oncall: distributed, triaged, module: fsdp

### 🚀 The feature, motivation and pitch

Hello, I'm working on training large models with Differential Privacy and I'm using PyTorch FSDP to avoid out of memory issues.

What I'm doing right now is having each gpu compute a sample gradient, clip it to a max norm and then summing it to a local variable. I would then like to reduce the local gradients across gpus by summing them together and then set this value to model's parameters `grad` attribute so that I can then call `optimizer.step()`.

My code basically looks like this:

        with FSDP.summon_full_params(model, with_grads=True):
            for i, param in enumerate(model.parameters()):
                if param.requires_grad:
                    all_params_grad = [torch.zeros_like(param.grad).to(rank) for _ in range(world_size)]
                    dist.all_gather(all_params_grad, params_grad[i].cuda())

                    # sum the gradients across the gpus
                    sum = torch.stack(all_params_grad).sum(0)

                    param.grad = sum.type(param.grad.type()).to(param.grad.device)

                    param.grad = privacy_engine.add_noise(param.grad) / batch_size

where `params_grad[i]` contains the local sum of the clipped sample gradients. The problem is that right after exiting the context, if I try to access the parameters grad values they will all be None and so taking a step won't update the model.

Is there a way to workaround this? Any help would really be appreciated!

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Segfault when running torch.atan2
Labels: triaged, module: complex

### 🐛 Describe the bug


Probably due to empty tensor:

```
import torch
import numpy as np
arg_1_tensor = torch.rand([], dtype=torch.float16)
arg_1 = arg_1_tensor.clone()
arg_2_tensor = torch.rand([3], dtype=torch.complex64)
arg_2 = arg_2_tensor.clone()
arg_3_tensor = torch.rand([3, 3, 3], dtype=torch.complex128)
arg_3 = arg_3_tensor.clone()
try:
  res = torch.atan2(arg_1,arg_2,out=arg_3,)
except Exception as e:
  print("Error:"+str(e))

```

Please note that this is a machine generated test cases (using fuzz testing). 

### Versions

```
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.15 (main, Nov 24 2022, 14:31:59)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: 
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1660 Ti
Nvidia driver version: 525.78.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==1.12.0
[pip3] torchaudio==0.12.0
[pip3] torchvision==0.13.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.23.5           py39h14f4228_0  
[conda] numpy-base                1.23.5           py39h31eccc5_0  
[conda] pytorch                   1.12.0          py3.9_cuda10.2_cudnn7.6.5_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.12.0               py39_cu102    pytorch
[conda] torchvision               0.13.0               py39_cu102    pytorch

```

cc @ezyang @anjali411 @dylanbespalko @mruberry @Lezcano @nikitaved
[complex] Jacobian of a non-holomorphic complex valued function
Labels: triaged, complex_autograd

### 🐛 Describe the bug

```python
import torch

def fn(x):
    return x.conj()

x = torch.randn(1, dtype=torch.cdouble)
jac = torch.autograd.functional.jacobian(fn, x)
print(jac) # tensor([[1.-0.j]], dtype=torch.complex128)
```

`conj` is not a holomorphic function and the Jacobian can't be represented with just a complex number.

cc: @lezcano 


### Versions

master
[BE] move _apply_to_tensors from FSDP to torch.distributed.utils, use in _recursive_to
Labels: oncall: distributed, triaged, better-engineering, module: ddp

### 🚀 The feature, motivation and pitch

_recursive_to function has some specific logic for iterating through container types, _apply_to_tensors in FSDP does this a bit more generally. We should move this to torch.distributed.utils and have _recursive_to use this function.

### Alternatives

_No response_

### Additional context

_No response_

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu
Segmentation fault when running torch.ge
Labels: needs reproduction, triaged

### 🐛 Describe the bug


Due to dimension mismatch:

```
import torch
import numpy as np
arg_1_tensor = torch.rand([3], dtype=torch.float32)
arg_1 = arg_1_tensor.clone()
arg_2_tensor = torch.randint(0,8,[], dtype=torch.uint8)
arg_2 = arg_2_tensor.clone()
arg_3_tensor = torch.neg(torch.rand([3, 3, 3], dtype=torch.complex64))
arg_3 = arg_3_tensor.clone()
try:
  res = torch.ge(arg_1,arg_2,out=arg_3,)
except Exception as e:
  print("Error:"+str(e))
```

### Versions

```
PyTorch version: 1.12.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.15 (main, Nov 24 2022, 14:31:59)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: 
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1660 Ti
Nvidia driver version: 525.78.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==1.12.0
[pip3] torchaudio==0.12.0
[pip3] torchvision==0.13.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.23.5           py39h14f4228_0  
[conda] numpy-base                1.23.5           py39h31eccc5_0  
[conda] pytorch                   1.12.0          py3.9_cuda10.2_cudnn7.6.5_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.12.0               py39_cu102    pytorch
[conda] torchvision               0.13.0               py39_cu102    pytorch

```