gopikrsmscs/torch-issues huggingface dataset:
[autograd.Function] freevar lifting is too aggressive?
Labels: triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

Seems like the freevar lifting things it needs to lift constants and then graph breaks if they are used in autograd.Function backwards:

```
import torch
from typing import *

def halve(x):
    return x * 0.5


class ScaleGradient(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        return x

    @staticmethod
    def backward(ctx, grad):
        return halve(x)

x = torch.randn(3, requires_grad=True)
def f(x):
    return ScaleGradient.apply(x)
output = torch.compile(f, backend='eager', fullgraph=True)(x)
```
gives:
```
    f.call_function(tx, sub_args, {})
  File "/raid/rzou/pt/debug-cpu2/torch/_dynamo/variables/higher_order_ops.py", line 950, in call_funct
ion
    unimplemented("NYI - freevars in autograd function.")
  File "/raid/rzou/pt/debug-cpu2/torch/_dynamo/exc.py", line 143, in unimplemented
    raise Unsupported(msg)
torch._dynamo.exc.Unsupported: NYI - freevars in autograd function.

from user code:
   File "/raid/rzou/pt/debug-cpu2/foo.py", line 20, in f
    return ScaleGradient.apply(x)

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information


You can suppress this exception and fall back to eager by setting:
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True
```

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov 

### Error logs

_No response_

### Minified repro

_No response_

### Versions

main
[autograd.Function] torch.compile w/ once_differentiable leads to opaque graph break
Labels: high priority, triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

Some internal models use once_differentiable. The error message is very opaque:
```py
import torch
from typing import *
from torch.autograd.function import once_differentiable


class ScaleGradient(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        return x

    @staticmethod
    @once_differentiable
    def backward(ctx, grad):
        return grad * 0.5

x = torch.randn(3, requires_grad=True)
def f(x):
    return ScaleGradient.apply(x)
output = torch.compile(f, backend='eager', fullgraph=True)(x)
```
returns
```
Unsupported: inline in skipfiles: ScaleGradient.backward  | wrapper /raid/rzou/pt/debug-cpu2/torch/aut
ograd/function.py

from user code:
   File "<ipython-input-20-149cb1010897>", line 18, in f
    return ScaleGradient.apply(x)
  File "/raid/rzou/pt/debug-cpu2/torch/_dynamo/variables/misc.py", line 241, in trampoline_autograd_bw
d
    return fn_cls.backward(*args, **kwargs)

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information


You can suppress this exception and fall back to eager by setting:
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True

```

cc @ezyang @gchanan @kadeng @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov 

### Error logs

_No response_

### Minified repro

_No response_

### Versions

main
[Dynamo x FSDP][3/x] TypedStorage and storage_offset
Labels: Stale, module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #106738
* #106906
* #106890
* __->__ #106888
* #106886
* #106884



cc @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @anijain2305 @ipiszy
Dynamo graph break when using pyton module `heapq` (manipulates with `list`s)
Labels: low priority, triaged, module: dynamo

### 🐛 Describe the bug

OP: https://discuss.pytorch.org/t/is-heapq-module-supported-for-compilation-by-dynamo/185863:

`heapq` is a standard python's module useful for priority queue loops for Dijkstra-like algos

@msaroufim: @voznesenskym  on github might say this is a good dynamo starter task XD

```python
import heapq
import torch

@torch.compile(fullgraph=True)
def program():
  h = []
  heapq.heappush(h, 3)
  heapq.heappush(h, 1)
  heapq.heappush(h, 4)
  val = heapq.heappop(h)
  return val * torch.randn(10)

program()
```

### Versions

N/A

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @anijain2305 @ipiszy
[Dynamo] Integration exporter's diagnostic system into ONNXRuntime backend
Labels: module: onnx, triaged

### 🚀 The feature, motivation and pitch

As title. DORT doesn't have any diagnostic features, so reusing the system from ONNX exporter seems a good idea.

DORT PR: https://github.com/pytorch/pytorch/pull/106589

### Alternatives

_No response_

### Additional context

_No response_
[Dynamo] revise ONNXRuntime backend's use of CapabilityBasedPartitioner
Labels: module: onnx, triaged

### 🚀 The feature, motivation and pitch

I believe the following code can be improved. Using string matching to identify fused sub-graphs looks unsafe.
```
            # Overriding fused_module's __call__() function with ort_acclerated_call()
            # This loop goes through all graph partitions (each of them is an ONNX-representable graph)
            # and override their _wrappped_call function with _ort_accelerated_call.
            # Inside _ort_accelerated_call, the partition's graph is exported into ONNX and executed by ORT.
            for node in partitioned_prim_graph_module.graph.nodes:
                # TODO(wschin): use a better way to identify fused submodule
                if node.op == "call_module" and "fused_" in node.name:
                    fused_module = getattr(partitioned_prim_graph_module, node.name)
                    # self.ort_acclerated_call is responsible for exporting graph to ONNX,
                    # creating ORT session, and running ORT session.
                    fused_module._wrapped_call = self._ort_acclerated_call
```

DORT PR: https://github.com/pytorch/pytorch/pull/106589

### Alternatives

_No response_

### Additional context

_No response_
[Dynam] a graph pass in Dynamo-ONNXRuntime backend needs revision
Labels: module: onnx, triaged

### 🚀 The feature, motivation and pitch

A pass `_replace_to_copy_with_to` was added in the early days of dynamo development. As dynamo becomes more matured, we should try to remove this. If this is really needed, this should be implemented as a pass in ONNX exporter.

```
            # TODO(wschin): this is required for removing aten::_to_copy in _replace_to_copy_with_to.
            _replace_to_copy_with_to(prim_graph_module)
```

DORT PR: https://github.com/pytorch/pytorch/pull/106589

### Alternatives

_No response_

### Additional context

_No response_
[Dyanmo] Pre-allocate flag should be a ONNXRuntime inference session level attribute
Labels: module: onnx, triaged

### 🚀 The feature, motivation and pitch

Now DORT stores pre-allocated flag in `OrtBackend.preallocate_output` which looks a bit more global than it can be. Because pre-allocated memory is not needed by all execution providers (e.g., CUDA execution provider doesn't need it). 

DORT PR: https://github.com/pytorch/pytorch/pull/106589

### Alternatives

_No response_

### Additional context

_No response_
[Dynamo] ONNXRuntime backend (DORT) requires some guards to re-partition extracted by Dynamo
Labels: module: onnx, triaged

### 🚀 The feature, motivation and pitch

Given a GraphModule, DORT partitions it into sub-GraphModules using
```
            partitioner = CapabilityBasedPartitioner(
                prim_graph_module,
                self._supported_ops,
                allows_single_node_partition=True,
            )
            partitioned_prim_graph_module = partitioner.partition_and_fuse()
            self._partitioner_cache[graph_module] = partitioned_prim_graph_module
```

Each time the same GraphModule `g` is encountered, we use the cached partition result in `self._partitioner_cache[g] `. However, this assumes that the partition result is independent to graph inputs, which could be wrong since some computation is shape/type-dependent. As a solution, we should guard at least input ranks. If input ranks changes, we re-partition again.

DORT PR: https://github.com/pytorch/pytorch/pull/106589

### Alternatives

_No response_

### Additional context

_No response_
[Dynamo] ONNXRuntime Backend Shold Allow External Allocator
Labels: module: onnx, triaged

### 🚀 The feature, motivation and pitch

DORT (Dynamo-ONNXRuntime) creates inference session in DORT using the following code.

```
def _create_onnx_session(onnx_proto, eps: Tuple[str, ...], session_options):
    # TODO(wschin): enable external allocators.
    return onnxruntime.InferenceSession(
        onnx_proto, providers=eps, sess_options=session_options
    )
```

Since no external allocator is provided, ORT and PyTorch manage memory using different allocators. This can result higher memory peak, so we should pass PyTorch's allocator to ORT.

DORT PR: https://github.com/pytorch/pytorch/pull/106589

### Alternatives

_No response_

### Additional context

_No response_
untimeError: The following operation failed in the TorchScript interpreter. Traceback of TorchScript (most recent call last): RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)
Labels: needs reproduction, oncall: jit

### 🐛 Describe the bug

  pos[:2] += torch_rand_float(-1., 1., (2, 1),
                              device=self.device).squeeze(1)


untimeError: The following operation failed in the TorchScript interpreter. Traceback of TorchScript (most recent call last): RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)

### Versions

ubuntu:20.04:
NVIDIA GeForce RTX 4060 Ti
cuda :11.3
- torch==1.10.0+cu113 
- torchvision==0.11.1+cu113
- torchaudio==0.10.0+cu113

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel
Broadcasting semantics notes
Labels: Stale

When a tensor has a dimension size 0, the resulting size is 0.

Add an example for that.

This is consistent with numpy:
```
>>> np.add(np.zeros([1, 0]), torch.ones([1, 1, 1]))
tensor([], size=(1, 1, 0), dtype=torch.float64)
```

Fixes #ISSUE_NUMBER
Please verify 1.14.1 ONNX release candidate on TestPyPI
Labels: module: onnx, triaged, module: infra

### 🚀 The feature, motivation and pitch

Hi ONNX partner,
We have released TestPyPI packages of ONNX 1.14.1: https://test.pypi.org/project/onnx/1.14.1rc1/ (ONNX 1.14.1rc1 is the latest version number for testing now).
Please verify it and let us know about any problems. Thank you for your help!

### Alternatives

_No response_

### Additional context

_No response_
[MPS] Make several ops unranked to avoid graph recompilation
Labels: triaged, open source, module: mps, release notes: mps, ciflow/mps

- Made `bmm`, `cat`, `softmax`, `copy_cast` and `index_select` unranked. This improves the performance of LLMs and other networks, because these graphs would be invariant to shape changes, and therefore, they won't recompile. 
- Binary ops: Flatten tensors bigger than 4D to 1D, and reuse existing buffer for in-place operations (for offset 0).
- Cache the `MPSGraphExecutable` in `MPSCachedGraph`. These executables have the typeInference disabled which should help with recompilation issues on MPSGraph.
- Handle transposes in second batch of matrices in `bmm`. This helps with performance in case the second batch (i.e., `other` argument) was transposed (like the case with `QxK.t()` in Transformers).


cc @kulinseth @albanD @malfet @DenisVieriu97 @abhudev
Optimizers should use learning rates passed as tensors directly
Labels: module: optimizer, triaged, actionable, module: dynamic shapes

### 🚀 The feature, motivation and pitch

If you pass a tensor as the learning rate when constructing an optimizer, it works, but internally it does an item() on every step, which breaks graph capture and triggers synchronization warnings if you have them enabled.

It would be nice if the optimizers (maybe just the fused versions?) directly used the tensor, because then you could adjust learning rate every step while using CUDA graphs.  I wind up forcing graph rebuilds every 1000 steps or so when I want to dynamically change learning rates.

### Alternatives

_No response_

### Additional context

_No response_

cc @vincentqb @jbschlosser @albanD @janeyx99 @crcrpar @ezyang
DISABLED test_conversions_all_patterns_backend_cutlass_cuda_float16 (__main__.TestSparseSemiStructuredCUDA)
Labels: module: sparse, triaged, module: flaky-tests, skipped

Platforms: linux, slow

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_conversions_all_patterns_backend_cutlass_cuda_float16&suite=TestSparseSemiStructuredCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/15706109496).

Over the past 3 hours, it has been determined flaky in 3 workflow(s) with 3 failures and 3 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_conversions_all_patterns_backend_cutlass_cuda_float16`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_sparse_semi_structured.py`

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer
DISABLED test_linear_inference_mode_False_backend_cutlass_cuda (__main__.TestSparseSemiStructuredCUDA)
Labels: module: sparse, triaged, module: flaky-tests, skipped

Platforms: linux, slow

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_linear_inference_mode_False_backend_cutlass_cuda&suite=TestSparseSemiStructuredCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/15706109496).

Over the past 3 hours, it has been determined flaky in 3 workflow(s) with 3 failures and 3 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_linear_inference_mode_False_backend_cutlass_cuda`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_sparse_semi_structured.py`

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer
DISABLED test_conversions_all_patterns_backend_cutlass_cuda_bfloat16 (__main__.TestSparseSemiStructuredCUDA)
Labels: module: sparse, triaged, module: flaky-tests, skipped

Platforms: linux, slow

This test was disabled because it is failing in CI. See [recent examples](https://hud.pytorch.org/flakytest?name=test_conversions_all_patterns_backend_cutlass_cuda_bfloat16&suite=TestSparseSemiStructuredCUDA) and the most recent trunk [workflow logs](https://github.com/pytorch/pytorch/runs/15706109496).

Over the past 3 hours, it has been determined flaky in 3 workflow(s) with 3 failures and 3 successes.

**Debugging instructions (after clicking on the recent samples link):**
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:
1. Click on the workflow logs linked above
2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
3. Grep for `test_conversions_all_patterns_backend_cutlass_cuda_bfloat16`
4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.


Test file path: `test_sparse_semi_structured.py`

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer
Fix prod double backward when there are 2+ zeros
Labels: triaged

### 🐛 Describe the bug

When there are two or more zeros, prod's gradient does not require grad. This is equivalent to saying that the double backwards is 0, i.e. there are no gradients to be computed. However, this is wrong; the double backwards should not be zero.
```
import torch

x = torch.tensor([2., 3, 0, 0], requires_grad=True)
y = torch.prod(x)
gx, = torch.autograd.grad(y.sum(), x, create_graph=True)
```

### Versions

main
Binary op support for (B, *, D) NT with (B, 1, 1) dense

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #106785
* #106786

Provides a CUDA kernel based on the existing ESUHM kernel for binary op computation between `(B, *, D)` NT and `(B, 1, 1)` dense, broadcasting over both the * and D dims.

This is needed for [image preprocessing in SAM](https://github.com/facebookresearch/segment-anything/blob/6fdee8f2727f4506cfbbe553e23b895e27956588/segment_anything/modeling/sam.py#L167).
[ux] Suppot torch.tensor(set([1,2,3]))
Labels: triaged, needs research, topic: new features, module: python frontend

### 🚀 The feature, motivation and pitch

If it's easy to do, it would save a few keystrokes ( + maybe same for `dict_keys` and `dict_values`) and maybe some copies and python objects on the heap for large inputs:
```python
import torch

torch.tensor(list(set([1,2,3])))
# tensor([1, 2, 3])

torch.tensor(set([1,2,3]))
# RuntimeError: Could not infer dtype of set

# torch.tensor(set([1,2,3]), dtype = torch.int64)
# TypeError: an integer is required (got type set)

# torch.tensor({1 : 'a', 2 : 'b', 3 : 'c'}.keys())
# RuntimeError: Could not infer dtype of dict_keys


# torch.tensor({1 : 'a', 2 : 'b', 3 : 'c'}.keys(), dtype = torch.int64)
# TypeError: an integer is required (got type dict_keys)
```

### Alternatives

_No response_

### Additional context

_No response_

cc @albanD
`torch::nn::MultiheadAttention`, `F::multi_head_attention_forward`: check embed_dim and num_heads
Labels: module: cpp, module: error checking, triaged, open source, ciflow/trunk, release notes: cpp, module: multi-headed-attention

Fixes #106700

Referred [check of python API](https://github.com/pytorch/pytorch/blob/main/torch/nn/modules/activation.py#L967-L971).

cc @jbschlosser @malfet
Enable Mypy Checking in torch/_inductor/codegen/triton.py
Labels: triaged, open source, module: inductor, ciflow/inductor

Fixes #105230

Summary:

As suggested in https://github.com/pytorch/pytorch/issues/105230 mypy checking is enabled in torch/_inductor/codegen/triton.py

After Fix:

mypy --follow-imports=skip torch/_inductor/codegen/triton.py Success: no issues found in 1 source file


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[Inductor][cpu] torchbench model doctr_det_predictor perf regression
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

**doctr_det_predictor** Perf regression tracked on https://github.com/pytorch/pytorch/issues/93531#issuecomment-1668825530

|   |  2023-08-06 nightly  |   |     |   |  2023-08-02 nightly  |   |     |   |  Result Comp  |     |   |
|  ---- |  ----  | ----  |  ----  | ----  |  ----  | ----  |  ----  | ----  | ----  |  ----  | ----  |
|  model | batch_size  | speedup | inductor  | eager | batch_size  | speedup | inductor  | eager | speedup ratio | eager ratio  | inductor ratio |
|doctr_det_predictor	|1	|0.649268	 |3.384136689	|2.19721166	|1	|1.182882	 |1.861407945	|2.201825953	|0.55	|1	|0.55

2023-08-06 nightly
SW	| Nightly commit	| Main commit
-- | -- | --
Pytorch|[a8638d6](https://github.com/pytorch/pytorch/commit/a8638d6)|[](https://github.com/pytorch/pytorch/commit/)
Torchbench|/|[770d5cf7](https://github.com/pytorch/benchmark/commit/770d5cf7)
torchaudio|[dc83b38](https://github.com/pytorch/audio/commit/dc83b38)|[](https://github.com/pytorch/audio/commit/)
torchtext|[c11d758](https://github.com/pytorch/text/commit/c11d758)| [](https://github.com/pytorch/text/commit/)
torchvision|[58366ab](https://github.com/pytorch/vision/commit/58366ab)|[](https://github.com/pytorch/vision/commit/)
torchdata|[1d231d1](https://github.com/pytorch/data/commit/1d231d1)|[](https://github.com/pytorch/data/commit/)
dynamo_benchmarks|[f228c8b](https://github.com/pytorch/pytorch/commit/f228c8b)|/

2023-08-02 nightly
SW	| Nightly commit	| Main commit
-- | -- | --
Pytorch|[c89b169](https://github.com/pytorch/pytorch/commit/c89b169)|[92cac6b](https://github.com/pytorch/pytorch/commit/92cac6b)
Torchbench|/|[770d5cf7](https://github.com/pytorch/benchmark/commit/770d5cf7)
torchaudio|[dc83b38](https://github.com/pytorch/audio/commit/dc83b38)|[66f661d](https://github.com/pytorch/audio/commit/66f661d)
torchtext|[c11d758](https://github.com/pytorch/text/commit/c11d758)| [60bea66](https://github.com/pytorch/text/commit/60bea66)
torchvision|[58366ab](https://github.com/pytorch/vision/commit/58366ab)|[a6dea86](https://github.com/pytorch/vision/commit/a6dea86)
torchdata|[1d231d1](https://github.com/pytorch/data/commit/1d231d1)|[757c032](https://github.com/pytorch/data/commit/757c032)
dynamo_benchmarks|[f228c8b](https://github.com/pytorch/pytorch/commit/f228c8b)|/


### Versions



bash [inductor_single_test.sh](https://github.com/chuanqi129/inductor-tools/blob/yudong/aws_auto/scripts/modelbench/inductor_single_run.sh) multiple inference performance torchbench doctr_det_predictor float32 first static default 0

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305
[Feature request] Add new API Tensor.device_as 
Labels: triage review, oncall: jit, feature, module: python frontend

### 🚀 The feature, motivation and pitch

This new Api can be like Tensor.type_as.
```python
a1 = torch.zeros(5, device='cuda')
a2 = torch.zeros(5, device='cpu')
a3 = a1.device_as(a2)
assert a3.device == a1.device
```
Mainly used in the jit.trace model.
Because Tensor.to only record constant parameter. 
This can cause exceptions when moving the model to other devices.

I am currently using this method to transfer tensor from A device to B device in the trace model.
```python
@torch.jit.script
def device_as(self, other):
    return self.to(device=other.device)
```

### Alternatives

_No response_

### Additional context

_No response_

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel @albanD
pytorch with ROCM on Windows
Labels: module: windows, module: rocm, triaged

### 🚀 The feature, motivation and pitch

pytorch with ROCM on Windows

Since the ROCM has supported Windows, when will the Pytorch be availabled on Windows?

### Alternatives

_No response_

### Additional context

_No response_

cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @vladimir-aubrecht @iremyux @Blackhex @cristianPanaite @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
[FSDP][WIP] [Do not review] Trace FSDP
Labels: Stale, release notes: distributed (fsdp), module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #106738
* #106906
* #106890
* #106888
* #106886
* #106884



cc @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @anijain2305 @ipiszy
Add reset_parameters to nn.Module base class
Labels: Stale, release notes: nn

**We need a good solution to the subclass issue https://github.com/pytorch/pytorch/issues/71404#issuecomment-1026011644 before landing probably, I'm working on it!**

[Some wip thoughts on the above](https://colab.research.google.com/drive/1RFZjWpGfBVZka-NU6jmCZrftvq5HdT5u)

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* #106508
* #106506
* __->__ #106715
Dev Container Support for PyTorch
Labels: triaged, better-engineering

# Summary 
This servers as the top-level project tracker for making contributions to the PyTorch project easier and more device agnostic.

The primary tool for doing this is the devcontainer spec. This spec provides two nice entrypoints, [codespaces](https://github.com/features/codespaces) which is Githubs compute service for running devcontainers. Github users get access to a free tier spec. 

There is comprehensive VSCode support for devcontainers. [VSCode integration](https://code.visualstudio.com/docs/devcontainers/containers)

### Sub Tasks
- [x] Initial devcontainer support for local CPU development: https://github.com/pytorch/pytorch/issues/106714
- [x] Initial devcontainer support for remote codespaces development. See attached screenshot for 
- [x] Initial devcontainer support for local GPU development:  https://github.com/pytorch/pytorch/issues/106714
- [ ] Document process for building locally + codespaces
- [ ] Add to contributing guide
- [ ] Add support for MPS based dev: #106717 
- [ ] Add support for AMD based dev: #106718
 <img width="1064" alt="Screenshot 2023-08-07 at 11 42 24 AM" src="https://github.com/pytorch/pytorch/assets/32754868/371270c9-2b88-4879-889f-cb9ed5689ec4">
Enable Mypy Checking in torch/_inductor/fx_passes/pad_mm.py
Labels: triaged, open source, module: inductor, ciflow/inductor

Fixes #105230

Summary:

As suggested in https://github.com/pytorch/pytorch/issues/105230 mypy checking is enabled in torch/_inductor/fx_passes/pad_mm.py

After Fix:

mypy --follow-imports=skip torch/_inductor/fx_passes/pad_mm.py Success: no issues found in 1 source file


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
torch.polygamma inconsistent with scipy.special.polygamma for n >= 1
Labels: triaged, module: numpy, module: special

kshitij12345 raised an issue two years ago concerning the fact that pytorch's implementation of the 1st order polygamma function incorrectly produces finite values for negative integers. The issue was closed, but the issue still seems to be present in the latest nightlies and the latest release of pytorch.

> n = 1
> 
> ```python
> >>> t = torch.tensor([-1.])
> >>> torch.polygamma(1, t)
> tensor([1.2914e+15])
> >>> scipy.special.polygamma(1, t.numpy())
> array([inf], dtype=float32)
> 
> >>> t = torch.tensor([-501.])
> >>> torch.polygamma(1, t)
> tensor([2.0831e+09])
> >>> scipy.special.polygamma(1, t.numpy())
> array([inf], dtype=float32)
> 
> >>> t = torch.tensor([-float('inf')])
> >>> torch.polygamma(1, t)
> tensor([nan])
> >>> scipy.special.polygamma(1, t.numpy())
> array([inf], dtype=float32)
> >>> 
> ```
> 
> n > 1
> 
> ```python
> >>> t = torch.tensor([float('inf')])
> >>> torch.polygamma(2, t)
> tensor([nan])
> >>> scipy.special.polygamma(2, t.numpy())
> array([-0.], dtype=float32)
> >>> torch.polygamma(3, t)
> tensor([nan])
> >>> scipy.special.polygamma(3, t.numpy())
> array([0.], dtype=float32)
> >>> torch.polygamma(4, t)
> tensor([nan])
> >>> scipy.special.polygamma(4, t.numpy())
> array([-0.], dtype=float32)
> ```
> 
> cc @mruberry @rgommers @kshitij12345


_Originally posted in https://github.com/pytorch/pytorch/issues/55357
Enable Mypy Checking in torch/_inductor/fx_passes/joint_graph.py
Labels: triaged, open source, module: inductor, ciflow/inductor

Fixes #105230

Summary:

As suggested in https://github.com/pytorch/pytorch/issues/105230 mypy checking is enabled in torch/_inductor/fx_passes/joint_graph.py

After Fix:

mypy --follow-imports=skip torch/_inductor/fx_passes/joint_graph.py Success: no issues found in 1 source file


cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov
[docs] URL and link format proposal to make function page URLs more concise
Labels: module: docs, triaged, enhancement

### 🐛 Describe the bug

These links end up copy-pasted in messengers for chat with colleagues, so making them have less bloat would be nice.

If the links are concise, one can also do quick lookups without going to relatively slower loading https://pytorch.org/docs - e.g. by typing https://pytorch.org/docs/torch.div or https://pytorch.org/docs/div https://docs.pytorch.org/torch.div or https://docs.pytorch.org/div. These pages can also be auto-generated or symlinked for all function names at least for the torch/tensor functions (and such pages can list all functions with the same name e.g. torch.sigmoid, F.sigmoid which may be good by itself https://github.com/pytorch/pytorch/issues/105318 or https://discuss.pytorch.org/t/various-quantized-quantizable-intrinsic-modules-purpose/183562 for LinearReLU)
<hr>
If we go to https://pytorch.org/docs/stable/torch.html, we have links like https://pytorch.org/docs/stable/generated/torch.div.html#torch.div

I'm proposing that instead these function links should be more concise and not include the hash. Going further, there should be symlinks https://pytorch.org/docs/torch.div.html or https://pytorch.org/docs/torch.div or even https://docs.pytorch.org/torch.div (and https://docs.pytorch.org/div)

When doing simple search like `torch.div` we get:
<img width="517" alt="image" src="https://github.com/pytorch/pytorch/assets/1041752/1612511a-0941-43c7-8514-18d763b2946b">
Clicking the first hit will get us to https://pytorch.org/docs/stable/generated/torch.div.html?highlight=torch+div#torch.div

Also, the highlight seems to have no effect now (as opposed to the Google's https://pytorch.org/docs/stable/generated/torch.div.html#:~:text=torch.div https://stackoverflow.com/questions/62161819/what-exactly-is-the-text-location-hash-in-an-url). So maybe worth dropping the highligh part from the Sphinx search hit URLs too? 

### Versions

N/A

cc @svekars @carljparker
Will torch.sparse.mm support multiplying two boolean matrices?
Labels: module: sparse, triaged, module: boolean tensor

### 🚀 The feature, motivation and pitch

I have two huge matrices. They are sparse Boolean matrices. I call torch.sparse.mm and found that it cannot be successful. Will the multiplication of bool sparse matrices be supported in the future?

### Alternatives

_No response_

### Additional context

_No response_

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer
Question about garbage collection without GPU sync 
Labels: triaged, module: CUDACachingAllocator

### 🐛 Describe the bug

I found that in `void garbage_collect_cached_blocks()`

```
    size_t gc_threshold = static_cast<size_t>(
        CachingAllocatorConfig::garbage_collection_threshold() *
        allowed_memory_maximum);
```
If the `torch._C._cuda_setMemoryFraction` function is not called，`allowed_memory_maximum` value is 0. The value of `gc_threshold` is also 0. The garbage collection without GPU sync mechanism does not take effect. And there's no prompt.
So I think it's more reasonable whether to use `device_total` (obtained using `cudaMemGetInfo`) when `allowed_memory_maximum` is 0.



### Versions

https://github.com/pytorch/pytorch/blob/dc22b4fdb1b4af1f0b7b77a78b30237d337d7109/c10/cuda/CUDACachingAllocator.cpp#L2577C47-L2577C47
Add functional collective all_to_all_single and support it in Inductor
Labels: fb-exported, module: inductor, module: dynamo, ciflow/inductor, release notes: AO frontend

Summary: `all_to_all_single` is being used by all the important All2All workloads (e.g. alltoall_pooled / alltoall_sequence / alltoallv). This PR adds functional collective `all_to_all_single` and support it in Inductor, to unlock our exploration on internal workloads.

Differential Revision: D48013416



cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @chenyang78 @kadeng @muchulee8 @aakhundov @anijain2305
Dynamo graph break on triplet_margin_with_distance_loss
Labels: triaged, module: dynamo

### 🐛 Describe the bug

python test/inductor/test_torchinductor_opinfo.py -k test_comprehensive_nn_functional_triplet_margin_with_distance_loss_cuda_float32

>   File "/scratch/eellison/work/pytorch/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
    return inner_fn(self, inst)
  File "/scratch/eellison/work/pytorch/torch/_dynamo/symbolic_convert.py", line 1156, in CALL_FUNCTION_EX
    self.call_function(fn, argsvars.items, kwargsvars.items)
  File "/scratch/eellison/work/pytorch/torch/_dynamo/symbolic_convert.py", line 562, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/scratch/eellison/work/pytorch/torch/_dynamo/variables/torch.py", line 723, in call_function
    *proxy_args_kwargs(args, kwargs),
  File "/scratch/eellison/work/pytorch/torch/_dynamo/utils.py", line 504, in proxy_args_kwargs
    f"call_function args: {typestr(*args)} {typestr(*list(kwargs.values()))}"
  File "/scratch/eellison/work/pytorch/torch/_dynamo/exc.py", line 143, in unimplemented
    raise Unsupported(msg)
torch._dynamo.exc.Unsupported: call_function args: TensorVariable() TensorVariable() TensorVariable() ConstantVariable(float) NNModuleVariable()

related: https://github.com/pytorch/pytorch/issues/105534

### Versions

master

cc @voznesenskym @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @aakhundov @peterbell10 @ngimel @yf225 @kadeng @muchulee8
Hackable distributed filesystem reader and writer
Labels: triaged, open source

I propose some changes so that the `FileSystemReader` and `FileSystemWriter` can be used on other file systems. User only needs to provide `path` as a subclass of `Path` that overrides the necessary interfaces. 

For example, one can utilize `tf.io.gfile` to implement an interface to save to or load from HDFS. The following code snippet shows a working implementation. 

```python
from pathlib import Path
import tensorflow as tf


class GFileWrapper(tf.io.gfile.GFile):
    def __init__(self, path, mode="r") -> None:
        super().__init__(path, mode)
        
    def write(self, data):
        return super().write(bytes(data))
    
    # a not quite efficient readinto, but it works
    def readinto(self, buffer):
        # read up to buffer's length
        data = self.read(len(buffer))
        length = len(data)
        buffer[:length] = data
        return length


class HdfsPath(type(Path())):
    def __new__(cls, *pathsegments):
        return super().__new__(cls, *pathsegments)
    
    @staticmethod
    def _fix_path(path):
        path = str(path)
        if path.startswith("hdfs:/") and not path.startswith("hdfs://"):
          path = path.replace("hdfs:/", "hdfs://")
        return path

    def open(self, mode="r", *args, **kwargs):
        return GFileWrapper(HdfsPath._fix_path(self), mode=mode)
    
    def mkdir(self, **kwargs) -> None:
        return tf.io.gfile.makedirs(HdfsPath._fix_path(self))
    
    def rename(self, target):
        return tf.io.gfile.rename(HdfsPath._fix_path(self), HdfsPath._fix_path(target))
```

```python
writer = FileSystemWriter(HdfsPath("hdfs://..."), sync_files=False)
reader = FileSystemReader(HdfsPath("hdfs://..."))
```
Confusing error message for DataLoader with num_workers=0 and non-zero timeout
Labels: module: dataloader, triaged

### 🐛 Describe the bug

It looks like setting a timeout on the DataLoader is illegal with num_workers=0, but the documentation doesn't mention it and the error message is very confusing. I would expect the timeout to be ignored under those circumstances, but at the very least this deserves a clear error message, since the current one reads as if a timeout occurred, rather than an illegal value for it was passed.

Minimal reproducing example
```python
from torch.utils.data import DataLoader
train_dataloader_full_tracks = DataLoader(
    [], num_workers=0, timeout=1
)
iter(train_dataloader_full_tracks)
```

Output
```
Traceback (most recent call last):
  File "dataloader_problem.py", line 6, in <module>
    iter(train_dataloader_full_tracks)
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 442, in __iter__
    return self._get_iterator()
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 385, in _get_iterator
    return _SingleProcessDataLoaderIter(self)
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 664, in __init__
    assert self._timeout == 0
AssertionError
```

### Versions

I use 2.0.0, but I see that the main branch has the same issue as of today.

cc @SsnL @VitalyFedyunin @ejguan @dzhulgakov
Dynamo debug decorator
Labels: Stale, module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #106625

Dynamo debug decorator

```
@torch._dynamo.debug
def inner_foo(x):
    return x * x

def fn(x, y):
    x2 = inner_foo(x)
    return x2 * y


x = torch.rand([4, 10])
y = torch.rand([4, 10])

torch._dynamo.optimize("eager")(fn)(x, y)
```
Output:
```
[2023-08-04 17:48:38,550] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object inner_foo at 0x7fd962616e40, file "/scratch/voz/work/pytorch/x.py", line 7>
[2023-08-04 17:48:38,550] torch._dynamo.symbolic_convert: [DEBUG]   9           0 LOAD_FAST                0 (x)
[2023-08-04 17:48:38,550] torch._dynamo.symbolic_convert: [DEBUG]               2 LOAD_FAST                0 (x)
[2023-08-04 17:48:38,550] torch._dynamo.symbolic_convert: [DEBUG]               4 BINARY_MULTIPLY
[2023-08-04 17:48:38,550] torch._dynamo.symbolic_convert: [DEBUG]               6 RETURN_VALUE
[2023-08-04 17:48:38,550] torch._dynamo.symbolic_convert: [DEBUG] 
[2023-08-04 17:48:38,550] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line inner_foo /scratch/voz/work/pytorch/x.py:7 (inline depth: 1)
[2023-08-04 17:48:38,550] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]     @torch._dynamo.debug
[2023-08-04 17:48:38,550] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line inner_foo /scratch/voz/work/pytorch/x.py:9 (inline depth: 1)
[2023-08-04 17:48:38,550] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]         return x * x
[2023-08-04 17:48:38,550] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2023-08-04 17:48:38,550] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TensorVariable()]
[2023-08-04 17:48:38,550] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), TensorVariable()]
[2023-08-04 17:48:38,555] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
[2023-08-04 17:48:38,555] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object inner_foo at 0x7fd962616e40, file "/scratch/voz/work/pytorch/x.py", line 7>
```
And drops a breakpoint in.

cc @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @anijain2305 @ipiszy
[feature request] [ux] Frontend methods for fused  elementwise affine transform: mul+add+dtype convert + support  `integer_tensor.mul_(float_constant)` and `float_tensor.mul(some_constant, out = integer_tensor)` maybe via new args `rounding_mode=...` and `dtype=...` + maybe support OpenCV-style saturated dtype conversions (e.g. `clamp_` before conversion)
Labels: triaged, module: type promotion, module: python frontend

### 🚀 The feature, motivation and pitch

This should be well-compiled as it's elementwise, but would be nice to have as a frontend method for colloquial use.

A prototypical example is conversion of float32 image to uint8 image:

```python
import torch
a = torch.rand(10)
b = torch.empty_like(a, dtype = torch.uint8)
f32_to_u8 = torch.mul(a, 255, out = b)
# RuntimeError: result type Float can't be cast to the desired output type Byte
```

Might be that these methods already exist in the form of helpers for quantization, but would be nice to expose them for wider usage.

OpenCV https://docs.opencv.org/4.x/d3/d63/classcv_1_1Mat.html#adf88c60c5b4980e05bb556080916978b has a similar method `convertTo` which multiplies by a scalar, than adds another scalar and then casts dtype. OpenCV also does saturate cast functionality (to make sure that uint8 doesn't overflow too badly) - might be useful for PyTorch too, but saturation might be done prior to actual conversion by `clamp_` or by providing also a fused `mul_add_clamp(..., dtype=...)` (`torch.add(a, b, alpha = ...)` exists but also doesn't support the needed casting).

Related: `addcmul` generalization discussion: https://github.com/pytorch/pytorch/issues/104849

(although I realize, this might be somwhat niche - also sth similar exists in torchvision: https://pytorch.org/vision/stable/generated/torchvision.transforms.Normalize.html)

Being able to do `int32_tensor.mul_(float_constant)` or `uint8_tensor.mul_(float_constant)` is also useful and related to https://github.com/pytorch/pytorch/issues/54389. It could certainly support rounding, similar to `torch.div(..., rounding_mode = 'trunc')` e.g. by doing int32_tensor.mul_(numerator).div_(denominator, rounding_mode = 'floor')

Currently:
```python
x = torch.tensor([2, 4, 8], dtype = torch.int32)

although docs say "None - default behavior. Performs no rounding and, if both input and other are integer types, promotes the inputs to the default scalar type."
x.div_(2) # RuntimeError: result type Float can't be cast to the desired output type Long

x.div_(2, rounding_mode = 'trunc') # works
x //= 2 # works
```

cc @nairbv @mruberry @albanD
Meta implementations of FFT operators often have incorrect strides
Labels: triaged, module: fft, oncall: pt2

## Issue description

`aten._fft_c2r` and `aten._fft_r2c` meta implementations currently return contiguous strides in all cases, which is not consistent with eager.

`aten._fft_c2c` makes a much better effort, but is still incorrect on CUDA when there are more than 3 transform dimensions or on CPU when using the pocketfft implementation instead of mkl-fft.

Ref:
- [c2r](https://github.com/pytorch/pytorch/blob/df8abaaf5f7e19d897aa30896bdbbe416ca2f825/torch/_meta_registrations.py#L279)
- [r2c](https://github.com/pytorch/pytorch/blob/df8abaaf5f7e19d897aa30896bdbbe416ca2f825/torch/_meta_registrations.py#L221)
- [c2c](https://github.com/pytorch/pytorch/blob/df8abaaf5f7e19d897aa30896bdbbe416ca2f825/torch/_meta_registrations.py#L205)

## Code example

This real-to-complex example falls back to eager with `torch.compile` due to `UnsupportedOperatorException`:
```python
import torch

def fn(x):
    return torch.fft.rfft(x)

torch.compile(fn, dynamic=True)
```

This complex-to-complex example compiles, but fails at runtime with stride mismatch:
```python
import torch

@torch.compile
def fn(x):
    return torch.fft.fftn(x, dim=(1, 2, 3, 4))

fn(torch.randn((5, 5, 5, 5, 5), dtype=torch.complex64, device="cuda"))
# AssertionError: expected size 5==5, stride 1==125 at dim=1
```

cc @mruberry @ezyang @msaroufim @wconstab @bdhirsh @anijain2305 @nkaretnikov
FFT Samples Inputs with More than Three Dimensions
Labels: module: tests, triaged, module: fft

### 🚀 The feature, motivation and pitch

This is worth capturing because cuFFT can only transform three dimensions at once, so, 4dim+ has divergent behavior. See https://github.com/pytorch/pytorch/pull/103616#discussion_r1237295970. 

### Alternatives

_No response_

### Additional context

_No response_

cc @mruberry @peterbell10
Stopgap patch for avoiding polluting cache
Labels: Stale, module: dynamo, ciflow/inductor

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom):
* __->__ #106613

This is a complete and utter hack.

See https://github.com/pytorch/pytorch/issues/106547 for details.

Signed-off-by: Edward Z. Yang <ezyang@meta.com>

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @anijain2305 @ipiszy
ROCm & Windows Support
Labels: module: rocm, triaged

### 🚀 The feature, motivation and pitch

AMD has release ROCm windows support, as [docs.amd.com](https://docs.amd.com) shows:

![8df2b59d834bd749e7b486cd02a6e4b](https://github.com/pytorch/pytorch/assets/69568351/373102ff-81b7-4029-a1be-889214296303)
![984e40b50fb006e4e94f3abfeba8fe7](https://github.com/pytorch/pytorch/assets/69568351/ded8a54a-db59-4c2f-b5cb-a86883cda63e)

Please add PyTorch support of Windows on AMD GPUs!

### Alternatives

_No response_

### Additional context

_No response_

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang
More Performant CachingHostAllocator for Pinned Memory Allocation
Labels: module: cuda, triaged

### 🚀 The feature, motivation and pitch

I intend to employ a memory allocator for pinned memory allocation and have come across the `CachingHostAllocator` in PyTorch. Regrettably, the practical memory consumption surpasses what is expected. The allocator follows a power-of-two allocation strategy without memory coalescing, resulting in substantial memory wastage. This inefficiency can lead to increased memory consumption and suboptimal utilization of resources.

The current implementation of the `CachingHostAllocator` in PyTorch for pinned memory allocation seems to exhibit suboptimal performance under certain conditions. As a crucial component in deep learning workloads, efficient memory allocation and management are essential to ensure optimal training and inference performance.



### Alternatives

I suggest the development of a more performant alternative to the existing `CachingHostAllocator` that addresses the performance concerns. This new allocator should focus on improving memory allocation speed, reducing memory fragmentation, and better leveraging modern hardware characteristics.



### Additional context

_No response_
```[tasklist]
### Tasks
```


cc @ptrblck
onednn ops supported in pytorch
Labels: triaged, module: intel

How I get to know what are the onednn ops supported in pytorch?
what aten functions are going to ideep and further to onednn?
IS there any documentation which I can refer to understand this.

cc @frank-wei @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10
[ONNX] Keep functional ops as functions in dynamo exported onnx
Labels: module: onnx, triaged

### 🚀 The feature, motivation and pitch

`nn.Module` may be represented as function in onnx
I was expecting the functional op to be expressed as such

Although most functional ops correspond to aten ops one-to-one, there are only a few parameter differences
But there are still some common functional ops that will be expressed as a combination of multiple aten ops, such as 
* F.normalize
* F.interpolate
* etc.

Representing it as a function can express the structure of the model more meaningfully, and is more friendly to the optimization of the backend

Otherwise downstream tools need to write rules and combine them back together, like https://github.com/Tencent/ncnn/blob/master/tools/pnnx/src/pass_level2/F_interpolate.cpp


### Alternatives

_No response_

### Additional context

_No response_