gopikrsmscs/torch-issues huggingface dataset:
Creating NumPy array with `dtype=object` of PyTorch tensors fails
Labels: feature, triaged, module: numpy

### 🐛 Describe the bug

PyTorch tensors can easily be stored in Python lists. However, lists are slow and cumbersome for some operations (e.g. more complex indexing operations).

NumPy arrays support storing any Python object by specifying `dtype=object` when creating the array.

However, when attempting to create a NumPy array with `dtype=object`, PyTorch tries to convert the tensors to NumPy arrays. This should not be done, as we're not interested in storing the tensors as arrays. We're only interested in storing references to these tensors.

To reproduce this behavior:
```python
>>> tensors = (torch.randn(2,2, device="cuda"), torch.randn(1,3, device="cpu"))
>>> list(tensors)
[tensor([[-0.0517,  0.4685],
        [ 0.5149,  0.0535]], device='cuda:0'), tensor([[ 0.2241,  2.3390, -0.6848]])]
>>> np.array(tensors, dtype=object)
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
```

The section of the code that is responsible for this behavior can be found in `_tensor.py:__array__()`.
```python
def __array__(self, dtype=None):
    if has_torch_function_unary(self):
        return handle_torch_function(Tensor.__array__, (self,), self, dtype=dtype)
    if dtype is None:
        return self.numpy()
    else:
        return self.numpy().astype(dtype, copy=False)
```

One way to fix this bug would be to add a check in the `__array__` method to handle the case when `dtype=object`. We must then create an object array and add the tensor to it.
```python
def __array__(self, dtype=None):
    if has_torch_function_unary(self):
        return handle_torch_function(Tensor.__array__, (self,), self, dtype=dtype)
    if dtype is None:
        return self.numpy()
    elif dtype == np.dtype(object):
        arr = np.empty(1, dtype=object)
        arr[0] = self
        return arr
    else:
        return self.numpy().astype(dtype, copy=False)
```

This way, we can create a NumPy array that stores references to CPU and CUDA Tensors, such as:
```python
>>> tensors = (torch.randn(2,2, device="cuda"), torch.randn(1,3,device="cpu"))
>>> np.array(tensors, dtype=object)
array([tensor([[ 1.9628, -0.2343],
               [ 0.1929, -1.1450]], device='cuda:0'),
       tensor([[ 0.7147, -1.3656,  0.8823]])], dtype=object)
```

The proposed solution is probably not the best. Maybe @rgommers has a better idea? I would be willing to develop a solution for this, if the maintainers are interested.

Interestingly, with the current code the conversion works if the tensor is on the CPU, but does not work if it is on the GPU, even though NumPy is only storing a reference to the tensors!

### Versions

Versions of relevant libraries:
[pip3] numpy==1.22.4
[pip3] pytorch-memlab==0.2.4
[pip3] raisim-gym-torch==0.0.0
[pip3] torch==1.11.0
[pip3] torch-tb-profiler==0.4.0
[pip3] torchvision==0.12.0
[conda] Could not collect


cc @mruberry @rgommers
Solve default argument induced include cycles by not using defaults / moving the defaults to inl
Labels: module: build, module: internals, triaged

### 🐛 Describe the bug

Here is an example of a particularly pernicious cause of import cycles in our C++ codebase:

```
struct Scalar {
    Scalar(int i) : i_(i) {};
    int i_;
};

struct Tensor {
    void add(const Tensor& other, const Scalar& s = 0);
};
```

To compile this, the FULL definition of Scalar must be available at the time Tensor is being defined, so that the default parameter can be specified. This means that Scalar must NOT include any headers that transitively depend on Tensor. In the past, when people have been monkeying around with Scalar (e.g., wanting it to depend on IValue), this easily causes include cycles, and for no good reason: Tensor doesn't *really* depend (as far as struct layout is concerned) on Scalar, and although intuitively there shouldn't be a dependency between Scalar and Tensor, there seemingly is one. You cannot even solve the problem by wrapping the Scalar in a c10::optional, as c10::optional is an incomplete type without Scalar being complete.

Fortunately, there is a resolution to this problem. In fact, there are two options:

First, we can simply not use defaulted arguments. Thus, instead of a single add definition, we instead have:

```
struct Tensor {
    void add(const Tensor& other);
    void add(const Tensor& other, const Scalar& s);
};
```

As most methods on Tensor are code-generated, this doesn't involve driving a big refactor; just modify the codegen to separately define each definition and generate the necessary boilerplate.

Second, we can define the default argument later. If we assume that add is going to be implemented inline in an inl header later, this is perfectly valid, and still defaults the argument:

```
struct Tensor {
    void add(const Tensor& other, const Scalar& s);
};

// Later, in Tensor_inl.h where we can assume we have Scalar struct defined now
void Tensor::add(const Tensor& other, const Scalar& s = 0) {
  ... definition ...
};
```

alas, you are obligated to also provide the definition at this point; if you are looking to define out-of-line, option one is probably easier.

I looked into this because I wanted to see if it would be feasible to implement Scalar using IValue. In the end I decided this refactor would take too much time to be worth it for just this. But maybe someone else will find it useful some time...

### Versions

master

cc @malfet @seemethere @bhosmer @smessmer @ljk53 @bdhirsh
`linalg.norm` cannot compute the grad in forward mode after script
Labels: oncall: jit

### 🐛 Describe the bug

`linalg.norm` cannot compute the grad in forward mode after script

```py
import torch

def m(input):
    return input.norm(p=1)

input = torch.tensor([[1., 2., 3., 4.]], dtype=torch.float32)

from torch.autograd.functional import jacobian

input1 = input.clone().requires_grad_()
jac1 = jacobian(m, input1, vectorize=True, strategy='forward-mode')
print(jac1)

jit_m = torch.jit.script(m)
input2 = input.clone().requires_grad_()
jac2 = jacobian(jit_m, input2, vectorize=True, strategy='forward-mode')
print(jac2)
```

```
tensor([[1., 1., 1., 1.]], grad_fn=<ReshapeAliasBackward0>)

RuntimeError: Trying to set a forward gradient that has a different size than that of the original Tensor, this is not supported. Tensor is of size [] while the given forward gradient is of size [1, 1].
```

### Versions

```
Collecting environment information...
PyTorch version: 1.13.0.dev20220923+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.2.0-19ubuntu1) 11.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.13 (main, Aug 25 2022, 23:26:10)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.19.5-051905-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060
Nvidia driver version: 515.65.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.3
[pip3] torch==1.13.0.dev20220923+cu116
[pip3] torchaudio==0.13.0.dev20220923+cu116
[pip3] torchvision==0.14.0.dev20220923+cu116
[conda] numpy                     1.23.3                   pypi_0    pypi
[conda] torch                     1.13.0.dev20220923+cu116          pypi_0    pypi
[conda] torchaudio                0.13.0.dev20220923+cu116          pypi_0    pypi
[conda] torchvision               0.14.0.dev20220923+cu116          pypi_0    pypi

```

cc @ezyang  @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
`as_tensor` will return a different dtype with script
Labels: oncall: jit

### 🐛 Describe the bug

`as_tensor` will return a different dtype with script

```py
import torch
def m(data):
    return torch.as_tensor(data)

input = torch.tensor(2.25, dtype=torch.bfloat16)

print(m(input).dtype)

jit_m = torch.jit.script(m)
print(jit_m(input).dtype)
```

```
torch.bfloat16
torch.float32
```

### Versions

PyTorch version: 1.13.0.dev20220923+cu116
FSDP's FlattenParamsWrapper breaks dynamo's faketensor wrapping
Labels: triaged, module: fsdp, bug, oncall: pt2

Repro script

```
import torch
import torchdynamo
from torch import nn
from torch.distributed.fsdp.flat_param import HandleConfig, HandleShardingStrategy
from torch.distributed.fsdp.flatten_params_wrapper import FlattenParamsWrapper
from torch.distributed.fsdp.fully_sharded_data_parallel import FullyShardedDataParallel as FSDP


class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net = nn.Sequential(
            nn.Linear(10, 10000),
            nn.ReLU(),
        )

    def forward(self, x):
        return self.net(x)

class Wrapper(nn.Module):
    def __init__(self, wrapped):
        super(Wrapper, self).__init__()
        self.mod = wrapped
    
    def forward(self, *args, **kwargs):
        return self.mod(*args, **kwargs)

def flatten():
    mod = ToyModel()
    # FlattenParamsWrapper is a helper class used inside FSDP.  It appears to not work with Dynamo
    mod = FlattenParamsWrapper(
        mod,
        params=list(mod.parameters()),
        device="cpu",
        config=HandleConfig(HandleShardingStrategy.FULL_SHARD, False, None, None),
    )
    mod = torchdynamo.optimize("aot_eager")(mod)
    inputs = torch.randn(20, 10)
    outputs = mod(inputs)
    print(outputs)



if __name__ == "__main__":
    flatten()
    """
    Currently getting this error:

    Exception: Invoking operators with non-Fake Tensor inputs in FakeTensorMode is not yet supported. Please convert all Tensors to FakeTensors first. Found in aten.t.default(*(tensor([[ 0.2899, -0.1847,  0.0950,  ...,  0.0304, -0.2226, -0.3076],
        [-0.1424,  0.2049, -0.2806,  ..., -0.0127, -0.0831,  0.0546],
        [ 0.1324, -0.0840, -0.0635,  ..., -0.0338, -0.1047,  0.2138],
        ...,
        [ 0.2627,  0.1499,  0.0301,  ..., -0.2043, -0.0677,  0.0513],
        [ 0.0060, -0.1727, -0.1449,  ..., -0.0631, -0.0106, -0.0005],
        [-0.0464, -0.1969,  0.1693,  ...,  0.1114,  0.0926, -0.2426]],
       grad_fn=<ViewBackward0>),), **{})

    
    """
```

cc @zhaojuanmao @mrshenli @rohan-varma @awgu @ezyang @soumith @msaroufim @ngimel @bdhirsh
`jit` could make some undifferentiable APIs differentiable
Labels: oncall: jit

### 🐛 Describe the bug

`jit` could make some undifferentiable APIs differentiable, like `numel, nelement`

```py
import torch
def m(input):
    return input.nelement()

device = 'cpu'

input = torch.tensor([1], dtype=torch.float32, device=device)

jit_m = torch.jit.trace(m, (input.clone(), ))

from torch.autograd.functional import jacobian

print(jacobian(jit_m, input.clone().requires_grad_(), vectorize=True, strategy='reverse-mode'))
print(jacobian(m, input.clone().requires_grad_(), vectorize=True, strategy='reverse-mode'))
```

```
tensor([0.])

TypeError: The outputs of the user-provided function given to jacobian must be either a Tensor or a tuple of Tensors but the given outputs of the user-provided function has type <class 'int'>.
```

### Versions

1.13.0
`detach_` behaves differently when computing the gradients in forward mode w/ `jit`
Labels: oncall: jit

### 🐛 Describe the bug

`detach_` behaves differently when computing the gradients in forward mode w/ `jit`

```py
import torch
def m(input):
    return input.detach_()

device = 'cuda'

input = torch.empty([1], dtype=torch.float32, device=device)

jit_m = torch.jit.trace(m, (input.clone(), ))

from torch.autograd.functional import jacobian

try:
    print(jacobian(m, input.clone().requires_grad_(), vectorize=True, strategy='forward-mode'))
except Exception as e:
    print(e)

print(jacobian(jit_m, input.clone().requires_grad_(), vectorize=True, strategy='forward-mode'))
```

```
Can't detach views in-place. Use detach() instead. If you are using DistributedDataParallel (DDP) for training, and gradient_as_bucket_view is set as True, gradients are views of DDP buckets, and hence detach_() cannot be called on these gradients. To fix this error, please refer to the Optimizer.zero_grad() function in torch/optim/optimizer.py as the solution.
tensor([[0.]], device='cuda:0')
```

Interestingly, the reverse mode can give a gradient without jit

```py
import torch
def m(input):
    return input.detach_()

device = 'cuda'

input = torch.empty([1], dtype=torch.float32, device=device)

jit_m = torch.jit.trace(m, (input.clone(), ))

from torch.autograd.functional import jacobian
print(jacobian(m, input.clone().requires_grad_(), vectorize=True, strategy='reverse-mode'))
# tensor([[0.]], device='cuda:0')
```

I am not sure which way is correct, but they do need to behave consistently.

### Versions

```
Collecting environment information...
PyTorch version: 1.13.0.dev20220923+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.2.0-19ubuntu1) 11.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.13 (main, Aug 25 2022, 23:26:10)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.19.5-051905-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060
Nvidia driver version: 515.65.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.3
[pip3] torch==1.13.0.dev20220923+cu116
[pip3] torchaudio==0.13.0.dev20220923+cu116
[pip3] torchvision==0.14.0.dev20220923+cu116
[conda] numpy                     1.23.3                   pypi_0    pypi
[conda] torch                     1.13.0.dev20220923+cu116          pypi_0    pypi
[conda] torchaudio                0.13.0.dev20220923+cu116          pypi_0    pypi
[conda] torchvision               0.14.0.dev20220923+cu116          pypi_0    pypi

```
torch.Tensor.transpose().contiguous() on dimension of size 1 gives  wrong stride 
Labels: module: docs, triaged

### 🐛 Describe the bug

See code snippet below

```python
>>> torch.empty(1, 2, 3, 4, 5).transpose(0,1).contiguous().stride()
(60, 120, 20, 5, 1) # Should be (60, 60, 20, 5, 1)
>>> torch.empty(2, 1, 3, 4, 5).contiguous().stride()
(60, 60, 20, 5, 1)
```



### Versions

Collecting environment information...
PyTorch version: 1.12.0a0+bd13bc6
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.22.3
Libc version: glibc-2.31

Python version: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-48-generic-x86_64-with-glibc2.10
Is CUDA available: False
CUDA runtime version: 11.6.124
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] functorch==0.2.0a0+bee44f8
[pip3] numpy==1.22.3
[pip3] pytorch-quantization==2.1.2
[pip3] torch==1.12.0a0+bd13bc6
[pip3] torch-tensorrt==1.1.0a0
[pip3] torchtext==0.13.0a0
[pip3] torchvision==0.13.0a0
[conda] magma-cuda110             2.5.2                         5    local
[conda] mkl                       2019.5                      281    conda-forge
[conda] mkl-include               2019.5                      281    conda-forge
[conda] numpy                     1.22.3           py38h1d589f8_2    conda-forge
[conda] pytorch-quantization      2.1.2                    pypi_0    pypi
[conda] torch                     1.12.0a0+bd13bc6          pypi_0    pypi
[conda] torch-tensorrt            1.1.0a0                  pypi_0    pypi
[conda] torchtext                 0.13.0a0                 pypi_0    pypi
[conda] torchvision               0.13.0a0                 pypi_0    pypi


cc @svekars @holly1238
[functorch] transforms like jacrev, jacfwd, grad, etc don't work with BatchNorm
Labels: triaged, module: functorch

### 🐛 Describe the bug

```python
import torch
import functorch

net = torch.nn.BatchNorm2d(10)
net.train()
t = torch.randn(3, 3, 10, 10)
functorch.jacrev(net)(t)
```

Error:
```
RuntimeError: During a grad (vjp, jvp, grad, etc) transform, the function provided attempted to call in-place operation (aten::add_.Tensor) that would mutate a captured Tensor. This is not supported; please rewrite the function being transformed to explicitly accept the mutated Tensor(s) as inputs.
```

This also leads to failures on multiple models from https://github.com/pytorch/benchmark

### Versions

master

cc @zou3519 @Chillee @samdow @soumith
Implement `rand_like` ref and implement nvfuser_impl for `uniform` prim
Labels: triaged, module: nvfuser

### 🐛 Describe the bug

In the TorchDynamo+AOT_Autograd+Primtorch stack, Dropout is currently implemented as a decomposition when traced by AOT_Autograd.  The decomposition calls `rand_like` to provide the randomization.  Currently, `rand_like` is neither implemented as a Primtorch Ref and the only randomization implemented in the prims is `uniform` which nvFuser does not implement, yet.

As a work around, nvFuser will add an nvprim for `rand_like`.  #85077 

The real solution is to add a ref for `rand_like` that calls the prim `uniform`.  In order for this to work, nvFuser needs to also implement `uniform`.

nvFuser has an implementation of `uniform` that is waiting to be checked into upstream.  [nvFuser Dev Fork PR 1986](https://github.com/csarofeen/pytorch/pull/1986)

### Versions

TOT
Measure impact of JIT decompositions, reconsider the design
Labels: oncall: jit

We landed https://github.com/pytorch/pytorch/pull/84976. What it does is:
- We have some decompositions written in Python.
- We want to use them from C++, in subsystems like forward-mode AD.
- The design we have is to TorchScript the Python so that it is callable from C++.

Unfortunately this has been causing some problems:
- TorchScript doesn't work with Python 3.11 so this blocks the Python 3.11 binaries (https://github.com/pytorch/pytorch/pull/85509#pullrequestreview-1117733698)
- flaky onnx failures might be related? https://github.com/pytorch/pytorch/issues/85445
- We are unsure of how much this adds to the startup time of `import torch`. Distributed folks may be sensitive to that.

Alternatively, it would not be too difficult to directly call the Python (from C++) instead of relying on TorchScript to shepherd the code through. This may be a better design in the long-term, the question is, should we do anything about this issue in the short term.

cc @soulitzer @malfet
Unexpected assertionError when export tensor.numel
Labels: triaged, bug, oncall: pt2

@voznesenskym Hi, I got this unexpected result when I was playing with the export API.

Repro 1:
```python
import torch
import torchdynamo
def f(x: torch.Tensor):
    return x.numel()

torchdynamo.config.capture_scalar_outputs = True
print(torchdynamo.explain(f, torch.ones(1)))
torchdynamo.export(f, torch.ones(1))
```
The above code outputs the following:

```
('Dynamo produced 0 graphswith -1 graph break and 0 ops\n Break reasons: \n\nTorchDynamo compilation metrics:\nFunction                                               Runtimes (s)\n---------------------------------------------------  --------------\nconvert_frame_assert.<locals>._convert_frame_assert          0.0057', [], [], [], [])
Traceback (most recent call last):
  File "test_dynamo.py", line 8, in <module>
    torchdynamo.export(f, torch.ones(1))
  File "/usr/lib/python3.8/site-packages/torchdynamo-1.13.0.dev0-py3.8-linux-x86_64.egg/torchdynamo/eval_frame.py", line 508, in export
    assert graph is not None, "whole graph export entails exactly one call"
AssertionError: whole graph export entails exactly one call
```
Try
Repro 2:
```python
import torch
import torchdynamo
def f(x: torch.Tensor):
    return x.numel() + x.sin().numel()

torchdynamo.config.capture_scalar_outputs = True
print(torchdynamo.explain(f, torch.ones(1)))
torchdynamo.export(f, torch.ones(1))
```
The above code produces the following error:
```
('Dynamo produced 1 graphswith 0 graph break and 0 ops\n Break reasons: \n\nTorchDynamo compilation metrics:\nFunction                                               Runtimes (s)\n---------------------------------------------------  --------------\nconvert_frame_assert.<locals>._convert_frame_assert          0.0084', [{Guard(name='x', source=<GuardSource.LOCAL: 0>, create_fn=<function GuardBuilder.TENSOR_MATCH at 0x7fe53c4c8f70>, is_volatile=False, guard_types=['TENSOR_MATCH'], code_list=None, obj_weakref=<weakref at 0x7fe53d444130; dead>, guarded_class_weakref=<weakref at 0x7fe5452804a0; to 'torch._C._TensorMeta' at 0x55f9ccd80230 (Tensor)>)}], [GraphModule()], [[]], [])
flat both [tensor([1.])] flat_results_traced [2]
Traceback (most recent call last):
  File "test_dynamo.py", line 8, in <module>
    torchdynamo.export(f, torch.ones(1))
  File "/usr/lib/python3.8/site-packages/torchdynamo-1.13.0.dev0-py3.8-linux-x86_64.egg/torchdynamo/eval_frame.py", line 517, in export
    matched_output_elements_positions = produce_matching(flat_both, flat_results_traced)
  File "/usr/lib/python3.8/site-packages/torchdynamo-1.13.0.dev0-py3.8-linux-x86_64.egg/torchdynamo/eval_frame.py", line 464, in produce_matching
    assert (
AssertionError: Dynamo input and output is a strict subset of traced input/output
```




cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
[AO] In sparisty schedulers, rename `last_epoch` to steps
Labels: oncall: quantization, triaged

It's safer to define t, dt, n and t_0 in terms of steps instead of epochs, because when the data is large (takes days to run 1 epoch), users might want to prune within one epoch.

_Originally posted by @junesg in https://github.com/pytorch/pytorch/pull/85232#discussion_r977067386_

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo
`max_pool2d_with_indices(self, ...)` shouldn't need to save `self` for backward
Labels: module: autograd, triaged

## Issue description

`max_pool2d_with_indices(self, ...)` saves self for backward: https://github.com/pytorch/pytorch/blob/bcf93181a0ca5db75bd038db0d5f7e4cee733db7/tools/autograd/derivatives.yaml#L2208-L2209 .

It shouldn't be necessary to save the storage of self for backward. max_pool2d is an operation that takes a big tensor and select elements from it; the backward pass should just involve taking the grad_output and the indices and scattering it to a big tensor.


cc @ezyang @albanD @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7
Can we rewrite numpy operators to pytorch operators?
Labels: good first issue, triaged, module: numpy

Currently, numpy operations cause graph break, while often exist in training / inference code. But many operators have same semantic in torch operators. Like [YOLOP](https://github.com/hustvl/YOLOP/blob/084d0bb19fa2db4ba40a6e1d6b120f0153d2f335/lib/core/evaluate.py#L230), it would get more than 5x speedup during inference, if we manual replace all the possible `numpy` to `torch`, so they can run in gpu.
So can we dynamicly rewrite these numpy code to torch code using torchdynamo, rewrite pattern may like:
`x = tensor.numpy(); b = np.a(x)` to  `x = torch.a(tensor); b = x.numpy()`
or rewrite all support numpy operator `y = numpy.a(x)` to `x = torch.from_numpy(x); y = torch.a(x); y = y.numpy()` and then erase all the redundant `y = y.numpy(); y = torch.from_numpy(y)`.



cc @mruberry @rgommers
Cannot index into a tensor using indices from another device - regression from 1.12
Labels: triaged, module: regression, module: advanced indexing

### 🐛 Describe the bug

This code works on MPS in pytorch stable `1.12.1`, and also I presume in CUDA (this comes from the [stable-diffusion model](https://github.com/CompVis/stable-diffusion/blob/69ae4b35e0a0f6ee1af8bb9a5d0016ccb27e36dc/ldm/models/diffusion/ddpm.py#L1030), which was trained on CUDA originally).
_Note: you will need to launch python with `PYTORCH_ENABLE_MPS_FALLBACK=1` to get a CPU fallback for `aten::index.Tensor`. that might also explain why it works in stable._

```python
from torch import tensor
tensor([1.], device='cpu')[tensor([0], device='mps')]
# result: tensor([1.])
```

fails in nightly `1.13.0.dev20220917` with:

```
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)
```

I was able to [change the model code to fix it](https://github.com/Birch-san/stable-diffusion/commit/46f793b5ecd5bdd70ee8c9da9c6f379aa566afc5). but since this worked on CUDA and it's a pattern people are using in the wild: other models may be affected.

### Versions

```
PyTorch version: 1.13.0.dev20220917
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.5 (arm64)
GCC version: Could not collect
Clang version: 13.0.0 (clang-1300.0.29.30)
CMake version: version 3.22.1
Libc version: N/A

Python version: 3.10.4 (main, Mar 31 2022, 03:37:37) [Clang 12.0.0 ] (64-bit runtime)
Python platform: macOS-12.5-arm64-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.4
[pip3] pytorch-lightning==1.7.5
[pip3] torch==1.13.0.dev20220917
[pip3] torch-fidelity==0.3.0
[pip3] torchdiffeq==0.2.3
[pip3] torchmetrics==0.9.3
[pip3] torchtyping==0.1.4
[pip3] torchvision==0.14.0.dev20220917
[conda] numpy                     1.22.4                   pypi_0    pypi
[conda] pytorch-lightning         1.7.5                    pypi_0    pypi
[conda] torch                     1.13.0.dev20220917          pypi_0    pypi
[conda] torch-fidelity            0.3.0                    pypi_0    pypi
[conda] torchdiffeq               0.2.3                    pypi_0    pypi
[conda] torchmetrics              0.9.3                    pypi_0    pypi
[conda] torchtyping               0.1.4                    pypi_0    pypi
[conda] torchvision               0.14.0.dev20220917          pypi_0    pypi
```

cc @kulinseth @albanD
`aminmax` will trigger INTERNAL ASSERT if input is empty on cuda
Labels: good first issue, triaged, actionable, module: edge cases

### 🐛 Describe the bug

`aminmax` will trigger INTERNAL ASSERT if input is empty on cuda

```py
import torch

def fn(input):
    return input.aminmax(dim=0)

device = "cuda"

input = torch.empty([36, 0], dtype=torch.float32)
input.uniform_(-16, 3)

fn(input.to(device))
# RuntimeError: iter.numel() > 0 && iter.ntensors() - iter.noutputs() == 1 && iter.noutputs() >= 1 INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1663571524876/work/aten/src/ATen/native/cuda/Reduce.cuh":1134, please report a bug to PyTorch.
```

On cpu, it will just return empty tensor

### Versions

torch: '1.13.0.dev20220919'

cc @pbelevich
Prim Output Spec is Not Always Consistent With Eager
Labels: triaged, module: primTorch

### 🐛 Describe the bug

For instance, [chunk prim](https://github.com/pytorch/pytorch/blob/master/torch/_refs/__init__.py#L2577 ) returns a tuple while [chunk aten](https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/native_functions.yaml#L1269) returns a list.

To test this you could test the specs [here](https://github.com/pytorch/pytorch/pull/85417#discussion_r976977602), among other places.

### Versions

master

cc @ezyang @mruberry @ngimel @Lezcano @fdrocha
Feature Request: Deterministic Algorithm for MaxPool3d
Labels: feature, module: nn, good first issue, triaged, module: determinism, module: pooling

### 🚀 The feature, motivation and pitch

I am working on an encoder that involves use of [torch.nn.MaxPool3d](https://pytorch.org/docs/stable/generated/torch.nn.MaxPool3d.html#torch.nn.MaxPool3d)
I was wondering if there can be a deterministic version for the same

### Alternatives

_No response_

### Additional context

_No response_

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @kurtamohler @kshitij12345 @saketh-are
please report a bug to PyTorch. Expected Object but got PyObject
Labels: triaged, module: torchbind

### 🐛 Describe the bug

There is an extention in Nvidia's fastertransformer which code like this:

https://github.dev/NVIDIA/FasterTransformer/blob/9e26bcd948e5fd59246e84e84225cef0c399c7d4/src/fastertransformer/models/vit/ViT.h#L19


but calling it in python got error:

```
python3.10/site-packages/torch/include/ATen/core/ivalue_inl.h":123, please report a bug to PyTorch. Expected Object but got PyObject
Traceback (most recent call last):
  File "/home/infer_vit_op.py", line 92, in run_vitnv_op
    vit = torch.classes.VisionTransformer.Class(vit_weights.weights, 
RuntimeError: isObject()INTERNAL ASSERT FAILED at "/home/.local/lib/python3.10/site-packages/torch/include/ATen/core/ivalue_inl.h":123, please report a bug to PyTorch. Expected Object but got PyObject
```



https://github.com/NVIDIA/FasterTransformer/issues/321

### Versions

1.13
Please put back missing rocm builds of Torch Vision.
Labels: oncall: binaries, module: rocm, triaged

### 📚 The doc issue

https://pytorch.org/get-started/previous-versions/

Several of the commands in this doc are broken because of missing rocm builds. I have an RX 580 and I can't use the newer versions. I have been trying all kinds of ridiculous workarounds to get it working, but if someone would put the old versions back, I might be able to get it working with supported builds.

### Suggest a potential alternative/fix

Replace the missing builds so the commands will work again.

cc @ezyang @seemethere @malfet @jeffdaily @sunway513 @jithunnair-amd @ROCmSupport @KyleCZH
[FSDP] MixedPrecision, CPUOffload, BackwardPrefetch etc should be documented
Labels: oncall: distributed, triaged, better-engineering, module: fsdp

### 📚 The doc issue

These are auxiliary data structures that do not show up in master documentation, should fix before 1.13 release: https://pytorch.org/docs/master/fsdp.html?highlight=fullyshardeddataparallel#torch.distributed.fsdp.FullyShardedDataParallel

### Suggest a potential alternative/fix

_No response_

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
Inconsistency between geometric distributions
Labels: module: distributions, module: molly-guard, triaged

### 🐛 Describe the bug

`torch.distributions.Geometric` and `torch.Tensor.geometric_` are inconsistent in whether they start from 0 or 1. The former will most commonly return 0 when sampled from, in effect giving you the number of failures before your first success when running repeated Bernoulli trials. The latter tells you on which Bernoulli trial you got your first success. So the latter is the former plus 1. This could be cause for confusion when working with geometric distributions, but changing it would obviously be a breaking change.

```python
In [119]: import torch

In [120]: d = torch.distributions.Geometric(0.3)

In [121]: d.sample()
Out[121]: tensor(0.)

In [122]: [d.sample().item() for _ in range(10)] # starts at 0
Out[122]: [1.0, 3.0, 2.0, 0.0, 1.0, 0.0, 5.0, 8.0, 1.0, 2.0]

In [125]: torch.zeros(10).geometric_(0.9) # starts at 1, will never give 0
Out[125]: tensor([1., 1., 1., 1., 1., 1., 1., 2., 1., 1.])
```

### Versions

```
Collecting environment information...
PyTorch version: 1.13.0.dev20220813
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.5 (arm64)
GCC version: Could not collect
Clang version: 13.1.6 (clang-1316.0.21.2.5)
CMake version: Could not collect
Libc version: N/A

Python version: 3.8.9 (default, Apr 13 2022, 08:48:06)  [Clang 13.1.6 (clang-1316.0.21.2.5)] (64-bit runtime)
Python platform: macOS-12.5-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.1
[pip3] torch==1.13.0.dev20220813
[pip3] torch-tb-profiler==0.4.0
[pip3] torchaudio==0.13.0.dev20220813
[pip3] torchinfo==1.7.0
[pip3] torchvision==0.14.0.dev20220813
[conda] Could not collect
```

cc @fritzo @neerajprad @alicanb @nikitaved
Point community docs to master
Labels: triaged, module: doc infra

### 📚 The doc issue

 We need to make sure that the docs under [community](https://github.com/pytorch/pytorch/tree/master/docs/source/community) in the left nav or other places point to master.

![Screen Shot 2022-09-20 at 10 24 33 AM (2)](https://user-images.githubusercontent.com/5317992/191324301-1248f872-535b-4af4-bac0-43feebd3f2fb.png)



### Suggest a potential alternative/fix

_No response_

cc @ezyang @zou3519 @holly1238
Crash in `torch.package.PackageExporter`
Labels: module: crash, triaged, oncall: package/deploy, imported, module: edge cases

### 🐛 Describe the bug

Passing invalid input to `f` or `importer` in `torch.package.PackageExporter`, can causes a crash with Aborted (core dumped).
## code

```python
import torch

torch.package.PackageExporter(f=1, importer=1)
```

## output
```bash
terminate called after throwing an instance of 'pybind11::error_already_set'
  what():  AttributeError: 'int' object has no attribute 'write'
Aborted (core dumped)
```

### Versions

PyTorch version: 1.12.1+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-117-generic-x86_64-with-glibc2.29
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] torch==1.12.1
[conda] Could not collect
Remove `TypedStorage` and use only `UntypedStorage`
Labels: triaged, module: python frontend

Now that typed storages have been removed from the C++ side and `torch.UntypedStorage` is in place, we can remove `torch.TypedStorage` and all of its subclasses, and just use `UntypedStorage` instead. First, we should add warnings to all of the methods of `TypedStorage` and its subclasses for at least one whole release.
torchrun substitutes host names for IP addresses
Labels: oncall: distributed, triaged, module: elastic

### 🐛 Describe the bug

# Observed Behavior

The `torchrun` command takes host arguments. If I specify those as IP addresses, `torchrun` appears to do a reverse lookup and then use the host names it finds for actually making connections. This fails in some environments.

The following demonstrates the reverse lookup. I created a fake host entry for 192.168.1.99 in /etc/hosts and specified the addresses numerically. Nevertheless, `torchrun` looks up the name and uses it for error reporting (and presumably connecting as well).

```
$ ve
(venv) $ sh -x bug
+ torchrun --master_addr 192.168.1.99 --rdzv_endpoint 192.168.1.99:1234 --no_python /bin/env
[W socket.cpp:558] [c10d] The client socket has failed to connect to [fake-host]:1234 (errno: 113 - No route to host).
[W socket.cpp:558] [c10d] The client socket has failed to connect to fake-host:1234 (errno: 113 - No route to host).
[E socket.cpp:610] [c10d] The client socket has failed to connect to any network address of (192.168.1.99, 1234).
```

The upshot is that `torchrun` fails in an environment where `python -m torch.distributed.launcher` works fine.

# Desired Behavior

If the user specifies hosts as IP addresses, `torchrun` should consistently use those IP addresses for both reporting and making connections.

### Versions

PyTorch version: 1.12.1+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.2.0-19ubuntu1) 11.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-47-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.7.99
GPU models and configuration: GPU 0: NVIDIA RTX A6000
Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.5.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.3
[pip3] torch==1.12.1+cu113
[pip3] torchaudio==0.12.1+cu113
[pip3] torchmore==0.1.0
[pip3] torchvision==0.13.1+cu113
[conda] Could not collect

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
Have NVIDIA driver and other related dependencies as part of the Linux AMI
Labels: module: ci, triaged

### 🚀 The feature, motivation and pitch

At the moment, NVIDIA driver and some other related dependencies like nvidia-docker2 are installed in the CI because they are not part of the base Amazon Linux AMI that we are using [ami-096198a0bccc6bad4](https://github.com/fairinternal/pytorch-gha-infra/blob/main/runners/main.tf#L59).  Specifically, the step is here:

* https://github.com/pytorch/pytorch/blob/master/.github/workflows/_linux-test.yml#L76-L85
* https://github.com/pytorch/pytorch/blob/master/.github/workflows/_binary-test-linux.yml#L173-L184

AFAIK, there is an ongoing task to host these dependencies on S3 to improve the CI reliability https://github.com/pytorch/pytorch/issues/75703.  Nevertheless, it makes sense to eventually have https://github.com/pytorch/pytorch/blob/master/.github/scripts/install_nvidia_utils_linux.sh as part of the Linux AMI we are using for CUDA testing.  As an example, Windows AMI already has them.

### Alternatives

_No response_

### Additional context

_No response_

cc @seemethere @malfet @pytorch/pytorch-dev-infra @atalman
nvFuser support for {ceil,floor,round,trunc}(int)
Labels: triaged, module: nvfuser

### 🐛 Describe the bug

In https://github.com/pytorch/pytorch/pull/85144, we add support for integers for these functions. In this case, they simply return a copy of the input.

For now, I'm just xfailing those tests, but we should add support for this in `nvfuser`.

### Versions

master
Add `persistent` option to `nn.Module.buffers`.
Labels: module: nn, triaged, actionable

### 🚀 The feature, motivation and pitch

For convenience, it would be nice if the [`nn.Module.buffers`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.buffers) could distinguish between persistent and non-persistent buffers.

```python
def buffers(self, recurse: bool = False, persistent: Optional[bool] = None) -> Iterator[Tensor]:
```

Which yields an iterator for either only persistent buffers (`True`), non-persistent buffers (`False`) or both (`None`).

### Alternatives

_No response_

### Additional context

Currently, one either has to make use of the private attribute `nn.Module._non_persistent_buffers_set`, or filter `self.buffers` via the entries of the `state_dict`.

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345 @saketh-are
c10d all_gather aborts with Signal 8 (SIGFPE) when tensor.numel() == 0
Labels: oncall: distributed, module: c10d

### 🐛 Describe the bug

Repro:

Create script "all_gather_test.py" containing:

```
import torch

torch.distributed.init_process_group('gloo')

# works
x = torch.ones((1, 1, 2))
y = [torch.zeros_like(x) for _ in range(torch.distributed.get_world_size())]
torch.distributed.all_gather(y, x)
print(y)

# aborts with
#  exitcode  : -8 (pid: 98824)
#  error_file: <N/A>
#  traceback : Signal 8 (SIGFPE) received by PID 98824
x = torch.ones((1, 0, 2))
y = [torch.zeros_like(x) for _ in range(torch.distributed.get_world_size())]
torch.distributed.all_gather(y, x)
print(y)
```

Run with :

`torchrun --nproc_per_node 4 all_gather_test.py`


I would expect the result to be a list of empty tensors; instead i get process aborts with signal 8 on all ranks.

### Versions

pythorch nightly Sept/15/2022

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501
[asan] ctc_loss fails test_make_fx_fake_exhaustive with ASAN
Labels: module: nn, triaged, module: sanitizers, oncall: pt2, module: fakeTensor, module: ProxyTensor

### 🐛 Describe the bug

```
==729==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61f00041847c at pc 0x7f01905b4988 bp 0x7ffc03665d90 sp 0x7ffc03665d88
[1680](https://github.com/pytorch/pytorch/actions/runs/3074016256/jobs/4966582993#step:10:1681)
READ of size 4 at 0x61f00041847c thread T0
[1681](https://github.com/pytorch/pytorch/actions/runs/3074016256/jobs/4966582993#step:10:1682)
    #0 0x7f01905b4987 in std::tuple<at::Tensor, at::Tensor> at::native::(anonymous namespace)::ctc_loss_cpu_template<float, (c10::ScalarType)4>(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, long)::'lambda'(long, long)::operator()(long, long) const (/opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so+0xdc2a987)
[1682](https://github.com/pytorch/pytorch/actions/runs/3074016256/jobs/4966582993#step:10:1683)
    #1 0x7f018fb23557 in std::function<void (long, long)>::operator()(long, long) const (/opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so+0xd199557)
```

The test fails on CI for PR https://github.com/pytorch/pytorch/pull/84752

Ref: https://github.com/pytorch/pytorch/pull/84752#issuecomment-1243847936

### Versions

[PR ](https://github.com/pytorch/pytorch/pull/84752)

cc @ezyang @gchanan @zou3519 @albanD @mruberry @jbschlosser @walterddr @saketh-are @SherlockNoMad @soumith
[MPS] load checkpoints gives zero weights when map_location is mps
Labels: triaged, has workaround, module: correctness (silent), module: mps

### 🐛 Describe the bug

When loading checkpoints of a larger model directly to MPS the weights are zero. For CPU it works. I tested it also with a tiny model and there it seems to work, so to replicate you need to download our weights (see code). It may be related to #79384 and #78551 . As a workaround it works to move model to CPU, set the weights there and then move everything to MPS.  Here is an example to replicate:

```
import torch
import requests
# Download checkpoint file
ckpt='FastSurferVINN_training_state_coronal.pkl'
fileurl='https://b2share.fz-juelich.de/api/files/0114331a-f788-48d2-9d09-f85d7494ed48/FastSurferVINN_training_state_coronal.pkl'
response = requests.get(fileurl, verify=False)
with open(ckpt, 'wb') as f:
    f.write(response.content)

# CPU load works:
model_state = torch.load(ckpt, map_location="cpu")
print(model_state["model_state"]["inp_block.bn0.weight"])
# ouput: tensor([2.0432, 1.2577, 4.1133, 7.4062, 3.9921, 1.8011, 2.0956])

# MPS load gives zeros:
model_state = torch.load(ckpt, map_location="mps")
print(model_state["model_state"]["inp_block.bn0.weight"])
#output tensor([0., 0., 0., 0., 0., 0., 0.], device='mps:0')
```

### Versions

Collecting environment information...
PyTorch version: 1.13.0.dev20220917
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.0 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.102)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.6 (default, Aug  5 2022, 15:21:02)  [Clang 14.0.0 (clang-1400.0.29.102)] (64-bit runtime)
Python platform: macOS-13.0-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.2
[pip3] torch==1.13.0.dev20220917
[pip3] torchio==0.18.83
[pip3] torchvision==0.14.0.dev20220916
[conda] Could not collect

cc @kulinseth @albanD
TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. 
Labels: oncall: jit, triaged

### 🐛 Describe the bug

model https://github.com/TencentARC/GFPGAN/releases/download/v1.3.0/GFPGANv1.3.pth
my code
```
# importing module
import sys
sys.path.append('./')

import torch
import torchvision
from gfpgan.archs.gfpganv1_clean_arch import GFPGANv1Clean
model_path = "GFPGANv1.3.pth"

model = GFPGANv1Clean(
    out_size=512,
    num_style_feat=512,
    channel_multiplier=2,
    decoder_load_path=None,
    fix_decoder=False,
    num_mlp=8,
    input_is_latent=True,
    different_w=True,
    narrow=1,
    sft_half=True)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
loadnet = torch.load(model_path)
if 'params_ema' in loadnet:
    keyname = 'params_ema'
else:
    keyname = 'params'

model.load_state_dict(loadnet[keyname], strict=True)    

# Switch the model to eval model
model.eval()
model = model.to(device)
example1 = torch.rand(1, 3, 512, 512)
# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, (example1))

# Save the TorchScript model
traced_script_module.save("gfpganv1_clean_model.pt")
```

function. Detailed error:
Tensor-likes are not close!

Mismatched elements: 784625 / 786432 (99.8%)
Greatest absolute difference: 0.1886063814163208 at index (0, 2, 375, 303) (up to 1e-05 allowed)
Greatest relative difference: 3520.956626506024 at index (0, 1, 238, 193) (up to 1e-05 allowed)
  _check_trace(




### Versions

Collecting environment information...
PyTorch version: 1.12.1+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 7 Professional
GCC version: (x86_64-posix-seh-rev3, Built by MinGW-W64 project) 12.1.0
Clang version: 14.0.4
CMake version: version 3.24.0-rc1
Libc version: N/A

Python version: 3.8.6 (tags/v3.8.6:db45529, Sep 23 2020, 15:52:53) [MSC v.1927 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-7-6.1.7601-SP1
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] pytorch2caffe==0.1.0
[pip3] torch==1.12.1
[pip3] torchaudio==0.12.1
[pip3] torchvision==0.13.1
[pip3] torchviz==0.0.2
[conda] Could not collect