torch.jit.script segmentation fault (pytorch debayer module) 1.10, 1.11 and nightly
Labels: oncall: jit

### 🐛 Describe the bug

Offending module(s) can be found by installing pytorch-debayer:
`pip install git+https://github.com/cheind/pytorch-debayer`

```
import torch
import debayer
m = debayer.Debayer3x3()
torch.jit.script(m)
```
```
>>> import torch
to>>> torch.__version__
'1.13.0.dev20220603'
>>> import debayer
>>> m = debayer.Debayer3x3()
>>> torch.jit.script(m)
[1]    111518 segmentation fault (core dumped)  python
```

### Versions

The segmentation fault occurs on at least 1.10, 1.11 and nightly.
```
Collecting environment information...
PyTorch version: 1.13.0.dev20220603
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.23.2
Libc version: glibc-2.31

Python version: 3.10.4 | packaged by conda-forge | (main, Mar 24 2022, 17:38:57) [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-5.4.0-113-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.6.55
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2070
Nvidia driver version: 510.47.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.3.2
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.3.2
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.4
[pip3] pytorch-debayer==1.4.1
[pip3] torch==1.13.0.dev20220603
[pip3] torchaudio==0.12.0.dev20220603
[pip3] torchvision==0.14.0.dev20220603
[conda] blas                      2.114                       mkl    conda-forge
[conda] blas-devel                3.9.0            14_linux64_mkl    conda-forge
[conda] cudatoolkit               11.3.1              h9edb442_10    conda-forge
[conda] libblas                   3.9.0            14_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            14_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            14_linux64_mkl    conda-forge
[conda] liblapacke                3.9.0            14_linux64_mkl    conda-forge
[conda] mkl                       2022.0.1           h8d4b97c_803    conda-forge
[conda] mkl-devel                 2022.0.1           ha770c72_804    conda-forge
[conda] mkl-include               2022.0.1           h8d4b97c_803    conda-forge
[conda] numpy                     1.22.4          py310h4ef5377_0    conda-forge
[conda] pytorch                   1.13.0.dev20220603 py3.10_cuda11.3_cudnn8.3.2_0    pytorch-nightly
[conda] pytorch-debayer           1.4.1                    pypi_0    pypi
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] torchaudio                0.12.0.dev20220603     py310_cu113    pytorch-nightly
[conda] torchvision               0.14.0.dev20220603     py310_cu113    pytorch-nightly
```