Segmentation fault on aarch64 (Rpi4) using Pytorch 2.1.0 & torchaudio 
Labels: oncall: binaries

### 🐛 Describe the bug

On my raspberry pi 4 (aarch64 arch) with pytorch 2.1.0 & torchaudio i have a segmentation fault using microsoft [unilm/beats](https://github.com/microsoft/unilm/blob/master/beats/README.md) model. Same code run fine on my macos 11 (x86_64 arch) with same package version and code.

On the pi:
Rapsbian OS version : 11.8 bullseye (uptodate)
packaged installed:
python392 and python311 (same seg fault)
numpy              1.26.0
pandas             2.1.1
Pillow             10.0.1
torch              2.1.0
torchaudio         2.1.0
torchvision        0.16.0


```python
$python3 -m venv "testenv"
$source testenv/bin/activate
$pip3 install torch torchaudio
$python3

import torch
from BEATs import BEATs, BEATsConfig

# load the fine-tuned checkpoints
checkpoint = torch.load('/path/to/model.pt')

cfg = BEATsConfig(checkpoint['cfg'])
BEATs_model = BEATs(cfg)
BEATs_model.load_state_dict(checkpoint['model'])
BEATs_model.eval()

# predict the classification probability of each class
audio_input_16khz = torch.randn(3, 10000)
padding_mask = torch.zeros(3, 10000).bool()

probs = BEATs_model.extract_features(audio_input_16khz, padding_mask=padding_mask)[0]

for i, (top5_label_prob, top5_label_idx) in enumerate(zip(*probs.topk(k=5))):
    top5_label = [checkpoint['label_dict'][label_idx.item()] for label_idx in top5_label_idx]
    print(f'Top 5 predicted labels of the {i}th audio are {top5_label} with probability of {top5_label_prob}')


Output:
python3.11/site-packages/torch/nn/utils/weight_norm.py:30: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
  warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
Segmentation fault
```

The sample code is working fine on aarch64 with pytorch 1.13.1 version (no seg fault), detail installed package:
numpy              1.24.1
pandas             1.1.5
Pillow             8.1.2
torch              1.13.1
torchaudio         0.13.1
torchvision        0.14.1

and it also work with torch==2.0.1 and torchaudio==2.0.2.

Many thanks for the help.


### Versions

PyTorch version: 2.1.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 11 (bullseye) (aarch64)
GCC version: (Debian 10.2.1-6) 10.2.1 20210110
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.11.6 (main, Oct  8 2023, 12:45:00) [GCC 10.2.1 20210110] (64-bit runtime)
Python platform: Linux-6.1.21-v8+-aarch64-with-glibc2.31
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
Vendor ID:                       ARM
Model:                           3
Model name:                      Cortex-A72
Stepping:                        r0p3
CPU max MHz:                     2000.0000
CPU min MHz:                     600.0000
BogoMIPS:                        108.00
L1d cache:                       128 KiB
L1i cache:                       192 KiB
L2 cache:                        1 MiB
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Vulnerable
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm crc32 cpuid

Versions of relevant libraries:
[pip3] numpy==1.26.0
[pip3] torch==2.1.0
[pip3] torchaudio==2.1.0
[pip3] torchvision==0.16.0
[conda] Could not collect

cc @seemethere @malfet @osalpekar @atalman