`torch.jit.load()` might unresponsive in IBM s390x when loading some certain torchscript saved by x86 machine.
Labels: oncall: jit

### 🐛 Describe the bug

## Step to reproduce
1. Load a `resnet50` model on a **x86 machine** and save it as an torchscript.
```python
# load modules
import torch
import numpy as np

model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet50', pretrained=True).to('cuda')
model.eval()

# Try to export the model to torchscript

import torch.jit

jit_script = torch.jit.script(model)
jit_script.save('resnet50.pt')
```
2. Copy the exported torchscript file `resnet50.pt` to a linuxone (IBM s390X) machine, then run the following code to load the model:
```python
import torch
import torch.jit
from torch.serialization import LoadEndianness

torch.serialization.set_default_load_endianness(LoadEndianness.LITTLE)

jit_script = torch.jit.load('resnet50.pt', map_location='cpu')
```
3. Then the loading process never completed, and if you inspect the os with `top` command, you will find `python` is keep allocating memory, until the system kill the process.
![image](https://github.com/pytorch/pytorch/assets/29349119/ac7fd15e-3267-4c14-85e0-ba4c95df1c22)

4. As a contract, if I complete the Step1 with a s390x machine, then everything works as expected.

## Other information

I believe this error may be related to the `byteorder` used when loading the torchscript, as discussed in issues #101688 and #101973. Although this problem should have been resolved in version 2.1.0, there might still be some bugs when loading a torchscript, which could be causing the issue.

### Versions

1. x86 machine used to save the `resnet50.pt` torchscript:
```
Collecting environment information...
PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Pro Insider Preview
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.25.1
Libc version: N/A

Python version: 3.11.4 | packaged by Anaconda, Inc. | (main, Jul  5 2023, 13:47:18) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.23550-SP0
Is CUDA available: True
CUDA runtime version: 11.8.89

CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060 Laptop GPU
Nvidia driver version: 537.13
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9


CurrentClockSpeed=3201


DeviceID=CPU0


Family=107


L2CacheSize=4096


L2CacheSpeed=


Manufacturer=AuthenticAMD


MaxClockSpeed=3201


Name=AMD Ryzen 7 5800H with Radeon Graphics         


ProcessorType=3


Revision=20480

Versions of relevant libraries:
[pip3] numpy==1.25.2
[pip3] torch==2.0.1
[pip3] torchaudio==2.0.2
[pip3] torchvision==0.15.2
[conda] Could not collect

```

2. s390x machine (aka LinuxOne) used to load the `resnet50.pt` torchscript:
```
PyTorch version: 2.1.0a0+git591cb77
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (s390x)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.14.0-284.25.1.el9_2.s390x-s390x-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False

CPU:
Architecture:                    s390x
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Big Endian
CPU(s):                          2
On-line CPU(s) list:             0,1
Vendor ID:                       IBM/S390
Machine type:                    8561
Thread(s) per core:              1
Core(s) per socket:              1
Socket(s) per book:              1
Book(s) per drawer:              1
Drawer(s):                       2
CPU dynamic MHz:                 5200
CPU static MHz:                  5200
BogoMIPS:                        3241.00
Dispatching mode:                horizontal
Flags:                           esan3 zarch stfle msa ldisp eimm dfp edat etf3eh highgprs te vx vxd vxe gs vxe2 vxp sort dflt sie
Hypervisor:                      z/VM 7.2.0
Hypervisor vendor:               IBM
Virtualization type:             full
L1d cache:                       256 KiB (2 instances)
L1i cache:                       256 KiB (2 instances)
L2d cache:                       4 MiB (1 instance)
L2i cache:                       4 MiB (1 instance)
L3 cache:                        256 MiB
L4 cache:                        960 MiB
NUMA node(s):                    1
NUMA node0 CPU(s):               0,1
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; etokens
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] numpy==1.21.5
[pip3] torch==2.1.0a0+git591cb77
[conda] Could not collect
```

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel