slow torch import on macos 
Labels: module: performance, triaged, module: macos

### 🐛 Describe the bug

I’m having a hard time understanding the slow import of torch in later versions, including the most recent 2.0 release. I dug as deep as I could, and ran `cProfile` on torch's `__init__.py`: 
```shell
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1287    1.824    0.001    1.824    0.001 {built-in method io.open_code}         <--------------
    45/25    1.045    0.023    1.579    0.063 {built-in method _imp.create_dynamic}  <--------------
     1287    0.136    0.000    0.136    0.000 {built-in method marshal.loads}
     5408    0.064    0.000    0.064    0.000 {built-in method posix.stat}
2841/2785    0.050    0.000    0.240    0.000 {built-in method builtins.__build_class__}
      424    0.047    0.000    0.080    0.000 assumptions.py:596(__init__)
   1635/1    0.042    0.000    4.315    4.315 {built-in method builtins.exec}
        2    0.040    0.020    0.040    0.020 {built-in method _ctypes.dlopen}
    51906    0.026    0.000    0.062    0.000 {built-in method builtins.getattr}
     1298    0.026    0.000    0.026    0.000 {method '__exit__' of '_io._IOBase' objects}
      164    0.023    0.000    0.023    0.000 {built-in method posix.listdir}
     1769    0.020    0.000    0.131    0.000 <frozen importlib._bootstrap_external>:1604(find_spec)
     1287    0.019    0.000    0.019    0.000 {method 'read' of '_io.BufferedReader' objects}
```

The `_imp.create_dynamic` is part of `importlib.machinery.ModuleSpec` and is called once inside `package_importer.py` here:
![image](https://user-images.githubusercontent.com/62313634/226159192-6022449e-f492-4ca2-bfbb-61f9967ce8ab.png)

I'm sorry in advance if this is not a bug, but I don't know where to post this and is becoming really annoying. I also ruled out is only on my end, a friend of mine has mac too, but he's on Big Sur, and uses python 3.10. 

### Versions

```shell
Collecting environment information...
PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.2.1 (x86_64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: version 3.25.2
Libc version: N/A

Python version: 3.11.0 (main, Mar  1 2023, 12:49:28) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-10.16-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz

Versions of relevant libraries:
[pip3] numpy==1.24.2
[pip3] torch==2.0.0
[pip3] torchaudio==2.0.1
[pip3] torchvision==0.15.1
[conda] numpy                     1.24.2                   pypi_0    pypi
[conda] torch                     2.0.0                    pypi_0    pypi
[conda] torchaudio                2.0.1                    pypi_0    pypi
[conda] torchvision               0.15.1                   pypi_0    pypi
```

cc @ngimel @malfet @albanD