Weird dataloader performance degradation caused by torch and numpy import order
Labels: triaged, module: openmp

### 🐛 Describe the bug

Hi, 

I recently noticed a weird behavior in Pytorch. The import order of torch and numpy can have a significant impact on the dataloader performance. In short (and see the complete example script below):
```
# Setting A: faster. 100 iterations take 91 seconds. Average load time 0.035.
import torch
import numpy as np

# Setting B: slower. 100 iterations take 158 seconds. Average load time ~0.45.
import numpy as np
import torch
```
It seems this performance is determined by the first time these two packages are imported (e.g., importing these two packages in the main script in B order and the dataloader script in A order, it will end up with B performance).

## Reproduction
I used AWS EC2 machines (and more specifically, zone `us-west-2` with `p4de24xlarge` instances). I am not sure if this is reproducible in other places.

### Step 1: Dockerfile
```
FROM nvidia/cuda:11.3.0-devel-ubuntu20.04

# Fix NV docker problem
RUN apt-key adv --fetch-keys https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub
RUN apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/7fa2af80.pub

# Install handy tools
RUN apt-get update && apt-get install vim htop tmux sudo git wget -y && rm -rf /var/lib/apt/lists/*

# Create user with sudo privilege
RUN addgroup --gid 1000 ubuntu
RUN adduser --disabled-password --gecos '' --uid 1000 --gid 1000 ubuntu 
RUN adduser ubuntu sudo
RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
USER ubuntu
USER 1000:1000
RUN sudo chown ubuntu:ubuntu /home/ubuntu/ # Actually, not sure if this is even needed

# Conda
ENV PATH="/home/ubuntu/miniconda3/bin:${PATH}"
ARG PATH="/home/ubuntu/miniconda3/bin:${PATH}"
RUN sudo apt-get install -y wget
RUN cd /home/ubuntu/ \
    && wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && bash Miniconda3-latest-Linux-x86_64.sh -b -p /home/ubuntu/miniconda3/ \
    && rm -f Miniconda3-latest-Linux-x86_64.sh 
RUN conda install python=3.9 pytorch torchvision torchaudio pytorch-cuda=11.8 xformers -c pytorch -c nvidia -c xformers -y
RUN pip install tqdm
RUN conda clean --all -y
```

### Step 2: After getting into the environment
Add these lines to `~/.bashrc` and `source ~/.bashrc` to init conda
``` #bash
conda_root="/home/ubuntu/miniconda3/"
conda_steup_bin="${conda_root}bin/conda"
__conda_setup="$($conda_steup_bin 'shell.bash' 'hook')"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "${conda_root}etc/profile.d/conda.sh" ]; then
        . "${conda_root}etc/profile.d/conda.sh"
    else
        export PATH="${conda_root}bin:$PATH"
    fi
fi
unset __conda_setup
```

### Step 3: Example script
```python
# Switching these two lines will get a different performance
import torch
import numpy as np

from torch.utils.data import Dataset
import time

class MyDataset(Dataset):
    def __getitem__(self, i):
        st = time.time()
        data = {k: np.random.rand(3, 512, 512) for k in range(6)} # it seems only numpy has the issue
        print(" [*] Worker load time {:.4f}".format(time.time()-st))
        return data

    def __len__(self):
        return 1000000

if __name__ == "__main__":
    from torch.utils.data import DataLoader
    from tqdm import tqdm

    dataloader = DataLoader(MyDataset())
    for data in tqdm(dataloader):
        pass
```

### Versions

Collecting environment information...
PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.9.16 (main, Mar  8 2023, 14:00:05)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.4.228-132.418.amzn2.x86_64-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.3.109
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB
GPU 2: NVIDIA A100-SXM4-40GB
GPU 3: NVIDIA A100-SXM4-40GB
GPU 4: NVIDIA A100-SXM4-40GB
GPU 5: NVIDIA A100-SXM4-40GB
GPU 6: NVIDIA A100-SXM4-40GB
GPU 7: NVIDIA A100-SXM4-40GB

Nvidia driver version: 470.161.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          96
On-line CPU(s) list:             0-95
Thread(s) per core:              2
Core(s) per socket:              24
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           85
Model name:                      Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
Stepping:                        7
CPU MHz:                         1916.685
BogoMIPS:                        5999.99
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       1.5 MiB
L1i cache:                       1.5 MiB
L2 cache:                        48 MiB
L3 cache:                        71.5 MiB
NUMA node0 CPU(s):               0-23,48-71
NUMA node1 CPU(s):               24-47,72-95
Vulnerability Itlb multihit:     KVM: Vulnerable
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Mmio stale data:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed:          Vulnerable
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==2.0.0
[pip3] torchaudio==2.0.0
[pip3] torchvision==0.15.0
[pip3] triton==2.0.0
[conda] blas                      1.0                         mkl  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2023.1.0         h6d00ec8_46342  
[conda] mkl-service               2.4.0            py39h5eee18b_1  
[conda] mkl_fft                   1.3.6            py39h417a72b_1  
[conda] mkl_random                1.2.2            py39h417a72b_1  
[conda] numpy                     1.24.3           py39hf6e8229_1  
[conda] numpy-base                1.24.3           py39h060ed82_1  
[conda] pytorch                   2.0.0           py3.9_cuda11.8_cudnn8.7.0_0    pytorch
[conda] pytorch-cuda              11.8                 h7e8668a_5    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                2.0.0                py39_cu118    pytorch
[conda] torchtriton               2.0.0                      py39    pytorch
[conda] torchvision               0.15.0               py39_cu118    pytorch

cc @SsnL @VitalyFedyunin @ejguan @NivekT @dzhulgakov