large number of temporary files generated when using dataloader with num_workers>0
Labels: high priority, module: dataloader, triaged, module: openmp

### 🐛 Describe the bug

Hi. 
I recently noticed an issue when I use the dataloader ('torch.utils.data.DataLoader') with passing 'num_workers>0'.  When this parallel process is activated, it generated a large number of temporary files starting with '__KMP_REGISTERED_LIB*' under a shard folder '/dev/shm' under my Linux system. And they are not automatically deleted, which surprised me. The number generated is proportional to the number of workers used. I think this should probably be fixed by something like temporary files should be deleted at the end of script. Thank you

### Versions

PyTorch version: 1.10.1+cu111                                                                                           Is debug build: False                                                                                                   CUDA used to build PyTorch: 11.1                                                                                        ROCM used to build PyTorch: N/A                                                                                                                                                                                                                 OS: Ubuntu 20.04.5 LTS (x86_64)                                                                                         GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0                                                                      Clang version: Could not collect                                                                                        CMake version: version 3.16.3                                                                                           Libc version: glibc-2.10                                                                                                                                                                                                                        Python version: 3.7.9 (default, Aug 31 2020, 12:42:55)  [GCC 7.3.0] (64-bit runtime)                                    Python platform: Linux-5.4.0-137-generic-x86_64-with-debian-bullseye-sid                                                Is CUDA available: True                                                                                                 CUDA runtime version: 11.0.221                                                                                          CUDA_MODULE_LOADING set to:                                                                                             GPU models and configuration:                                                                                           GPU 0: A100-SXM4-40GB                                                                                                   GPU 1: A100-SXM4-40GB                                                                                                   GPU 2: A100-SXM4-40GB                                                                                                   GPU 3: A100-SXM4-40GB                                                                                                   GPU 4: A100-SXM4-40GB                                                                                                   GPU 5: A100-SXM4-40GB                                                                                                   GPU 6: A100-SXM4-40GB                                                                                                   GPU 7: A100-SXM4-40GB                                                                                                                                                                                                                           Nvidia driver version: 450.216.04                                                                                       cuDNN version: Probably one of the following:                                                                           /usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn.so.8.1.1                                                         /usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.1.1                                               /usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.1.1                                               /usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.1.1                                               /usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.1.1                                               /usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.1.1                                               /usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.1.1                                               /usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn.so.8.2.1                                                         /usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.2.1                                               /usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.2.1                                               /usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.2.1                                               /usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.2.1                                               /usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.2.1                                               /usr/local/cuda-11.3/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.2.1                                               HIP runtime version: N/A                                                                                                MIOpen runtime version: N/A                                                                                             Is XNNPACK available: True                                                                                                                                                                                                                      Versions of relevant libraries:                                                                                         [pip3] block.bootstrap.pytorch==0.1.6                                                                                   [pip3] bootstrap.pytorch==0.0.13                                                                                        [pip3] numpy==1.21.2                                                                                                    [pip3] torch==1.10.1+cu111                                                                                              [pip3] torchaudio==0.10.1+rocm4.1                                                                                       [pip3] torchsummary==1.5.1                                                                                              [pip3] torchvision==0.11.2+cu111                                                                                        [conda] blas                      1.0                         mkl                                                       [conda] block-bootstrap-pytorch   0.1.6                    pypi_0    pypi                                               [conda] bootstrap-pytorch         0.0.13                   pypi_0    pypi                                               [conda] cudatoolkit               11.1.74              h6bb024c_0    nvidia                                             [conda] ffmpeg                    4.3                  hf484d3e_0    pytorch                                            [conda] mkl                       2021.3.0           h06a4308_520                                                       [conda] mkl-service               2.4.0            py37h7f8727e_0                                                       [conda] mkl_fft                   1.3.1            py37hd3c417c_0                                                       [conda] mkl_random                1.2.2            py37h51133e4_0                                                       [conda] numpy                     1.20.0                   pypi_0    pypi                                               [conda] numpy-base                1.21.2           py37h79a1101_0                                                       [conda] pytorch-mutex             1.0                        cuda    pytorch                                            [conda] torch                     1.10.1+cu111             pypi_0    pypi                                               [conda] torchaudio                0.10.1+rocm4.1           pypi_0    pypi                                               [conda] torchsummary              1.5.1                    pypi_0    pypi                                               [conda] torchvision               0.11.2+cu111             pypi_0    pypi 

cc @ezyang @gchanan @zou3519 @SsnL @VitalyFedyunin @ejguan @NivekT