Large slow down by not calling `torch.set_num_threads`
Labels: module: performance, module: cpu, triaged, module: linear algebra

### 🐛 Describe the bug

By default PyTorch sets the number of threads to the number of cores. Strangely, I found that if I don't call `torch.set_num_threads` in a computer with 20 cores (and use the default value of 20) the job runs very slowly. However, if I call `torch.set_num_threads(20)`  the job runs much faster. Very stange ...

The minimal working example below illustrates this problem. If I call this script without arguments, it will not call  `torch.set_num_threads` and in a computer with 20 cores it takes 7 seconds. But if I call this script with argument `--set_num_threads` it takes 4 seconds.

In another computer with 6 cores, if I do not call `set_num_threads` the script takes 5.3 seconds, but if I call  `set_num_threads` the script takes 4.98 seconds.

The script below is a minimal working example. I am experiencing very large slow downs, on the order of 10, on a more complex piece of code.

This [discussion](https://discuss.pytorch.org/t/how-to-check-if-pytorch-is-using-blas/50068/3) pointed me in the right direction to understand this problem.

``` python                                                                                                                                                                                                                                             
import sys
import time
import argparse
import torch

def main(argv):
    parser = argparse.ArgumentParser()
    parser.add_argument("--set_num_threads", action="store_true",
                        help="set the number of threads to the number of cores")
    parser.add_argument("--n_iter", type=int, help="number of iterations", default=100)
    parser.add_argument("--n_rows", type=int, help="number of rows",
                        default=1000)
    parser.add_argument("--n_cols", type=int, help="number of columns",
                        default=1000)
    args = parser.parse_args()

    set_num_threads = args.set_num_threads
    n_iter = args.n_iter
    n_rows = args.n_rows
    n_cols = args.n_cols

    if set_num_threads > 0:
        print("called torch.set_num_threads")
        torch.set_num_threads(torch.get_num_threads())
    else:
        print("NOT called torch.set_num_threads")

    print(f"Using {torch.get_num_threads()} threads")

    start_time = time.time()
    for i in range(n_iter):
        m = torch.randn((n_rows, n_cols), dtype=torch.double)
        b = torch.randn((n_rows, 1), dtype=torch.double)
        m.requires_grad = True
        L = torch.linalg.cholesky(torch.matmul(m, m.T))
        x = torch.cholesky_solve(b, L)
        m.requires_grad = False
    end_time = time.time()
    print(f"Processing time {end_time-start_time}")

    breakpoint()


if __name__ == "__main__":
    main(sys.argv)
```

### Versions

Collecting environment information...
PyTorch version: 1.12.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.17

Python version: 3.7.13 (default, Mar 29 2022, 02:18:16)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-132-generic-x86_64-with-debian-bullseye-sid
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] gpytorch==1.2.1
[pip3] mypy==0.910
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] numpydoc==1.4.0
[pip3] torch==1.12.1
[pip3] torchvision==0.13.1a0
[conda] _pytorch_select           0.2                       gpu_0  
[conda] blas                      1.0                         mkl  
[conda] cpuonly                   1.0                           0    pytorch
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] gpytorch                  1.2.1                    pypi_0    pypi
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py37h7f8727e_0  
[conda] mkl_fft                   1.3.1            py37hd3c417c_0  
[conda] mkl_random                1.2.2            py37h51133e4_0  
[conda] numpy                     1.21.5           py37h6c91a56_3  
[conda] numpy-base                1.21.5           py37ha15fc14_3  
[conda] numpydoc                  1.4.0            py37h06a4308_0  
[conda] pytorch                   1.12.1          cpu_py37he8d8e81_0  
[conda] torchvision               0.13.1          cpu_py37h164cc8f_0 

cc @ngimel @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @Lezcano