DataLoader CPU utilization and slow training

Haydnspass: <p>Hi,</p>
<p>My project runs fast on my workstation at around 100% GPU utilization on an RTX 3090 but very slow on a server machine with an H100 and many CPU cores.<br>
The code simulates data, so I don’t think it is related to reading/write to/from SSD. I noticed that no matter how many workers I set on the cluster, 2 threads are at 100% utilization, and all workers are almost idle. If I set 64 workers, the GPU waits for CPU, goes through 64 batches at ~100% utilization and then waits again. I used this high number to make it easier to see.</p>
<p>I wonder what this could be. Could it be related to some OMP stuff? I tried setting OMP_NUM_THREADS=1 without luck. pin_memory defaults to False.</p>
<p>Please find a screenshot of the utilization on the cluster attached.<br>
The environment is</p>
<pre><code class="lang-auto"># packages in environment at /g/ries/users/Lucas/slurm/conda/envs/decode_dev:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
absl-py                   1.4.0              pyhd8ed1ab_0    conda-forge
accessible-pygments       0.0.4              pyhd8ed1ab_0    conda-forge
aiohttp                   3.8.4           py310h2372a71_1    conda-forge
aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
alabaster                 0.7.13             pyhd8ed1ab_0    conda-forge
alsa-lib                  1.2.8                h166bdaf_0    conda-forge
antlr-python-runtime      4.9.3              pyhd8ed1ab_1    conda-forge
aom                       3.5.0                h27087fc_0    conda-forge
asttokens                 2.2.1              pyhd8ed1ab_0    conda-forge
async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
attr                      2.5.1                h166bdaf_1    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
babel                     2.12.1             pyhd8ed1ab_1    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.5              pyhd8ed1ab_0    conda-forge
backports.zoneinfo        0.2.1           py310hff52083_7    conda-forge
beautifulsoup4            4.12.2             pyha770c72_0    conda-forge
black                     23.3.0                   pypi_0    pypi
blas                      1.0                         mkl    conda-forge
blinker                   1.6.2              pyhd8ed1ab_0    conda-forge
blosc                     1.21.4               h0f2a231_0    conda-forge
brotli                    1.0.9                h166bdaf_8    conda-forge
brotli-bin                1.0.9                h166bdaf_8    conda-forge
brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
brunsli                   0.1                  h9c3ff4c_0    conda-forge
bump2version              1.0.1              pyh9f0ad1d_0    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
c-blosc2                  2.9.3                hb4ffafa_0    conda-forge
ca-certificates           2023.5.7             hbcca054_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                5.3.0              pyhd8ed1ab_0    conda-forge
caerus                    0.1.9                    pypi_0    pypi
cairo                     1.16.0            ha61ee94_1014    conda-forge
certifi                   2023.5.7           pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h255011f_3    conda-forge
cfitsio                   4.2.0                hd9d235c_0    conda-forge
charls                    2.4.2                h59595ed_0    conda-forge
charset-normalizer        3.1.0              pyhd8ed1ab_0    conda-forge
click                     8.1.3           unix_pyhd8ed1ab_2    conda-forge
cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
commonmark                0.9.1                      py_0    conda-forge
contourpy                 1.1.0           py310hd41b1e2_0    conda-forge
cryptography              41.0.1          py310h75e40e8_0    conda-forge
cuda-cudart               11.8.89                       0    nvidia
cuda-cupti                11.8.87                       0    nvidia
cuda-libraries            11.8.0                        0    nvidia
cuda-nvrtc                11.8.89                       0    nvidia
cuda-nvtx                 11.8.86                       0    nvidia
cuda-runtime              11.8.0                        0    nvidia
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cytoolz                   0.12.0          py310h5764c6d_1    conda-forge
dask-core                 2023.6.0           pyhd8ed1ab_0    conda-forge
dav1d                     1.2.1                hd590300_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
deprecated                1.2.14             pyh1a96a4e_0    conda-forge
docutils                  0.19            py310hff52083_1    conda-forge
exceptiongroup            1.1.1              pyhd8ed1ab_0    conda-forge
execnet                   1.9.0              pyhd8ed1ab_0    conda-forge
executing                 1.2.0              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
fftw                      3.3.10          nompi_hc118613_108    conda-forge
filelock                  3.12.2             pyhd8ed1ab_0    conda-forge
findpeaks                 2.4.7                    pypi_0    pypi
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.40.0          py310h2372a71_0    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
frozenlist                1.3.3           py310h5764c6d_0    conda-forge
fsspec                    2023.6.0           pyh1a96a4e_0    conda-forge
future                    0.18.3             pyhd8ed1ab_0    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
giflib                    5.2.1                h0b41bf4_3    conda-forge
gitdb                     4.0.10             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.31             pyhd8ed1ab_0    conda-forge
glib                      2.76.3               hfc55251_0    conda-forge
glib-tools                2.76.3               hfc55251_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gmpy2                     2.1.2           py310h3ec546c_1    conda-forge
google-auth               2.20.0             pyh1a96a4e_0    conda-forge
google-auth-oauthlib      1.0.0              pyhd8ed1ab_0    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
grpcio                    1.55.1          py310h1b8f574_1    conda-forge
gst-plugins-base          1.22.0               h4243ec0_2    conda-forge
gstreamer                 1.22.0               h25f0c4b_2    conda-forge
gstreamer-orc             0.4.34               hd590300_0    conda-forge
h5py                      3.9.0           nompi_py310h367e799_100    conda-forge
harfbuzz                  6.0.0                h8e241bc_0    conda-forge
hdf5                      1.14.0          nompi_hb72d44e_103    conda-forge
hydra-core                1.2.0              pyhd8ed1ab_0    conda-forge
hypothesis                6.79.3             pyha770c72_0    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
imagecodecs               2023.1.23       py310ha3ed6a1_0    conda-forge
imageio                   2.31.1             pyh24c5eb1_0    conda-forge
imagesize                 1.4.1              pyhd8ed1ab_0    conda-forge
importlib-metadata        6.7.0              pyha770c72_0    conda-forge
importlib_metadata        6.7.0                hd8ed1ab_0    conda-forge
importlib_resources       5.12.0             pyhd8ed1ab_0    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
ipython                   8.14.0             pyh41d4057_0    conda-forge
jack                      1.9.22               h11f4161_0    conda-forge
jedi                      0.18.2             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h0b41bf4_3    conda-forge
jxrlib                    1.1                  h7f98852_2    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4           py310hbf28c38_1    conda-forge
krb5                      1.20.1               h81ceb04_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lazy_loader               0.2                pyhd8ed1ab_0    conda-forge
lcms2                     2.15                 hfd0df8a_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libabseil                 20230125.2      cxx17_h59595ed_2    conda-forge
libaec                    1.0.6                hcb278e6_1    conda-forge
libavif                   0.11.1               h8182462_2    conda-forge
libblas                   3.9.0            16_linux64_mkl    conda-forge
libbrotlicommon           1.0.9                h166bdaf_8    conda-forge
libbrotlidec              1.0.9                h166bdaf_8    conda-forge
libbrotlienc              1.0.9                h166bdaf_8    conda-forge
libcap                    2.67                 he9d0100_0    conda-forge
libcblas                  3.9.0            16_linux64_mkl    conda-forge
libclang                  15.0.7          default_h7634d5b_2    conda-forge
libclang13                15.0.7          default_h9986a30_2    conda-forge
libcublas                 11.11.3.6                     0    nvidia
libcufft                  10.9.0.58                     0    nvidia
libcufile                 1.6.1.9                       0    nvidia
libcups                   2.3.3                h36d4200_3    conda-forge
libcurand                 10.3.2.106                    0    nvidia
libcurl                   8.1.2                h409715c_0    conda-forge
libcusolver               11.4.1.48                     0    nvidia
libcusparse               11.7.5.86                     0    nvidia
libdb                     6.2.32               h9c3ff4c_0    conda-forge
libdeflate                1.17                 h0b41bf4_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               h28343ad_4    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libflac                   1.4.3                h59595ed_0    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgcrypt                 1.10.1               h166bdaf_0    conda-forge
libgfortran-ng            13.1.0               h69a702a_0    conda-forge
libgfortran5              13.1.0               h15d22d2_0    conda-forge
libglib                   2.76.3               hebfc3b9_0    conda-forge
libgpg-error              1.47                 h71f35ed_0    conda-forge
libgrpc                   1.55.1               h59456c1_1    conda-forge
libhwloc                  2.9.1                hd6dc26d_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
liblapack                 3.9.0            16_linux64_mkl    conda-forge
libllvm14                 14.0.6               hcd5def8_3    conda-forge
libllvm15                 15.0.7               hadd5161_1    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnpp                    11.8.0.86                     0    nvidia
libnsl                    2.0.0                h7f98852_0    conda-forge
libnvjpeg                 11.9.0.86                     0    nvidia
libogg                    1.3.4                h7f98852_1    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     15.3                 hbcd7760_1    conda-forge
libprotobuf               4.23.2               hd1fb520_5    conda-forge
libsndfile                1.2.0                hb75c966_0    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libsystemd0               253                  h8c4010b_1    conda-forge
libtiff                   4.5.0                h6adf6a1_2    conda-forge
libtool                   2.4.7                h27087fc_0    conda-forge
libudev1                  253                  h0b41bf4_1    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp-base              1.3.0                h0b41bf4_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxkbcommon              1.5.0                h79f4944_1    conda-forge
libxml2                   2.10.3               hca2bb57_4    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
libzopfli                 1.0.3                h9c3ff4c_0    conda-forge
lightning-utilities       0.8.0                    pypi_0    pypi
line_profiler             4.0.3           py310hd41b1e2_0    conda-forge
llvm-openmp               16.0.6               h4dfa4b3_0    conda-forge
llvmlite                  0.40.0          py310h1b8f574_0    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
markdown                  3.4.3              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.3           py310h2372a71_0    conda-forge
mat73                     0.60                     pypi_0    pypi
matplotlib                3.7.1           py310hff52083_0    conda-forge
matplotlib-base           3.7.1           py310he60537e_0    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mkl                       2022.2.1         h84fe81f_16997    conda-forge
mpc                       1.3.1                hfe3b2da_0    conda-forge
mpfr                      4.2.0                hb012696_0    conda-forge
mpg123                    1.31.3               hcb278e6_0    conda-forge
mpmath                    1.3.0              pyhd8ed1ab_0    conda-forge
multidict                 6.0.4           py310h1fa729e_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mypy-extensions           1.0.0                    pypi_0    pypi
mysql-common              8.0.33               hf1915f5_0    conda-forge
mysql-libs                8.0.33               hca2cd23_0    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
networkx                  3.1                pyhd8ed1ab_0    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.89                 he45b914_0    conda-forge
numba                     0.57.0          py310h0f6aa51_2    conda-forge
numpy                     1.21.6          py310h45f3432_0    conda-forge
oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
omegaconf                 2.2.3              pyhd8ed1ab_0    conda-forge
opencv-python             4.7.0.72                 pypi_0    pypi
openjpeg                  2.5.0                hfec8fc6_2    conda-forge
openssl                   3.1.1                hd590300_1    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pandas                    2.0.2           py310h7cbd5c2_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
partd                     1.4.0              pyhd8ed1ab_0    conda-forge
pathspec                  0.11.1                   pypi_0    pypi
patsy                     0.5.3              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
peakdetect                1.1                      pypi_0    pypi
pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.4.0           py310h023d228_1    conda-forge
pip                       23.1.2             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
platformdirs              3.8.0              pyhd8ed1ab_0    conda-forge
pluggy                    1.2.0              pyhd8ed1ab_0    conda-forge
ply                       3.11                       py_1    conda-forge
pooch                     1.7.0              pyha770c72_3    conda-forge
prompt-toolkit            3.0.38             pyha770c72_0    conda-forge
prompt_toolkit            3.0.38               hd8ed1ab_0    conda-forge
protobuf                  4.23.2          py310hb875b13_1    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pulseaudio                16.1                 hcb278e6_3    conda-forge
pulseaudio-client         16.1                 h5195f5e_3    conda-forge
pulseaudio-daemon         16.1                 ha8d29e2_3    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydantic                  1.10.9          py310h2372a71_0    conda-forge
pydata-sphinx-theme       0.13.3             pyhd8ed1ab_0    conda-forge
pygments                  2.15.1             pyhd8ed1ab_0    conda-forge
pyjwt                     2.7.0              pyhd8ed1ab_0    conda-forge
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pyparsing                 3.1.0              pyhd8ed1ab_0    conda-forge
pyqt                      5.15.7          py310hab646b1_3    conda-forge
pyqt5-sip                 12.11.0         py310heca2aa9_3    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pytest                    7.4.0              pyhd8ed1ab_0    conda-forge
pytest-xdist              3.3.1              pyhd8ed1ab_0    conda-forge
python                    3.10.12         hd12c33a_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python_abi                3.10                    3_cp310    conda-forge
pytorch                   2.0.0           py3.10_cuda11.8_cudnn8.7.0_0    pytorch
pytorch-cuda              11.8                 h7e8668a_5    pytorch
pytorch-lightning         1.8.6                    pypi_0    pypi
pytorch-mutex             1.0                        cuda    pytorch
pytz                      2023.3             pyhd8ed1ab_0    conda-forge
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
pywavelets                1.4.1           py310h0a54255_0    conda-forge
pyyaml                    6.0             py310h5764c6d_5    conda-forge
qt-main                   5.15.8               h5d23da1_6    conda-forge
re2                       2023.03.02           h8c504da_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
recommonmark              0.7.1              pyhd8ed1ab_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.1              pyhd8ed1ab_0    conda-forge
rsa                       4.9                pyhd8ed1ab_0    conda-forge
scikit-image              0.20.0          py310h9b08913_1    conda-forge
scikit-learn              1.2.2           py310hf7d194e_2    conda-forge
scipy                     1.11.0          py310ha4c1d20_0    conda-forge
seaborn                   0.12.2               hd8ed1ab_0    conda-forge
seaborn-base              0.12.2             pyhd8ed1ab_0    conda-forge
setuptools                68.0.0             pyhd8ed1ab_0    conda-forge
sip                       6.7.9           py310hc6cd4ac_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     3.0.5              pyh44b312d_0    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
snowballstemmer           2.2.0              pyhd8ed1ab_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
sphinx                    5.3.0              pyhd8ed1ab_0    conda-forge
sphinx-autodoc-typehints  1.21.8             pyhd8ed1ab_0    conda-forge
sphinx-markdown-tables    0.0.17                   pypi_0    pypi
sphinxcontrib-applehelp   1.0.4              pyhd8ed1ab_0    conda-forge
sphinxcontrib-devhelp     1.0.2                      py_0    conda-forge
sphinxcontrib-htmlhelp    2.0.1              pyhd8ed1ab_0    conda-forge
sphinxcontrib-jsmath      1.0.1                      py_0    conda-forge
sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge
sphinxcontrib-serializinghtml 1.1.5              pyhd8ed1ab_2    conda-forge
spline                    0.11.1dev0      np121py310h93a0a19_1    haydnspass/label/dev
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
statsmodels               0.14.0          py310h278f3c1_1    conda-forge
structlog                 23.1.0             pyhd8ed1ab_0    conda-forge
sympy                     1.12            pypyh9d50eac_103    conda-forge
tbb                       2021.9.0             hf52228f_0    conda-forge
tensorboard               2.13.0             pyhd8ed1ab_0    conda-forge
tensorboard-data-server   0.7.0           py310h34c0648_0    conda-forge
tensorboardx              2.6.1                    pypi_0    pypi
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tifffile                  2023.4.12          pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
torchmetrics              0.11.0                   pypi_0    pypi
torchtriton               2.0.0                     py310    pytorch
tornado                   6.3.2           py310h2372a71_0    conda-forge
tqdm                      4.65.0             pyhd8ed1ab_1    conda-forge
traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.6.3                hd8ed1ab_0    conda-forge
typing_extensions         4.6.3              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
unicodedata2              15.0.0          py310h5764c6d_0    conda-forge
urllib3                   1.26.15            pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
werkzeug                  2.3.6              pyhd8ed1ab_0    conda-forge
wget                      3.2                      pypi_0    pypi
wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
wrapt                     1.15.0          py310h1fa729e_0    conda-forge
xcb-util                  0.4.0                h516909a_0    conda-forge
xcb-util-image            0.4.0                h166bdaf_0    conda-forge
xcb-util-keysyms          0.4.0                h516909a_0    conda-forge
xcb-util-renderutil       0.3.9                h166bdaf_0    conda-forge
xcb-util-wm               0.4.1                h516909a_0    conda-forge
xkeyboard-config          2.38                 h0b41bf4_0    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.4                h0b41bf4_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yarl                      1.9.2           py310h2372a71_0    conda-forge
zernike                   0.0.32                   pypi_0    pypi
zfp                       1.0.0                h27087fc_3    conda-forge
zipp                      3.15.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zlib-ng                   2.0.7                h0b41bf4_0    conda-forge
zstd                      1.5.2                h3eb15da_6    conda-forge
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f6d425ba5c94246fbf4ce90eb4b26ee586e99763.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f6d425ba5c94246fbf4ce90eb4b26ee586e99763" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f6d425ba5c94246fbf4ce90eb4b26ee586e99763_2_676x499.jpeg" alt="image" data-base62-sha1="zdxYQrvQS6ZHns6VpRAC2673yQH" width="676" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f6d425ba5c94246fbf4ce90eb4b26ee586e99763_2_676x499.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f6d425ba5c94246fbf4ce90eb4b26ee586e99763_2_1014x748.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f6d425ba5c94246fbf4ce90eb4b26ee586e99763_2_1352x998.jpeg 2x" data-dominant-color="DFE4E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1420 386 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Haydnspass: <p>“Fixed” by <a href="https://github.com/pytorch/builder/pull/1389" class="inline-onebox" rel="noopener nofollow ugc">Fix processor affinity for fork child by jjyyxx · Pull Request #1389 · pytorch/builder · GitHub</a></p>