Mp.spawn breaks testing?

Dolores_Garcia: <p>I am training Pointcept, a well know repo, with one of their examples:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/Pointcept/Pointcept">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/Pointcept/Pointcept" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/e5f99e3e2071deeadcfad9050185d6049d505297275c1d41bebe25bc704a4c99/Pointcept/Pointcept" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/Pointcept/Pointcept" target="_blank" rel="noopener nofollow ugc">GitHub - Pointcept/Pointcept: Pointcept: a codebase for point cloud...</a></h3>

  <p>Pointcept: a codebase for point cloud perception research. Latest works: PPT, MSC (CVPR'23), PTv2 (NeurIPS'22) - GitHub - Pointcept/Pointcept: Pointcept: a codebase for point cloud percepti...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
The model runs an trains in 4 A100 gpus but when the evaluation starts there is a weird bug (below).<br>
This is hard to debug as using 1 gpu gives an out of memory error. Evaluation happens inside of the same distributed process so it’s not cleat why it’s breaking.</p>
<hr>
<p>[2023-10-25 17:48:30,221 INFO evaluator.py line 112 31130] &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; Start Evaluation &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/<strong>init</strong>.py”, line 16, in <br>
from . import context<br>
ImportError: cannot import name ‘context’ from partially initialized module ‘multiprocessing’ (most likely due to a circular import) (/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/<strong>init</strong>.py)<br>
Exception in thread Thread-3:<br>
Traceback (most recent call last):<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/threading.py”, line 932, in _bootstrap_inner<br>
self.run()<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/site-packages/tensorboardX/event_file_writer.py”, line 202, in run<br>
data = self._queue.get(True, queue_wait_duration)<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/queues.py”, line 111, in get<br>
res = self._recv_bytes()<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/connection.py”, line 216, in recv_bytes<br>
buf = self._recv_bytes(maxlength)<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/connection.py”, line 414, in _recv_bytes<br>
buf = self._recv(4)<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/connection.py”, line 383, in _recv<br>
raise EOFError<br>
EOFError<br>
wandb: Waiting for W&amp;B process to finish… (failed 1). Press Control-C to abort syncing.<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/spawn.py”, line 116, in spawn_main<br>
exitcode = _main(fd, parent_sentinel)<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/spawn.py”, line 125, in _main<br>
prepare(preparation_data)<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/spawn.py”, line 236, in prepare<br>
_fixup_main_from_path(data[‘init_main_from_path’])<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/spawn.py”, line 287, in _fixup_main_from_path<br>
main_content = runpy.run_path(main_path,<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/runpy.py”, line 265, in run_path<br>
return _run_module_code(code, init_globals, run_name,<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/runpy.py”, line 97, in _run_module_code<br>
_run_code(code, mod_globals, init_globals,<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “/afs/cern.ch/work/m/mgarciam/private/UD_pointcept/exp/s3dis/semseg-pt-v2m2-0-base_a2/code/tools/train.py”, line 8, in <br>
from pointcept.engines.defaults import (<br>
File “/afs/cern.ch/work/m/mgarciam/private/UD_pointcept/exp/s3dis/semseg-pt-v2m2-0-base_a2/code/pointcept/engines/defaults.py”, line 14, in <br>
from torch.nn.parallel import DistributedDataParallel<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/site-packages/torch/<strong>init</strong>.py”, line 16, in <br>
import ctypes<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/ctypes/<strong>init</strong>.py”, line 545, in <br>
from ctypes._endian import BigEndianStructure, LittleEndianStructure<br>
ModuleNotFoundError: No module named ‘ctypes._endian’<br>
Traceback (most recent call last):aded (0.000 MB deduped)<br>
File “”, line 1, in <br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/spawn.py”, line 116, in spawn_main<br>
exitcode = _main(fd, parent_sentinel)<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/spawn.py”, line 125, in _main<br>
prepare(preparation_data)<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/spawn.py”, line 236, in prepare<br>
_fixup_main_from_path(data[‘init_main_from_path’])<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/spawn.py”, line 287, in _fixup_main_from_path<br>
main_content = runpy.run_path(main_path,<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/runpy.py”, line 265, in run_path<br>
return _run_module_code(code, init_globals, run_name,<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/runpy.py”, line 97, in _run_module_code<br>
_run_code(code, mod_globals, init_globals,<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “/afs/cern.ch/work/m/mgarciam/private/UD_pointcept/exp/s3dis/semseg-pt-v2m2-0-base_a2/code/tools/train.py”, line 8, in <br>
from pointcept.engines.defaults import (<br>
File “/afs/cern.ch/work/m/mgarciam/private/UD_pointcept/exp/s3dis/semseg-pt-v2m2-0-base_a2/code/pointcept/engines/defaults.py”, line 14, in <br>
from torch.nn.parallel import DistributedDataParallel<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/site-packages/torch/<strong>init</strong>.py”, line 811, in <br>
from .functional import *  # noqa: F403<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/site-packages/torch/functional.py”, line 7, in <br>
import torch.nn.functional as F<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/site-packages/torch/nn/<strong>init</strong>.py”, line 3, in <br>
from .parallel import DataParallel<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/site-packages/torch/nn/parallel/<strong>init</strong>.py”, line 1, in <br>
from .parallel_apply import parallel_apply<br>
ModuleNotFoundError: No module named ‘torch.nn.parallel.parallel_apply’<br>
Traceback (most recent call last):aded (0.000 MB deduped)<br>
File “exp/s3dis/semseg-pt-v2m2-0-base_a2/code/tools/train.py”, line 39, in <br>
main()<br>
File “exp/s3dis/semseg-pt-v2m2-0-base_a2/code/tools/train.py”, line 27, in main<br>
launch(<br>
File “/afs/cern.ch/work/m/mgarciam/private/UD_pointcept/exp/s3dis/semseg-pt-v2m2-0-base_a2/code/pointcept/engines/launch.py”, line 75, in launch<br>
mp.spawn(<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py”, line 240, in spawn<br>
return start_processes(fn, args, nprocs, join, daemon, start_method=‘spawn’)<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py”, line 198, in start_processes<br>
while not context.join():<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py”, line 160, in join<br>
raise ProcessRaisedException(msg, error_index, failed_process.pid)<br>
torch.multiprocessing.spawn.ProcessRaisedException:</p>
<p>– Process 1 terminated with the following error:<br>
Traceback (most recent call last):<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py”, line 69, in _wrap<br>
fn(i, *args)<br>
File “/afs/cern.ch/work/m/mgarciam/private/UD_pointcept/exp/s3dis/semseg-pt-v2m2-0-base_a2/code/pointcept/engines/launch.py”, line 145, in _distributed_worker<br>
main_func(*cfg)<br>
File “/afs/cern.ch/work/m/mgarciam/private/UD_pointcept/exp/s3dis/semseg-pt-v2m2-0-base_a2/code/tools/train.py”, line 20, in main_worker<br>
trainer.train()<br>
File “/afs/cern.ch/work/m/mgarciam/private/UD_pointcept/exp/s3dis/semseg-pt-v2m2-0-base_a2/code/pointcept/engines/train.py”, line 168, in train<br>
self.after_epoch()<br>
File “/afs/cern.ch/work/m/mgarciam/private/UD_pointcept/exp/s3dis/semseg-pt-v2m2-0-base_a2/code/pointcept/engines/train.py”, line 101, in after_epoch<br>
h.after_epoch()<br>
File “/afs/cern.ch/work/m/mgarciam/private/UD_pointcept/exp/s3dis/semseg-pt-v2m2-0-base_a2/code/pointcept/engines/hooks/evaluator.py”, line 109, in after_epoch<br>
self.eval()<br>
File “/afs/cern.ch/work/m/mgarciam/private/UD_pointcept/exp/s3dis/semseg-pt-v2m2-0-base_a2/code/pointcept/engines/hooks/evaluator.py”, line 114, in eval<br>
for i, input_dict in enumerate(self.trainer.val_loader):<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 681, in <strong>next</strong><br>
data = self._next_data()<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 1348, in _next_data<br>
self._shutdown_workers()<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 1474, in _shutdown_workers<br>
w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL)<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/process.py”, line 149, in join<br>
res = self._popen.wait(timeout)<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/popen_fork.py”, line 44, in wait<br>
if not wait([self.sentinel], timeout):<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/multiprocessing/connection.py”, line 931, in wait<br>
ready = selector.select(timeout)<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/selectors.py”, line 415, in select<br>
fd_event_list = self._selector.poll(timeout)<br>
File “/eos/user/m/mgarciam/envs/pointcept1/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py”, line 66, in handler<br>
_error_if_any_worker_fails()<br>
RuntimeError: DataLoader worker (pid 32636) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.</p>