My Pytorch model stop logging without error

username: NAsrin791
cooked: <p>Hello,<br>
I am training a model called EDVR using pytorch, after some arbitrary iterations(900,6800, etc) it stops logging in the terminal without any error. However, the GPU shows memory usage. the details are below:</p>
<ul>
<li>I am training EDVR model using train.py  from this github: <a href="https://github.com/xinntao/EDVR" rel="nofollow noopener">https://github.com/xinntao/EDVR</a>  and the model dependencies are:</li>
</ul>
<ul>
<li>Python 3</li>
<li>[PyTorch &gt;= 1.1]</li>
<li>NVIDIA GPU + [CUDA]</li>
<li>[Deformable Convolution]</li>
</ul>
<ul>
<li>I use one NVIDIA-SMI  GPU with about 11GiB memory, CUDA version 10.1</li>
<li>Pytorch version 1.3.0</li>
<li>The number of workers is set to 3</li>
<li>The code supports distributed training too using this command for example:<br>
python -m torch.distributed.launch --nproc_per_node=8 --master_port=4321 train.py -opt training-log.yml --launcher pytorch</li>
<li>I have only one GPU so I run it using this command in terminal:<br>
python train.py -opt training-log.yml --launcher none<br>
-I connected to the ubuntu system which I run the cone on using Teamviewer app</li>
</ul>
<p>I will be grateful if you help me to solve this problem.</p>

username: ptrblck
cooked: <p>Could you update to the latest stable release (<code>1.5.0</code>) and rerun the code, please?<br>
Let us know, if your code still hangs.</p>

username: NAsrin791
cooked: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>, I installed pytorch 1.5.0,  when I run train.py  this error occurs: Failed to import DCNv2 module.<br>
to compile this module I need to run the below setup.py script <code>python setup.py develop</code>  to make a cuda extension, however, it fails to compile. Could you please guide me what should I change in this script?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a60a1d24df7ac75b5a9b5df91ad7e7bd8626e0e0.png" data-download-href="https://discuss.pytorch.org/uploads/default/a60a1d24df7ac75b5a9b5df91ad7e7bd8626e0e0" title="Untitled"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a60a1d24df7ac75b5a9b5df91ad7e7bd8626e0e0_2_690x331.png" alt="Untitled" data-base62-sha1="nGQVAI6nP9eEoINVwxuvLAGnrAk" width="690" height="331" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a60a1d24df7ac75b5a9b5df91ad7e7bd8626e0e0_2_690x331.png, https://discuss.pytorch.org/uploads/default/original/3X/a/6/a60a1d24df7ac75b5a9b5df91ad7e7bd8626e0e0.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/6/a60a1d24df7ac75b5a9b5df91ad7e7bd8626e0e0.png 2x" data-dominant-color="2F2F27"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Untitled</span><span class="informations">888×426 147 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>What is the error message you are seeing when you try to build the extension?</p>

username: NAsrin791
cooked: <p>Dear  <a class="mention" href="/u/ptrblck">@ptrblck</a>, this is the error occurred:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/4/04771d8a2e942681fb4b95a722ffc7539df277ec.png" data-download-href="https://discuss.pytorch.org/uploads/default/04771d8a2e942681fb4b95a722ffc7539df277ec" title="1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/4/04771d8a2e942681fb4b95a722ffc7539df277ec.png" alt="1" data-base62-sha1="Dv720UhGNDkQbVssPLcO4W5Zak" width="690" height="405" data-dominant-color="411131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1358×799 96.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The initial error message seems to be raised before the last one.<br>
Could you post the stack trace directly by wrapping it into three backticks ``` please?</p>

username: NAsrin791
cooked: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>, yes, you are right, another error occurred before, here it is:</p>
<pre><code class="lang-auto">running egg_info
creating deform_conv.egg-info
writing deform_conv.egg-info/PKG-INFO
writing dependency_links to deform_conv.egg-info/dependency_links.txt
writing top-level names to deform_conv.egg-info/top_level.txt
writing manifest file 'deform_conv.egg-info/SOURCES.txt'
reading manifest file 'deform_conv.egg-info/SOURCES.txt'
writing manifest file 'deform_conv.egg-info/SOURCES.txt'
running build_ext
building 'deform_conv_cuda' extension
creating /media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/build
creating /media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/build/temp.linux-x86_64-3.7
creating /media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/build/temp.linux-x86_64-3.7/src
Emitting ninja build file /media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/build/temp.linux-x86_64-3.7/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] c++ -MMD -MF /media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/build/temp.linux-x86_64-3.7/src/deform_conv_cuda.o.d -pthread -B /home/ml/miniconda3/envs/nasrin/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include -I/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include/TH -I/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/ml/miniconda3/envs/nasrin/include/python3.7m -c -c /media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda.cpp -o /media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/build/temp.linux-x86_64-3.7/src/deform_conv_cuda.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=deform_conv_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
FAILED: /media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/build/temp.linux-x86_64-3.7/src/deform_conv_cuda.o 
c++ -MMD -MF /media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/build/temp.linux-x86_64-3.7/src/deform_conv_cuda.o.d -pthread -B /home/ml/miniconda3/envs/nasrin/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include -I/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include/TH -I/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/ml/miniconda3/envs/nasrin/include/python3.7m -c -c /media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda.cpp -o /media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/build/temp.linux-x86_64-3.7/src/deform_conv_cuda.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=deform_conv_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda.cpp: In function ‘void shape_check(at::Tensor, at::Tensor, at::Tensor*, at::Tensor, int, int, int, int, int, int, int, int, int, int)’:
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda.cpp:68:31: error: ‘AT_CHECK’ was not declared in this scope
            weight.ndimension());
                               ^
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda.cpp: In function ‘int deform_conv_forward_cuda(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, int)’:
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda.cpp:192:73: error: ‘AT_CHECK’ was not declared in this scope
   AT_CHECK((offset.size(0) == batchSize), "invalid batch size of offset");
                                                                         ^
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda.cpp: In function ‘int deform_conv_backward_input_cuda(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, int)’:
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda.cpp:298:76: error: ‘AT_CHECK’ was not declared in this scope
   AT_CHECK((offset.size(0) == batchSize), 3, "invalid batch size of offset");
                                                                            ^
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda.cpp: In function ‘int deform_conv_backward_parameters_cuda(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, float, int)’:
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda.cpp:413:73: error: ‘AT_CHECK’ was not declared in this scope
   AT_CHECK((offset.size(0) == batchSize), "invalid batch size of offset");
                                                                         ^
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda.cpp: In function ‘void modulated_deform_conv_cuda_forward(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, bool)’:
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda.cpp:493:70: error: ‘AT_CHECK’ was not declared in this scope
   AT_CHECK(input.is_contiguous(), "input tensor has to be contiguous");
                                                                      ^
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda.cpp: In function ‘void modulated_deform_conv_cuda_backward(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, bool)’:
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda.cpp:574:70: error: ‘AT_CHECK’ was not declared in this scope
   AT_CHECK(input.is_contiguous(), "input tensor has to be contiguous");
                                                                      ^
[2/2] /usr/local/cuda/bin/nvcc -I/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include -I/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include/TH -I/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/ml/miniconda3/envs/nasrin/include/python3.7m -c -c /media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda_kernel.cu -o /media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/build/temp.linux-x86_64-3.7/src/deform_conv_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=deform_conv_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -std=c++14
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda_kernel.cu: In lambda function:
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda_kernel.cu:258:247: warning: ‘T* at::Tensor::data() const [with T = double]’ is deprecated: Tensor.data&lt;T&gt;() is deprecated. Please use Tensor.data_ptr&lt;T&gt;() instead. [-Wdeprecated-declarations]
/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:341:1: note: declared here
   T * data() const {
 ^
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda_kernel.cu:258:310: warning: ‘T* at::Tensor::data() const [with T = double]’ is deprecated: Tensor.data&lt;T&gt;() is deprecated. Please use Tensor.data_ptr&lt;T&gt;() instead. [-Wdeprecated-declarations]
/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:341:1: note: declared here
   T * data() const {
 ^
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda_kernel.cu:258:361: warning: ‘T* at::Tensor::data() const [with T = double]’ is deprecated: Tensor.data&lt;T&gt;() is deprecated. Please use Tensor.data_ptr&lt;T&gt;() instead. [-Wdeprecated-declarations]
/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:341:1: note: declared here
   T * data() const {
.
.
.
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda_kernel.cu:845:2264: warning: ‘T* at::Tensor::data() const [with T = c10::Half]’ is deprecated: Tensor.data&lt;T&gt;() is deprecated. Please use Tensor.data_ptr&lt;T&gt;() instead. [-Wdeprecated-declarations]
/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:341:1: note: declared here
   T * data() const {
 ^
/media/ml/datadrive2/Nasrin/EDVR-master/codes/models/archs/dcn/src/deform_conv_cuda_kernel.cu:845:2320: warning: ‘T* at::Tensor::data() const [with T = c10::Half]’ is deprecated: Tensor.data&lt;T&gt;() is deprecated. Please use Tensor.data_ptr&lt;T&gt;() instead. [-Wdeprecated-declarations]
/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:341:1: note: declared here
   T * data() const {
 ^
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1400, in _run_ninja_build
    check=True)
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 22, in &lt;module&gt;
    ], cmdclass={'build_ext': BuildExtension}, zip_safe=False)
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/setuptools/command/develop.py", line 38, in run
    self.install_for_development()
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/setuptools/command/develop.py", line 140, in install_for_development
    self.run_command('build_ext')
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 84, in run
    _build_ext.run(self)
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 580, in build_extensions
    build_ext.build_extensions(self)
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 205, in build_extension
    _build_ext.build_extension(self, ext)
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
    depends=ext.depends)
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 423, in unix_wrap_ninja_compile
    with_cuda=with_cuda)
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1140, in _write_ninja_file_and_compile_objects
    error_prefix='Error compiling objects for extension')
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1413, in _run_ninja_build
    raise RuntimeError(message)
RuntimeError: Error compiling objects for extension
</code></pre>

username: ptrblck
cooked: <p>It seems the error is:</p>
<pre><code class="lang-python">error: ‘AT_CHECK’ was not declared in this scope
</code></pre>
<p>Could you replace the deprecated <code>AT_CHECK</code> calls with <code>TORCH_CHECK</code> and try to rebuild?</p>

username: NAsrin791
cooked: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks a lot, this solved compiling problem.  Now I start to train the model and I hope it won’t hang again.</p>

username: NAsrin791
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
Unfortunately, the logging hung after 63100 iterations this time, I really appreciate any idea to fix it.</p>

username: NAsrin791
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
I couldn’t solve the problem, after stucking when I kill the process using ctrl+c in terminal, I get the below traceback, Does it give information to solve the problem?<br>
Thanks for your help</p>
<pre><code class="lang-auto">  File "train.py", line 311, in &lt;module&gt;
    main()
  File "train.py", line 149, in main
    for _, train_data in enumerate(train_loader):
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
    idx, data = self._get_data()
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
    success, data = self._try_get_data()
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/multiprocessing/queues.py", line 104, in get
    if not self._poll(timeout):
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/multiprocessing/connection.py", line 920, in wait
    ready = selector.select(timeout)
  File "/home/ml/miniconda3/envs/nasrin/lib/python3.7/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt ```</code></pre>

username: ptrblck
cooked: <p>Are you using multiple workers in your <code>DataLoader</code>? If so, could you use <code>num_workers=0</code> and rerun the script, please?<br>
Also, I would recommend to update to the latest stable PyTorch release (<code>1.5</code>).</p>

username: triple-tam
cooked: <p>This worked for me! Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: ptrblck
cooked: <p>Good to hear this workaround is working. However, note that this is not the solution, but just a step to isolate the issue further. If <code>num_workers&gt;0</code> is raising an error, this could point towards an issue with multiprocessing (or how it’s used inside the script).</p>