Python multiprocessing struct.error: 'i' format requires -2147483648 <= number <= 2147483647

Pytorch_Text_User: <p>I have a huge text data and at certain amount of data (usually more than 700MB), I keep getting this message error.</p>
<p>Cpython…<br>
Python multiprocessing struct.error: ‘i’ format requires -2147483648 &lt;= number &lt;= 2147483647</p>
<p>But I don’t use any of that and it’s probably because of pytorch features such as Dataset and so on.</p>
<p>Any idea how to solve this?</p>

tom: <p>Yeah, well, usually you should get a backtrace that helps pinpointing where that might be happening.</p>
<p>Best regards</p>
<p>Thomas</p>

Pytorch_Text_User: <p>Thanks for response, message is like this.</p>
<p>File “/app/train.py”, line 48, in train<br>
for dataset in train_data_loader:<br>
File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 631, in <strong>next</strong><br>
idx, batch = self._get_batch()<br>
File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 610, in _get_batch<br>
return self.data_queue.get()<br>
File “/opt/conda/lib/python3.6/multiprocessing/queues.py”, line 94, in get<br>
res = self._recv_bytes()<br>
File “/opt/conda/lib/python3.6/multiprocessing/connection.py”, line 216, in recv_bytes<br>
buf = self._recv_bytes(maxlength)<br>
File “/opt/conda/lib/python3.6/multiprocessing/connection.py”, line 407, in _recv_bytes<br>
buf = self._recv(4)<br>
File “/opt/conda/lib/python3.6/multiprocessing/connection.py”, line 379, in _recv<br>
chunk = read(handle, remaining)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 274, in handler<br>
_error_if_any_worker_fails()<br>
RuntimeError: DataLoader worker (pid 92) is killed by signal: Killed.<br>
User session exited</p>
<p>Best regards<br>
Ryan</p>

tom: <p>That isn’t the backtrace or error message for the initial error but one that would appear to be the one on the other end after the initial error happened or so.</p>
<p>At any rate, the original message seems to be a bug of Python &lt; 3.8 handling large objects badly when it communicates between the processing:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/python/cpython/blob/master/Lib/multiprocessing/connection.py#L392" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/python/cpython/blob/master/Lib/multiprocessing/connection.py#L392" target="_blank" rel="nofollow noopener">python/cpython/blob/master/Lib/multiprocessing/connection.py#L392</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="382" style="counter-reset: li-counter 381 ;">
<li>            if remaining == size:</li>
<li>                raise EOFError</li>
<li>            else:</li>
<li>                raise OSError("got end of file during message")</li>
<li>        buf.write(chunk)</li>
<li>        remaining -= n</li>
<li>    return buf</li>
<li>
</li>
<li>def _send_bytes(self, buf):</li>
<li>    n = len(buf)</li>
<li class="selected">    if n &gt; 0x7fffffff:</li>
<li>        pre_header = struct.pack("!i", -1)</li>
<li>        header = struct.pack("!Q", n)</li>
<li>        self._send(pre_header)</li>
<li>        self._send(header)</li>
<li>        self._send(buf)</li>
<li>    else:</li>
<li>        # For wire compatibility with 3.7 and lower</li>
<li>        header = struct.pack("!i", n)</li>
<li>        if n &gt; 16384:</li>
<li>            # The payload is large so Nagle's algorithm won't be triggered</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Of course, that you do things send that large, chances are that something is amiss with what your program does for multiprocessing. Is something (i.e. another library) keeping you from using <a href="https://pytorch.org/docs/stable/multiprocessing.html" rel="nofollow noopener">PyTorch’s multiprocessing wrapper</a>?</p>
<p>Best regards</p>
<p>Thomas</p>

Pytorch_Text_User: <p>I think I found issues based on your comment.</p>
<p>from torch.utils.data import DataLoader</p>
<p>train_data_loader = DataLoader(train_data_meta,<br>
shuffle=False, collate_fn=lambda x: x[0],<br>
num_workers=config.Req[“num_workers”], drop_last=True)</p>
<p>Yes, it’s about multiprocessing and if num_workers &gt; 1 and data is large, the program is killed or error occurs because of that.</p>
<p>I use python 3.5~3.6 and I will probably think to change my python version higher.</p>