Pytorch Windows EOFError: Ran out of input when num_workers>0

MLAI: <p>When trying to do multiprocessing in windows I get the following error.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 159, in &lt;module&gt;
    main()
  File "train.py", line 111, in main
    for i_batch, sample_batched in enumerate(Dataloader_Train):
  File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
    return _DataLoaderIter(self)
  File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
    w.start()
  File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\context.py", line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\context.py", line 313, in _Popen
    return Popen(process_obj)
  File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\popen_spawn_win32.py", line 66, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\reduction.py", line 59, in dump
    ForkingPickler(file, protocol).dump(obj)
OverflowError: cannot serialize a bytes object larger than 4 GiB

(py3_pt3_orig) C:\Users\TestUser\Documents\eliaseulig\DeepDSA&gt;Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\spawn.py", line 106, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\spawn.py", line 116, in _main
    self = pickle.load(from_parent)
EOFError: Ran out of input
</code></pre>
<p>I wrapped the whole code (except package loading) into a main() function. It works perfectly fine when setting num_workers=0, but due to heavy data augmentation this slows down the training and does not let my GPU use its full potential.</p>
<p>Thanks in advance.</p>

ptrblck: <p>What is your batch size and how large is each sample?<br>
I would like to reproduce the error as I’m wondering why a single worker seems to be able to load the data while multiple workers crash.</p>

MLAI: <p>Thank you! Batch size is 32 and each sample is a dict containing two torch.FloatTensors of size (1,256,256) each.</p>

peterjc123: <p>The actually error is <code>OverflowError: cannot serialize a bytes object larger than 4 GiB</code>. You have to reduce the size of the input.</p>

peterjc123: <aside class="quote no-group" data-username="peterjc123" data-post="4" data-topic="25918">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peterjc123/48/2790_2.png" class="avatar"> peterjc123:</div>
<blockquote>
<p>OverflowError: cannot serialize a bytes object larger than 4 GiB</p>
</blockquote>
</aside>
<p>Plus: The related python bug: <a href="https://bugs.python.org/issue17560" rel="noopener nofollow ugc">link</a><br>
However, according to this <a href="https://github.com/stan-dev/pystan/issues/197" rel="noopener nofollow ugc">issue</a>, this one can be solved by using pickle version 4. But it cannot be controlled on our side. It’s actually a Python bug. As the workground, we could implement something like <a href="https://stackoverflow.com/questions/45119053/how-to-change-the-serialization-method-used-by-pythons-multiprocessing" rel="noopener nofollow ugc">this</a> that overrides the default pickle version.</p>

MLAI: <p>Thanks for the reply and sorry for the late answer. Unfortunately neither reducing the batch size, nor reducing the input size (by just making the  images smaller) helps with the problem. How can two 256x256 tensors exceed 4gb of ram? I also tried changing the DEFAULT_PROTOCOL variable in torch.serialization.py to 4 (instead of 2) which seems to manage the pickle version used but without any effect.</p>

MLAI: <p>I solved the issue by changing protocol=None in pythons multiprocessing/reduction.py to 4.</p>
<pre><code class="lang-auto">class ForkingPickler(pickle.Pickler):
    '''Pickler subclass used by multiprocessing.'''
    _extra_reducers = {}
    _copyreg_dispatch_table = copyreg.dispatch_table

    def __init__(self, *args):
        super().__init__(*args)
        self.dispatch_table = self._copyreg_dispatch_table.copy()
        self.dispatch_table.update(self._extra_reducers)

    @classmethod
    def register(cls, type, reduce):
        '''Register a reduce function for a type.'''
        cls._extra_reducers[type] = reduce

    @classmethod
    def dumps(cls, obj, protocol=4):
        buf = io.BytesIO()
        cls(buf, protocol).dump(obj)
        return buf.getbuffer()

    loads = pickle.loads

register = ForkingPickler.register

def dump(obj, file, protocol=4):
    '''Replacement for pickle.dump() using ForkingPickler.'''
    ForkingPickler(file, protocol).dump(obj)
</code></pre>
<p>Thank you for your help!</p>

mars: <p>I got the same problem with data loaded by pickle, is there any solution? thanks.</p>

ptrblck: <p>Are the suggestions in this thread not working?<br>
If so, could you post the complete error message please?</p>

ptrblck: <p>Are you seeing this issue only using your data loading or are you seeing similar issues using random data?<br>
Are you using the if-clause protection as explained <a href="https://pytorch.org/docs/stable/notes/windows.html#usage-multiprocessing">here</a>?</p>

mars: <p>yes, I only saw this issue using my own data loading, my system is ubuntu 16.04,  the if-clause protection work too?</p>

ptrblck: <p>You could add it, but it shouldn’t be a problem. Sorry, I confused your setup with the title of the topic.<br>
Is changing the <code>pickle</code> version or the protocol not working for you?</p>

mars: <p>I tried the method suggested in this post, but resulted in another error.</p>
<pre><code class="lang-auto">_pickle.UnpicklingError: pickle data was truncated
</code></pre>

ptrblck: <p>This error might point to a corrupt file.<br>
Could you download it again or recreate it?</p>

mars: <p>Ok, i will try, thanks</p>

mars: <p>I solved the issue by loading the json in the class object, not in the main function.</p>

susinder: <p>I’m seeing same problem. Running windows, python 3.7.7, latest pytorch version. Issue seen only with num_workers&gt;0. I tried modifying Anaconda3\envs\test\Lib\multiprocessing\reduction.py with def dump(obj, file, protocol=4, but it didn’t help. Is there a different way to force protocol version or any other workaround. I’m running the code in <a href="https://github.com/seungwonpark/melgan.git" rel="nofollow noopener">https://github.com/seungwonpark/melgan.git</a>.</p>

yuenn: <p>Hi <a class="mention" href="/u/mars">@mars</a>,</p>
<p>I also meet the problem after I set  protocol=4. Could you share your code to work around the problem? Thank you!</p>
<pre><code class="lang-python"> Traceback (most recent call last):
   File "&lt;string&gt;", line 1, in &lt;module&gt;
   File "/opt/conda/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
     exitcode = _main(fd)
   File "/opt/conda/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
     self = reduction.pickle.load(from_parent)
 _pickle.UnpicklingError: pickle data was truncated
 Traceback (most recent call last):
   File "&lt;string&gt;", line 1, in &lt;module&gt;
   File "/opt/conda/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
     exitcode = _main(fd)
   File "/opt/conda/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
     self = reduction.pickle.load(from_parent)
 _pickle.UnpicklingError: pickle data was truncated
 Traceback (most recent call last):
   File "&lt;string&gt;", line 1, in &lt;module&gt;
   File "/opt/conda/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
     exitcode = _main(fd)
   File "/opt/conda/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
     self = reduction.pickle.load(from_parent)
 _pickle.UnpicklingError: pickle data was truncated
 Traceback (most recent call last):
   File "&lt;string&gt;", line 1, in &lt;module&gt;
   File "/opt/conda/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
     exitcode = _main(fd)
   File "/opt/conda/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
     self = reduction.pickle.load(from_parent)
 _pickle.UnpicklingError: pickle data was truncated
 Traceback (most recent call last):
   File "/opt/conda/bin/fairseq-train", line 11, in &lt;module&gt;
     load_entry_point('fairseq', 'console_scripts', 'fairseq-train')()
   File "/code/fairseq/fairseq_cli/train.py", line 370, in cli_main
     nprocs=args.distributed_world_size,
   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
     while not spawn_context.join():
   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 107, in join
     (error_index, name)
 Exception: process 5 terminated with signal SIGKILL
</code></pre>

meifish: <p>Hello!</p>
<p>I am running Pytorch Windows version and encounter the same EOFError that is induced by num_worker &gt; 0. It works when num_worker is exactly 0. I tried <a class="mention" href="/u/mlai">@MLAI</a>’s solution, changing the <code>protocol=4</code> in <code>ForkingPickler</code> class , but it doesn’t work on multiple workers scenario.</p>
<p>I wonder <a class="mention" href="/u/mlai">@MLAI</a> have you tried running on multiple workers?</p>
<p>Thanks!</p>

chaslie: <p>hi,</p>
<p>I was having a similar problem in pycharm &amp; spyder, but after rebuilding the Virtual enviroment it is now working in Spyder, just waiting for the save point and then i’ll try in pycharm.</p>
<p>not sure if this helps, but it maybe worth a shot.</p>
<p>cheers,</p>
<p>chaslie</p>