DataLoader Multiprocessing error: can't pickle odict_keys objects when num_workers > 0

zwacke: <p>I’m using windows10 64-bit, python 3.7.3 in Jupyter Notebook(anaconda) environment, intel i9-7980XE:</p>
<p>When I try to enumerate over the DataLoader() object with num_workers &gt; 0 like:</p>
<pre><code class="lang-auto">&gt; if __name__=='__main__':
&gt;     ...    
&gt;     DL = DataLoader(data, batch_size=8, shuffle=True, num_workers=8)
&gt;     for i in enumerate(DL ):
&gt;         print(i)
</code></pre>
<p>I get the following error (also when using next(iter(DL )):</p>
<p>TypeError: can’t pickle odict_keys objects</p>
<p>Full Error:</p>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-19-ebb16a21aa89&gt; in &lt;module&gt;
  ...
----&gt; 6     for i in enumerate(DL ):
...

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in __iter__(self)
    817 
    818     def __iter__(self):
--&gt; 819         return _DataLoaderIter(self)
    820 
    821     def __len__(self):

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in __init__(self, loader)
    558                 #     before it starts, and __del__ tries to join but will get:
    559                 #     AssertionError: can only join a started process.
--&gt; 560                 w.start()
    561                 self.index_queues.append(index_queue)
    562                 self.workers.append(w)

C:\ProgramData\Anaconda3\lib\multiprocessing\process.py in start(self)
    110                'daemonic processes are not allowed to have children'
    111         _cleanup()
--&gt; 112         self._popen = self._Popen(self)
    113         self._sentinel = self._popen.sentinel
    114         # Avoid a refcycle if the target function holds an indirect

C:\ProgramData\Anaconda3\lib\multiprocessing\context.py in _Popen(process_obj)
    221     @staticmethod
    222     def _Popen(process_obj):
--&gt; 223         return _default_context.get_context().Process._Popen(process_obj)
    224 
    225 class DefaultContext(BaseContext):

C:\ProgramData\Anaconda3\lib\multiprocessing\context.py in _Popen(process_obj)
    320         def _Popen(process_obj):
    321             from .popen_spawn_win32 import Popen
--&gt; 322             return Popen(process_obj)
    323 
    324     class SpawnContext(BaseContext):

C:\ProgramData\Anaconda3\lib\multiprocessing\popen_spawn_win32.py in __init__(self, process_obj)
     87             try:
     88                 reduction.dump(prep_data, to_child)
---&gt; 89                 reduction.dump(process_obj, to_child)
     90             finally:
     91                 set_spawning_popen(None)

C:\ProgramData\Anaconda3\lib\multiprocessing\reduction.py in dump(obj, file, protocol)
     58 def dump(obj, file, protocol=None):
     59     '''Replacement for pickle.dump() using ForkingPickler.'''
---&gt; 60     ForkingPickler(file, protocol).dump(obj)
     61 
     62 #

TypeError: can't pickle odict_keys objects
</code></pre>

peterjc123: <p>Well, you could not use multiprocessing in interactive shell. Please try using them in normal shell.</p>

zwacke: <p>I tried running in a regular .py script from powershell, resulting in the same error, just with additional extension:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

</code></pre>

zwacke: <p>I eventually solved my problem and i’ll leave the solution here so hopefully someone else will be spared the pain.</p>
<p>It had nothing to do with python version or interactive shells. I tried different environments, none made it work. The error was related to pickling/dictionaries/windows/python.</p>
<p>My pytorch data(torch.utils.data.Dataset) object was abstracting a classification dataset gained from an xml. The pipeline was .xml to data_dict = dict{‘classnames’: dict{‘example 1’: img_path, …} …} to pytorch dataset with list of all elements. In the pytorch data class I gathered all classnames to access as attribute by:</p>
<p><code>self.classes = data_dict.keys()</code></p>
<p>which caused the error because the data_dict.keys() was only a shallow copy of the pointer towards the keys listed in the class where I use ElementTree to extract the dict out of the .xml ! I could resolve the issue by assigning seperate memory:</p>
<p><code>self.classes = list(data_dict.keys())</code></p>
<p>Note, that dicts and odicts are not in general troublesome. assigning a dict as attribute did not cause the error.</p>

lander1003: <p>can you explain more clearly,I am new in pytorch, and I meet the same questions with you, I have no idea where to add the sentence you have said"self.classes=list(data_dict.keys())",thank you</p>

zwacke: <p>My problem was that my class assignment was merely a pointer to a pointer that pointed into a file, i.e. the values for my classes were directly read from the disk memory. That caused the pickling to fail. In python, often a variable assignment from another variable is only a new pointer to the same memory, i.e. no new memory is assigned. In my case I had to wrap the iterable dict.keys() in a list. While both return exactly the same values, the list() constructor assigns new memory in RAM.</p>
<p>It might be that you have a similar problem in your pipeline if you read from a csv, xml, json or whatever. Make sure that in your code at one point you make a deep copy of whatever values you read in so that the variables for pickling do not point into the hard disk memory space but in RAM.</p>

lander1003: <p>Do you have the correct code in github?</p>

111194: <p>excuse me , I meet the same questions with you , but because I’m new in python and pytorch, I really can’t understand how to solve it , can you help me?</p>

111194: <p>my code in github is <a href="https://github.com/WuJie1010/Facial-Expression-Recognition.Pytorch" rel="nofollow noopener">https://github.com/WuJie1010/Facial-Expression-Recognition.Pytorch</a>, when I run mainpro_FER I meet the same question with you , I’d really appreciate it if you can help me, Thank you !!!</p>

zwacke: <p>your problem could be in your file fer.py, in these lines:</p>
<pre><code class="lang-auto"> self.data = h5py.File('./data/data.h5', 'r', driver='core')
 if self.split == 'Training':
            self.train_data = self.data['Training_pixel']
            self.train_labels = self.data['Training_label']
</code></pre>
<p>self.train_data points into the .h5 file. Try wrapping it with a list constructor, or whatever data type is appropriate, to create a deep copy instead of a shallow reference:</p>
<pre><code class="lang-auto"> self.data = h5py.File('./data/data.h5', 'r', driver='core')
 if self.split == 'Training':
            self.train_data = list(self.data['Training_pixel'])
            self.train_labels = list(self.data['Training_label'])
</code></pre>
<p>Apply the same for your other datasets.</p>

111194: <p>okok , thank you very much ,I’m going to have a try,thank you thank you</p>

111194: <p>But it still doesn’t work</p>

mars: <p>did you solve the problem?</p>

yptheangel: <aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/vision/issues/689" target="_blank" rel="nofollow noopener">github.com/pytorch/vision</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/vision/issues/689" target="_blank" rel="nofollow noopener">TypeError: can't pickle Environment objects when num_workers &gt; 0 for LSUN</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-12-17" data-time="15:13:01" data-timezone="UTC">03:13PM - 17 Dec 18 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/ArtjomUEA" target="_blank" rel="nofollow noopener">
          <img alt="ArtjomUEA" src="https://avatars3.githubusercontent.com/u/45484357?v=4" class="onebox-avatar-inline" width="20" height="20">
          ArtjomUEA
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">The program fails to create an iterator for a DataLoader object when the used dataset is LSUN and the amount of...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: datasets</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">wontfix</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>use linux instead? if using windows, num_workers=0</p>

Howl: <p>In my case, error said ‘h5py object cannnot be pickled’. And I think the reason is that python will copy all member variable of class when use multiprocess, while h5py object cannot be copy for different process.<br>
So if you set num_workers=0, it will avoid this error. And my solution is avoiding use h5py object as a member variable.</p>
<p>By the way, the true reason of using <code>self.classes = list(data_dict.keys())</code> maybe that the type of <code>data_dict.keys()</code> in python3 is <code>&lt;class 'dict_keys'&gt;</code>, which cannot be pickled either.</p>

ctpn: <p>This worked for me, thanks!</p>