EOFError: Ran out of input when enumerating the Train Loader

username: jashanvir
cooked: <p>I am getting an error when I enumerate my custom data set. My Dataset has 13 pickle files which I load and then processing it using my custom build Dataset class. However when i tried to enumerate my dataset I am ran out of input.<br>
Here is the trace back of the error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train_2.py", line 137, in &lt;module&gt;
    train(model, device,criterion, trainLoader, optimizer, epoch,losses)
  File "train_2.py", line 33, in train
    for batchIdx, (data, target) in enumerate(trainLoader):
  File "C:\Users\user_name\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 501, in __iter__
__mp_main__
    return _DataLoaderIter(self)
  File "C:\Users\user_name\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 289, in __init__
    w.start()
  File "C:\Users\user_name\AppData\Local\Continuum\anaconda3\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\user_name\AppData\Local\Continuum\anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\user_name\AppData\Local\Continuum\anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\user_name\AppData\Local\Continuum\anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\user_name\AppData\Local\Continuum\anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
OSError: [Errno 22] Invalid argument
C:\Users\user_name\AppData\Local\Continuum\anaconda3\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "C:\Users\user_name\AppData\Local\Continuum\anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\user_namer\AppData\Local\Continuum\anaconda3\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
</code></pre>
<p>The code for the data loading is the following:-<br>
Here my data set is divided into 13 pickle files.</p>
<pre><code class="lang-auto">class PATCHLABEL(data.Dataset):
    base_folder = 'data_batchs'
    train_list = [
            ['data_batch_1'],
            ['data_batch_2'],
            ['data_batch_3'],
            ['data_batch_4'],
            ['data_batch_5'],
            ['data_batch_6'],
            ['data_batch_7'],
            ['data_batch_8'],
            ['data_batch_9'],
            ['data_batch_10'],
            ['data_batch_11'],
            ['data_batch_12'],
            ['data_batch_13'],
        ]
    test_list = [
            ['test_batch'],
        ]
    patch_dim = 1299 
    train_size = 37922*13#492986
    test_size = 31468
    @property
    def targets(self):
        if self.train:
            return self.train_labels
        else:
            return self.test_labels

    def __init__(self, root, train=True,
                 transform=None, target_transform=None):
        self.root = os.path.expanduser(root)
        self.transform = transform
        self.target_transform = target_transform
        self.train = train  # training set or test set

        if self.train:
            self.train_data = []
            self.train_labels = []
            for fentry in self.train_list:
                f = fentry[0]
                file = os.path.join(self.root, self.base_folder, f)
                fo = open(file, 'rb')
                entry = pickle.load(fo,encoding='latin1')
                xTrain = entry[:,:self.patch_dim] #Extracting Images
                yTrain = entry[:,self.patch_dim:] #Extracting Labels
                self.train_data.append(xTrain[:,3:])
                self.train_labels += list(yTrain[:,2])
                fo.close()
            self.train_data = np.concatenate(self.train_data)
            self.train_data = self.train_data.reshape((self.train_size, 36, 36))

        else:
            f = self.test_list[0][0]
            file = os.path.join(self.root, self.base_folder, f)
            fo = open(file, 'rb')
            entry = pickle.load(fo,encoding='latin1')

            xTrain = entry[:,:self.patch_dim] #Extracting Images
            yTrain = entry[:,self.patch_dim:] #Extracting Labels
            self.test_data = xTrain[:,3:]
            self.test_labels = list(yTrain[:,2])
            fo.close()
            self.test_data = self.test_data.reshape((self.test_size,36, 36))


    def __getitem__(self, index):
        """
        Args:
            index (int): Index
        Returns:
            tuple: (image, target) where target is index of the target class.
        """
        if self.train:
            patch, target = self.train_data[index], self.train_labels[index]
        else:
            patch, target = self.test_data[index], self.test_labels[index]

        if self.transform is not None:
            patch = self.transform(patch)
        if self.target_transform is not None:
            target = self.target_transform(target)

        return patch, target

    def __len__(self):
        if self.train:
            return len(self.train_data)
        else:
            return len(self.test_data)
</code></pre>
<p>I am using PyTorch version 0.4.1 and Windows 10.<br>
Any suggestions on how i can solve this error ?<br>
Thanks</p>

username: Deeply
cooked: <p>I think the cause of the problem is this line<br>
<code>entry = pickle.load(fo,encoding='latin1')</code></p>

username: jashanvir
cooked: <p>Thank you for the response. However, I am getting the same error even with encoding. Plus, the code ran until the data loading. Only when i enumerate and get a batch values, i am getting the error.</p>

username: peterjc123
cooked: <p>Have you used <code>if __name__ == '__main__':</code> to protect your main function? More details <a href="https://pytorch.org/docs/master/notes/windows.html#multiprocessing-error-without-if-clause-protection" rel="nofollow noopener">here</a>. You can also try to run the script in command prompt. If the error persists, then you can try to set num_worker of DataLoader to zero.</p>

username: jashanvir
cooked: <p>Thanks. Yes i used <code>if __name__ =='__main__':</code> and also I tried running my script on command prompt too but same problem. However by changing num_worker of DataLoader to zero solves the problem. But I am still not sure why changing it any value other than 0 makes a difference? Is there anything wrong in my code?</p>

username: peterjc123
cooked: <p><a class="mention" href="/u/jashanvir">@jashanvir</a> Could you give me a more simple and small code snippet so that I can reproduce your issue?</p>

username: askkzk
cooked: <p>I got a same error and solved by this num_worker=0.<br>
Thank you for your help!!!</p>

username: peterjc123
cooked: <p>It seems that it’s the same problem with this one: <a href="https://discuss.pytorch.org/t/pytorch-windows-eoferror-ran-out-of-input-when-num-workers-0/25918/2">Pytorch Windows EOFError: Ran out of input when num_workers&gt;0</a>. The input exceeds the limit of Pickle (4GB). We’ll have to use pickle version 4 to solve this.</p>

username: Torki
cooked: <p>I use pickle version 4.0 but the problem persists again.<br>
my problem occurs just when i try to run my code on windows platform with num_worker&gt;0 and when i cover my loader loop with<br>
if <strong>name</strong> == “<strong>main</strong>”:<br>
I don’t want to set num_worker=0 because of slow performance in 0 worker.</p>

username: sumeyyekale
cooked: <p>I have the same problem. How did this problem get solved?</p>