Dataloader iteration hang up

username: dieuroi
cooked: <p>I want to train a model on my own dataset, I implement a custom dataset and a custom dataloader according to <strong>Data Loading and Processing Tutorial</strong>. When I run the training process, I can not get the data patch. The process hang up for a long time and no errors are reported.</p>
<p><strong>Dataset</strong></p>
<pre><code class="lang-auto">class CustomDataset(BaseDataset):
    def initialize(self, opt):
        self.root = opt.dataroot
        print 'self.root: {}'.format(self.root)
        self.imgs = os.listdir(self.root)
        self.imgs = sorted(self.imgs)
        self.fdir = os.path.join(os.path.abspath(os.path.join(self.root,'..')), 'features')
        self.feas = [os.path.join(self.fdir, img) for img in self.imgs]
        self.imgs = [os.path.join(self.root, img) for img in self.imgs]
        self.transform = transforms.Compose([transforms.ToTensor(),])

    def __getitem__(self, index):
        i_path = self.imgs[index]
        f_path = self.feas[index]
        img = Image.open(i_path).convert('RGB')
        img = self.transform(img)
        feature = torch.Tensor(cPickle.load(open(f_path, 'rb')))
        #feature = torch.Tensor([1,])
        #print 'len(img):{}'.format(img.size())
        #print 'len(feature):{}'.format(feature.size())
        input_dict = {'img': img, 'feature': feature}
        return input_dict

    def __len__(self):
        return len(self.imgs)

    def name(self):
        return 'CustomDataset'
</code></pre>
<p><strong>DataLoader</strong></p>
<pre><code class="lang-auto">def CreateDataset(opt):
    dataset = None
    from custom_dataset import CustomDataset
    dataset = CustomDataset()
    dataset.initialize(opt)
    return dataset


class CustomDataLoader(BaseDataLoader):
    def name(self):
        return 'CustomDataLoader'


    def initialize(self, opt):
        BaseDataLoader.initialize(self, opt)
        self.dataset = CreateDataset(opt)
        self.dataloader = torch.utils.data.DataLoader(
                self.dataset, 
                batch_size=opt.batchSize, 
                shuffle=not opt.shuffle, 
                num_workers=int(opt.workers))


    def load_data(self):
        return self
    

    def __len__(self):
        return min(len(self.dataset), self.opt.max_dataset_size)
    

    def __iter__(self):
        for i, data in enumerate(self.dataloader):
            print 'custom_dataloader:{}'.format(i)
            if i &gt;= self.opt.max_dataset_size:
                break
            yield data

</code></pre>

username: SimonW
cooked: <p>Ah, yeah, there is an issue with dataloader workers can sometime be signaled or errors but the main process just hangs. Try running with <code>num_workers = 0</code> to get a proper debug trace. This is (sort of) fixed in <a href="https://github.com/pytorch/pytorch/pull/3474">https://github.com/pytorch/pytorch/pull/3474</a> if you are interested.</p>

username: dieuroi
cooked: <p>Thank you for your solution. After I run the process with <code>num_workers = 0</code> , the dataloader goes iterable. However, I still confuse about this problem.</p>

username: SimonW
cooked: <p>Well, if you set <code>num_workers = 0</code>, you will take a performance hit. it seems that the issue is related to multiprocessing then.</p>
<p>Could you change to use <code>torch.load</code> and <code>torch.save</code> for your tensors? Pickling tensors with default pickle module is known to be very slow in some cases.</p>

username: dieuroi
cooked: <p>Thank you for your answer. I reinstalled my os and pytorch requirements these days. I still get that only  <code>num_workers = 0</code> can perform well.<br>
I use <code>torch.load</code> and <code>torch.save</code> for my tensors, thank you so much for your help.</p>

username: SimonW
cooked: <p>I am suspecting that the <code>pickle.load</code> (or now <code>torch.load</code> if you changed to use that) are not playing well with multiprocessing somehow. Could you comment that out, and put something like <code>feature=None</code> and see if you can iterate through the dataset?</p>
<p>If you don’t mind, could you try installing pytorch from the github source? There have been a number of improvements on dataloaders since last release. Some of them are specifically done to prevent dataloader from hanging. The instructions in <a href="http://README.md">README.md</a> at <a href="https://github.com/pytorch/pytorch">https://github.com/pytorch/pytorch</a> are not very complicated.</p>

username: dieuroi
cooked: <p>I think <code>pickle.load</code> play well with multiprocessing while ‘img’ not. I commented image reading out like following:</p>
<pre><code class="lang-auto">feature = pickle.load(xx)
img = feature
</code></pre>
<p>The dataload process then went well.</p>

username: SimonW
cooked: <p>That’s interesting. Could you try reinstalling/upgrading Pillow?</p>

username: pniaz20
cooked: <p>This solution also worked for me. My problem was not a custom dataset, but a custom-defined image augmentation transform. When I used typical <code>torchvision.transform</code> transformations, the data loader was iterable even with <code>num_workers=2</code>. However, when I added my custom-made transform, the <code>iter(trainloader)</code> was just freezing without any errors.</p>
<p>Thanks for the solution. Did not understand why this issue exists though. I also did not understand the point about using load and save rather than the default piclkle.</p>