Pytorch-lightning train_DataLoader object taking forever to loop through the data

username: Harish_Uthravalli
cooked: <p>I have an LSTM model which I want to use for time forecasting. I have a class that extends <code>pl.LightningDataModule</code> and in this class I am trying to create data loaders to train my model.</p>
<h1><a name="class-btcpricedatamodule-1" class="anchor" href="#class-btcpricedatamodule-1"></a>Class BTCPriceDataModule</h1>
<pre><code class="lang-auto">class BTCPriceDataModule(pl.LightningDataModule):

    def __init__(self, train_sequences, test_sequences, batch_size = 8):
        super().__init__()
        self.train_sequence = train_sequences
        self.test_sequences = test_sequences
        self.batch_size = batch_size

    def setup(self, stage=None):
        self.train_dataset = BTCDataset(self.train_sequence)
        self.test_dataset = BTCDataset(self.test_sequences)

    def train_dataloader(self):
        print("coming here")
        return DataLoader(
            self.train_dataset,
            batch_size = self.batch_size,
            shuffle = False,
            num_workers=2
        )
    
    def val_dataloader(self):
        return DataLoader(
            self.test_dataset,
            batch_size=1,
            shuffle=False,
            num_workers=1
        )

    def test_dataloader(self):
        return DataLoader(
            self.test_dataset,
            batch_size=1,
            shuffle=False,
            num_workers=1
        ) 
</code></pre>
<h1><a name="class-btcdataset-2" class="anchor" href="#class-btcdataset-2"></a>Class BTCDataset</h1>
<pre><code class="lang-auto">class BTCDataset(Dataset):
    def __init__(self,sequences):
        self.sequences = sequences
    
    def __len__(self):
        return len(self.sequences)
    
    def __getitem__(self, idx):
        sequence, label = self.sequences[idx]

        return dict(
            sequence=torch.Tensor(sequence.to_numpy()),
            label = torch.tensor(label).float()
        )
</code></pre>
<p>When I run <code>trainer.fit()</code> my sanity check takes forever to complete so I started to debug my code to check if there is any problem while processing the data. When I pass my sequential data through the DataLoader it never is able to even finish one single loop. Example:</p>
<pre><code class="lang-auto">data_module = BTCPriceDataModule(train_sequences, test_sequences, batch_size=BATCH_SIZE)
data_module.setup()

for i in data_module.train_dataloader():
    print(i['sequence'].shape)
    print(i['label'].shape)
    break
</code></pre>
<p>This loop never finishes. I think I gave it enough time and also reduced my dataset size but still it would not compile. <code>len(data_module.train_dataloader())</code> is 1921 so I am pretty sure it is not very heavy on my CPU to take more than 5 minutes to run. How can I debug this problem? Or is there a solution for this?</p>

username: ptrblck
cooked: <p>Could you explain what “compile” means in this context?<br>
The <code>DataLoader</code> would not compile anything, so I guess I might be misunderstanding your explanation.<br>
To debug the actual issue set <code>num_workers=0</code> and check if this would work. If so, your setup seems to run into issues using multiprocessing.</p>

username: Harish_Uthravalli
cooked: <p>Sorry, I meant run instead of compile.</p>
<p>I changed num_workers = 0 and it worked. I wonder why though?</p>

username: ptrblck
cooked: <p>If <code>num_workers=0</code> works fine it could indicate an issue using multiprocessing and based on your description it seems the code just hangs. You could try to update to the latest nightly release to see if you would be running into the same issue or replace the dataset with e.g. <code>torchvision.data.FakeData</code> to test if other datasets would work.</p>