Loading HDF5 data for training a network in Pytorch is super slow

Soheila_Hesaraki: <p>Hi. My problem is the speed of HDF5 data loading and in the rest I will explain the problem and background.</p>
<p>I’ve recently used Pytorch’s Dataloader to load huge data to train neural networks. (33.33 GB data containing log amplitude of STFT audio files). As my tensor shape is huge ( batch_size, 625,513), I have to keep the batch size at most at 4, and use gradient accumulator. The small batch size leads to a lot of steps for training in each epoch (2167 steps for 8667 samples per epoch).</p>
<p>My problem is that, in every step of each epoch, it takes about 2 minutes to get the data, the other parts of each step is in few mil seconds or so. As I have so many steps per epoch, I cannot train the network now!</p>
<p>So, If you could help me, I would be really appreciate that.</p>
<p>Here is my data loading part and training loop:</p>
<pre><code>import torch
from torch.utils.data import DataLoader

class TorchGenerator(Dataset):

       # Constructor
       def __init__(self, x, y):
             self.x = x
             self.y = y

       def __len__(self):
              return self.x.shape[0]

        # Getter
       def __getitem__(self, idx):
           samplex = torch.tensor(self.x[idx], dtype= torch.float , device='cuda')
           sampley = torch.tensor(self.y[idx], dtype= torch.float , device='cuda')

     return samplex, sampley

 data_f = h5py.File(path, "r")
x_train = data_f["X_train_arr"]
y_train = data_f["Y_train_arr"]

 training_data = TorchGenerator(x_train,y_train)
 train_dataloader = DataLoader(training_data, batch_size=BATCH_SIZE, num_workers=8, pin_memory=True)

def train(train_dataloader, STEPS, model, loss_fn, optimizer):
      counter = 1
      model.train()
      optimizer.zero_grad()
      lossbatch = [ ]

      for x, y in train_dataloader:

            x = x.cuda()
            y = y.cuda()

           predict = model(x)
           y_t = y[:, :, :, 1].squeeze(dim=-1).long()
           loss = loss_fn(predict, y_t)
           loss.backward()

           if counter % STEPS == 0:
                optimizer.step()
                optimizer.zero_grad()

           loss_item = loss.detach()

      lossbatch.append(loss_item)
      counter += 1

loss_t = torch.mean(torch.stack(lossbatch))

return loss_t
</code></pre>
<p>P.S.1 : I put time.time() in every single step and I realized the bottleneck is the data loading part (the very beginning of each step of an epoch)</p>
<p>P.S2: As you see I generated the data with 8 workers, but it didn;t improve the speed that much.</p>

ejguan: <p>Try to remove the cuda code inside Dataset and use <code>pin_memory=True</code> in DataLoader. After getting data from <code>DataLoader</code>, you can move your batch data to cuda device.</p>

Soheila_Hesaraki: <p>Thanks for your reply. But what do you mean by removing cuda code inside the dataset? I have to use x.cuda() and y.cuda(), otherwise I will face <a href="https://github.com/pytorch/pytorch/issues/77159" rel="noopener nofollow ugc">this Issue</a> .</p>
<p>For pin memory suggestion, as you see in the script, i’m using it.</p>

ejguan: <aside class="quote no-group" data-username="Soheila_Hesaraki" data-post="3" data-topic="158433">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/soheila_hesaraki/48/51667_2.png" class="avatar"> Soheila_Hesaraki:</div>
<blockquote>
<p>But what do you mean by removing cuda code inside the dataset?</p>
</blockquote>
</aside>
<p>Change</p>
<pre><code class="lang-auto">        def __getitem__(self, idx):
           samplex = torch.tensor(self.x[idx], dtype= torch.float , device='cuda')
           sampley = torch.tensor(self.y[idx], dtype= torch.float , device='cuda')
</code></pre>
<p>to</p>
<pre><code class="lang-auto">        def __getitem__(self, idx):
           samplex = torch.tensor(self.x[idx], dtype= torch.float)
           sampley = torch.tensor(self.y[idx], dtype= torch.float)
</code></pre>
<p>And, can you please wrap your code into <code>if __name__ == "__main__":</code>?</p>

Soheila_Hesaraki: <p><a class="mention" href="/u/ejguan">@ejguan</a> Thanks for you reply. But getting items in getitem in GPU format leads to <a href="https://github.com/pytorch/pytorch/issues/77159" rel="noopener nofollow ugc">this issue.</a>. Also wrapping my code didn’t help that much.</p>
<p>To those who might read it later, I could fix my issue by calculating the exact time that each command take in my training, between layers and generating data by putting these lines before and after each process (each command).</p>
<pre><code>         torch.cuda.synchronize()
         start = time.time()
         "The process"
         torch.cuda.synchronize()
         end = time.time()
         print("name of process", end-start)
</code></pre>
<p>After doing that, I realized the data loading wasn’t the part that takes so long! And I could find the real issue.</p>