Loading data taking a lot of time

magp18: <p>I have tried to make my training faster for the whole weekend but it still needs about 3 hours for one epoch which I don’t think is normal as my model is not that deep. My dataset has around 200k images and they are all in one folder, without labels. They have different sizes with the smalles being 128x128 but I crop them in the middle and resize them to 64x64. The backbone is similar to the DCGAN pytorch tutorial which is where I started and then did some modifications as to have it running on my GPU. However this did not speed my training at all and my GPU is almost not even involved (0~1% it is a GeforceGTX 1080 ). I have been reading similar topics the whole weekend and tried to adapt my code acordingly, writing my own custom dataset, having cuda(non_blocking) etc. but nothing helped. A often asked question was if HDD or SDD was being used, I am using an HDD.</p>
<p>Timing the iterations on the dataloader through batches, the average is around 180 seconds per batch, sometimes jumping to ~325 seconds.</p>
<p>My full code:</p>
<p>ImageFile.LOAD_TRUNCATED_IMAGES = True</p>
<p>matplotlib.rcParams[‘animation.embed_limit’] = 2**128</p>
<p>start_time = time.time()</p>
<h1>seed for reproducibility</h1>
<p>seed = 12<br>
random.seed(seed)<br>
th.manual_seed(seed)<br>
<span class="hashtag">#number</span> of gpu<br>
ngpu = 1</p>
<h1>size of noise vector for the generator to start with</h1>
<p>nz = 100<br>
<span class="hashtag">#number</span> of channels in image 1 -&gt; gray 3 -&gt; RGB<br>
nc = 3<br>
<span class="hashtag">#learning</span> rate<br>
lr = 0.001<br>
<span class="hashtag">#number</span> of epochs<br>
nr_epochs = 1</p>
<h1>batch size</h1>
<p>batch_size = 64<br>
<span class="hashtag">#size</span> of the image to work with<br>
image_size = 64</p>
<h1>number of workers for the data loader</h1>
<p>workers = 3<br>
<span class="hashtag">#dataset</span> = “./dataset/Images/”<br>
dataset = “./dataset/”</p>
<p>Tensor = th.cuda.FloatTensor</p>
<p>class JTA200(th.utils.data.Dataset):<br>
def <strong>init</strong>(self, image_paths, transform=None):<br>
self.root = image_paths<br>
self.transform = transform<br>
self.image_paths = []<br>
for path in os.listdir(self.root):<br>
full_path = os.path.join(self.root, path)<br>
if os.path.isfile(full_path):<br>
self.image_paths.append(full_path)<br>
def <strong>getitem</strong>(self, index):<br>
image_path = self.image_paths[index]<br>
x = Image.open(image_path)<br>
x = x.convert(‘RGB’)<br>
if self.transform is not None:<br>
x = self.transform(x)<br>
return x</p>
<pre><code>def __len__(self):
    return len(self.image_paths)
</code></pre>
<p>def to_img(x):<br>
out = 0.5 * (x + 1)<br>
out = out.clamp(0, 1)<br>
out = out.view(-1, nc, image_size, image_size)<br>
return out</p>
<p>def weights_init(m):<br>
classname = m.<strong>class</strong>.<strong>name</strong><br>
if classname.find(‘Conv’) != -1:<br>
th.nn.init.normal_(m.weight.data, 0.0, 0.02).to(device)<br>
elif classname.find(‘BatchNorm’) != -1:<br>
th.nn.init.normal_(m.weight.data,1.0,0.02).to(device)<br>
th.nn.init.constant_(m.bias.data, 0).to(device)</p>
<p>class Flatten(th.nn.Module):<br>
def forward(self, x):<br>
return x.view(x.shape[0], -1)</p>
<p>class Descriminator(th.nn.Module):<br>
def <strong>init</strong>(self,kernel_size, p):<br>
super(Descriminator,self).<strong>init</strong>()<br>
self.conv = th.nn.Sequential(<br>
th.nn.Dropout2d§,<br>
th.nn.Conv2d(nc,64,kernel_size, padding=1,stride=2), <span class="hashtag">#32x32x64</span><br>
th.nn.LeakyReLU(0.2, True),<br>
th.nn.Dropout2d§,<br>
th.nn.Conv2d(64, 64, kernel_size,padding=1, stride=2), <span class="hashtag">#16x16x64</span><br>
th.nn.BatchNorm2d(64),<br>
th.nn.LeakyReLU(0.2, True),<br>
th.nn.Dropout2d§,<br>
th.nn.Conv2d(64, 32, kernel_size,padding=1,stride=2), <span class="hashtag">#8x8x32</span><br>
th.nn.BatchNorm2d(32),<br>
th.nn.LeakyReLU(0.2, True),<br>
Flatten(),<br>
th.nn.Dropout2d§,<br>
th.nn.Linear(2048,84),<br>
th.nn.LeakyReLU(0.2, True),<br>
th.nn.Dropout2d§,<br>
th.nn.Linear(84,32),<br>
th.nn.LeakyReLU(0.2, True),<br>
th.nn.Linear(32,1),<br>
th.nn.Sigmoid()<br>
)<br>
def forward(self, input):<br>
y = self.conv(input)<br>
return y</p>
<p>class Generator(th.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Generator,self).<strong>init</strong>()<br>
self.deconv = th.nn.Sequential(<br>
th.nn.ConvTranspose2d(nz,64,4,1,0), # 100x64x1<br>
th.nn.BatchNorm2d(64),<br>
th.nn.ReLU(True),<br>
th.nn.ConvTranspose2d(64,64,4,2,1), # 100x64x1<br>
th.nn.BatchNorm2d(64),<br>
th.nn.ReLU(True),<br>
th.nn.ConvTranspose2d(64,64,4,2,1), # 100x64x1<br>
th.nn.BatchNorm2d(64),<br>
th.nn.ReLU(True),<br>
th.nn.ConvTranspose2d(64,32,4,2,1), # 100x32x1<br>
th.nn.BatchNorm2d(32),<br>
th.nn.ReLU(True),<br>
th.nn.ConvTranspose2d(32,nc,4,2,1), # 100x3x1<br>
<span class="hashtag">#to</span> return data in -1 to 1 range tanh is used<br>
th.nn.Tanh()<br>
)</p>
<pre><code>def forward(self, input):
    generated_img = self.deconv(input)
    return generated_img
</code></pre>
<p><span class="hashtag">#Decide</span> whether to run it on gpu or cpu<br>
if <strong>name</strong> == ‘<strong>main</strong>’:<br>
cudnn.benchmark = True</p>
<pre><code>device = th.device("cuda" if (th.cuda.is_available() and ngpu &gt; 0) else "cpu")
if th.cuda.is_available(): th.cuda.manual_seed(12)

dataset = dset.ImageFolder(root=dataset,
                           transform=transforms.Compose([
                               transforms.Resize(image_size),
                               transforms.CenterCrop(image_size),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))
                           ]))
#dataset = JTA200(dataset, transform=transforms.Compose([transforms.Resize(image_size),
#                                                             transforms.CenterCrop(image_size),
#                                                             transforms.ToTensor(),
#                                                             transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))]))
#create the data loader
dataloader = th.utils.data.DataLoader(dataset, batch_size = batch_size,shuffle=True,
                                      num_workers=workers, pin_memory= True)

# Check images
real_batch = next(iter(dataloader))
real_batch = [_.cuda(non_blocking = True) for _ in real_batch]
plt.figure(figsize=(8,8))
plt.axis("off")
plt.title("Training images")
plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64],padding = 2,
                                    normalize=True).cpu(),(1,2,0)))
</code></pre>
<h1>plt.imshow(np.transpose(vutils.make_grid(real_batch.to(device)[:64], padding=2,</h1>
<h1>normalize=True).cpu(),(1,2,0)))</h1>
<pre><code>plt.show()

#kernel size for the convolutions / deconvolutions
kernel_size = 3
#Define Dropout probability
p = 0.5

generator = Generator().to(device)
# Handle multi-gpu
if (device.type == 'cuda') and (ngpu &gt;= 1):
    netG = th.nn.DataParallel(generator, list(range(ngpu)))

if (glob.glob("./model/G*")):
    print("Loading Generator ...")
    generator.load_state_dict(th.load(glob.glob("./model/G*")[0]))
    generator.eval()
else:
    # initialize weights for the generator
    generator.apply(weights_init)

descriminator = Descriminator(kernel_size,p).to(device)
# same as before for operations in the descriminator
if (device.type == 'cuda') and (ngpu &gt;= 1):
    descriminator = th.nn.DataParallel(descriminator,list(range(ngpu)))

if (glob.glob("./model/D*")):
    print("Loading discriminator ...")
    descriminator.load_state_dict(th.load(glob.glob("./model/D*")[0]))
    descriminator.eval()
else:
    descriminator.apply(weights_init)

# Initialize the loss function
loss = th.nn.BCELoss().cuda()

# batch of latent vectors to visualize generators progress
fixed_noise = th.randn(batch_size,nz,1,1,device=device)

ground_truth = 1
fake_label = 0

optimizerD = th.optim.Adam(descriminator.parameters(),lr=lr, betas=(0.5,0.999))
optimizerG = th.optim.Adam(generator.parameters(),lr=lr, betas=(0.5,0.999))

# Training
# statistical info
img_list = []
G_losses = []
D_losses = []
epochs = []
loss_legend = ['Discriminator', 'Generator']

iters = 0

print("------Training started------")
time_batch = 0
for epoch in range(nr_epochs):
    for i,data in enumerate(dataloader,0):
        #time_batch = time.time()
        # Train with real images for discriminator
        descriminator.zero_grad() # reset gradients to not mix between batches
        # format bach
        real_cpu = data[0].to(device)
        #real_cpu = data.to(device)
        b_size = real_cpu.size(0)
        label = th.full((b_size,), ground_truth,device=device)
        # forward propagate in Discriminator
        output = descriminator(real_cpu).view(-1)
        #calculate error and backpropagate
        loss_D_real = loss(output,label) # the lower the better it spots real images
        loss_D_real.backward()
        D_x = output.mean().item()
        # Train with fake images
        noise = th.randn(b_size,nz,1,1, device=device)
        # generate images with G
        fake = generator(noise)
        label.fill_(fake_label)
        #Classify all fake images
        output = descriminator(fake.detach()).view(-1)
        # Calculate classifier loss
        loss_D_fake = loss(output,label) # how well the descriminator spots fakes
        loss_D_fake.backward()
        D_G_z1 = output.mean().item() # average of descriminator guesses on fake data , should be 0.5
        # add gradients real and fake
        lossD = loss_D_real+loss_D_fake
        #update D
        optimizerD.step()

        # -------------- Generator ---------------
        generator.zero_grad()
        label.fill_(ground_truth)
        output = descriminator(fake).view(-1)
        lossG = loss(output,label) # the lower, the higher the chances D is being fooled
        lossG.backward()
        D_G_z2 = output.mean().item() #mean of how well discriminators spot real images, should be 0.5
        optimizerG.step()
        G_losses.append(lossG.item())
        D_losses.append(lossD.item())
        epochs.append(epoch + i / len(dataloader))

        if (iters % 25 == 0) or ((epoch == nr_epochs-1) and (i == len(dataloader)-1)):
            count = 0
            with th.no_grad():
                fake = generator(fixed_noise).detach().cpu()
                if (iters % 300 == 0):
                    for i in range(batch_size - 1):
                        # img = Image.fromarray(fake[i], 'RGB')
                        img = fake[i].data
                        img = to_img(img)
                        vutils.save_image(img, "./Generated/{}.png".format(count))
                        count += 1
        if i % 50 == 0:
            print('[%d/%d][%d/%d]\tLoss_D: %.4f\tLoss_G: %.4f\tD(x): %.4f\tD(G(z)): %.4f / %.4f Time: %f'
                  % (epoch, nr_epochs, i, len(dataloader),
                     lossD.item(), lossG.item(), D_x, D_G_z1, D_G_z2, time.time() - time_batch))

            img_list.append(vutils.make_grid(fake,padding=2,normalize=True))
            time_batch = time.time()
        iters+=1
th.save(descriminator.state_dict(), "./model/Dmodel.pth.tar")
th.save(generator.state_dict(), "./model/Gmodel.pth.tar")
plt.figure(figsize=(10,5))
plt.title("Generator and Discriminator Loss During Training")
plt.plot(G_losses,label="G")
plt.plot(D_losses,label="D")
plt.xlabel("iterations")
plt.ylabel("Loss")
plt.legend()
plt.show()

fig = plt.figure(figsize=(8,8))
plt.axis("off")
ims = [[plt.imshow(np.transpose(i,(1,2,0)), animated= True)] for i in img_list]
ani = animation.ArtistAnimation(fig, ims, interval=1000,repeat_delay=1000,blit=True)
#ani.save("animation.mp4")
#HTML(ani.to_jshtml())
plt.show()

print("--- %s seconds for training ---" % (time.time() - start_time))
# Grab a batch of real images from the dataloader
real_batch = next(iter(dataloader))
#next_batch = [_.cuda(non_blocking = True) for _ in real_batch]
# Plot the real images
plt.figure(figsize=(15,15))
plt.subplot(1,2,1)
plt.axis("off")
plt.title("Real Images")
plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding=5, normalize=True).cpu(),(1,2,0)))
</code></pre>
<h1>plt.imshow(np.transpose(vutils.make_grid(real_batch.to(device)[:64], padding=5, normalize=True).cpu(),(1,2,0)))</h1>
<pre><code># Plot the fake images from the last epoch
plt.subplot(1,2,2)
plt.axis("off")
plt.title("Fake Images")
plt.imshow(np.transpose(img_list[-1],(1,2,0)))
plt.show()</code></pre>

ptrblck: <aside class="quote no-group" data-username="magp18" data-post="1" data-topic="80651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/magp18/48/24304_2.png" class="avatar"> magp18:</div>
<blockquote>
<p>I am using an HDD</p>
</blockquote>
</aside>
<p>This might create a data loading bottleneck.<br>
You could use the <code>data_time</code> meter from the <a href="https://github.com/pytorch/examples/blob/b9f3b2ebb9464959bdbf0c3ac77124a704954828/imagenet/main.py#L280">ImageNet example</a> to track the data loading time.</p>
<p><a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">This post</a> gives you some more information in case you haven’t read it already.</p>

magp18: <p>Thank you for your reply. I did already track the data loading time with time.time(), which already gave me the idea that this was indeed the problem. I will try to give some ideas on the purposed post a try.</p>