Estimated time of an epoch

username: TahaMagdy
cooked: <p>How can I calculate the estimated time of an epoch?!</p>

username: ptrblck
cooked: <p>You could start a timer before training for one epoch and calculate the elapsed time afterwards.</p>
<pre><code class="lang-auto">def train(epoch):
    for batch_idx, (data, target) in enumerate(train_loader):
    # your training code ...


t0 = time.time()
train(1)
print('{} seconds'.format(time.time() - t0)</code></pre>

username: TahaMagdy
cooked: <p>Thank you,<br>
I have learnt Keras first. I have seen a progress bar like the following:</p>
<p>[============================================================] 100%, 0 seconds.</p>
<p>It is shown immediately once an epoch starts.</p>
<p>I will use your solution, but I wonder if what I have explained can be done using pytorch.</p>

username: ptrblck
cooked: <p>I took this from some PyTorch examples and I like this style:</p>
<pre><code class="lang-auto">def train(epoch):
    '''
    Main training loop
    '''
    # Set model to train mode
    model.train()
    # Iterate training set
    for batch_idx, (data, mask) in enumerate(train_loader):
        if use_cuda:
            data = data.cuda()
            mask = mask.cuda()
        data = Variable(data)
        mask = Variable(mask.squeeze())
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output.squeeze(), mask)
        loss.backward()
        optimizer.step()
        
        if batch_idx % 10 == 0:
            loss_data = loss.data[0]
            train_losses.append(loss_data)
            print(
                'Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.
                format(epoch, batch_idx * len(data), len(train_loader.dataset),
                       100. * batch_idx / len(train_loader), loss_data))

</code></pre>

username: Neda
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Is it essential to put model.train() before train the model for all models?</p>

username: Neda
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Calculate the elapsed time would be the same when we are running the model on CUDA?</p>

username: ptrblck
cooked: <p>Generally yes. After calling <code>model.train()</code> some layers like <code>nn.BatchNorm</code> will change their behavior, e.g. by updating the running estimates and using the batch statistics to normalize your activations.</p>
<p>Depending on your workload your training procedure should be faster running on the GPU. However, if your workload is quite small or if you have some bottlenecks in your code, the CPU time might be close to the GPU one.</p>

username: wiem_chakroun
cooked: <p>even when running WGAN-GP we should call netG.train(), netD.train()?<br>
I have problem while running wgan-gp in the first 10 epoch it goes quickly (1 day) but then it tooks 1 epoch per day. My data is 8 500 hdf5 image (3D) and i am running my model on only 1 GPU. I can’t really know if it’s from the dataloader or from the network architecture or is it a normal thing ?</p>

username: ptrblck
cooked: <p>It seems you are seeing a slowdown in training and it shouldn’t depend on the training/eval mode of the model.</p>
<p>Could you adapt the data loading measurement from the <a href="https://github.com/pytorch/examples/blob/0634306968ed7f28e2b11b69d3b2df348b8ec53c/imagenet/main.py#L280" rel="nofollow noopener">ImageNet example</a>?<br>
<code>data_time</code> should go towards zero if you don’t have a data loading bottleneck.</p>
<p>Also, are you the only user on this machine or could the machine be blocked by other processes during your training?<br>
Do you see any increase in GPU memory usage during training?</p>

username: wiem_chakroun
cooked: <p>so yes i have adapted and it is equal to zero (data_time Data  0.000 ( 0.000)).<br>
nobody is using with me the machine and no the GPU memory does not increase during the training process.</p>

username: ptrblck
cooked: <p>Does <code>nvidia-smi</code> show the same GPU utilization throughout your training?</p>
<p>Is the data loading time also zero after the training slows down and is the data stored locally on your machine?</p>

username: wiem_chakroun
cooked: <p>yes <code>nvidia-smi</code> don’t show the same utilization.<br>
So Data is still zero.  BUT the GPU memory usage increase from one epoch to other. Yes the data is stored locally.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="wiem_chakroun" data-post="12" data-topic="14988">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wiem_chakroun/48/17648_2.png" class="avatar"> wiem_chakroun:</div>
<blockquote>
<p>BUT the GPU memory usage increase from one epoch to other.</p>
</blockquote>
</aside>
<p>That’s good to know.<br>
Are you storing the <code>loss</code> or some other output of your model somewhere?<br>
For debugging or printing purposes you should <code>detach()</code> the outputs or store the <code>loss.item()</code> e.g. in a loss.<br>
Otherwise, you will store the whole computation graph, which might eventually yield an out of memory error and might also slow down your code.</p>
<p>Could this be the case in your code?</p>

username: wiem_chakroun
cooked: <p>ok so i am storing <code>loss.item()</code> yes but i don’t use <code>detach()</code> anywhere in my code.</p>
<p>could you please show me a snippet of code where should i put <code>detach()</code>?<br>
actually i even see people in github who use <code>detach()</code> after the initialization of the noise</p>

username: ptrblck
cooked: <p>If you are using <code>item()</code>, you won’t need to use <code>detach</code> additionally.<br>
However, you would need to call <code>detach()</code> on other tensors you are storing and which are still connected to the computation graph.<br>
Could you post a code snippet to reproduce this slow down and how long would it take to reach the slow down?</p>

username: wiem_chakroun
cooked: <p>how could i detect which tensors are still connected to the computation graph?<br>
So i have printed after each process the <code>time.time()</code> and i have found that:</p>
<ul>
<li>In the first epoch it takes 1 second to do the discriminator backpropagation and zero for generator backpropagation.</li>
<li>After 28 epoch it takes 3 second to do the discriminator backpropagation and 4 second for generator backpropagation.<br>
Actually i use <code>loss.item()</code> while storing in tensorboard does this make a difference?</li>
</ul>
<pre><code class="lang-auto"> for epoch in range(opt.niter):
    print ("epoch||||||||||||||||", time.time())
    for i, batch in enumerate(dataloader,0):
        data_time = AverageMeter('Data', ':6.3f')
        print("data_time",data_time)
        print ("iteration============", time.time())
        for iter in range(3):
            print ("discriminator_iteration", time.time())
            f = open(work_dir+"training_curve.csv", "a")
            autograd.detect_anomaly()       
            ############################
            # (1) Update D network
            ############################
            ### PART ---1--- training the discriminator on real data ####
            optimizerD.zero_grad()
            x = Variable(batch.type(Tensor))
            noise = Variable(Tensor(batch.size(0), nz, 1, 1, 1).normal_(0, 1))
            x_tilde = Variable(netG(noise), requires_grad=True)
            epsilon = Variable(Tensor(batch.size(0), 1, 1, 1, 1).normal_(0, 1))
            x_hat = epsilon*x + (1 - epsilon)*x_tilde
            x_hat = torch.autograd.Variable(x_hat, requires_grad=True)
            dw_x = netD(x_hat)
        
            grad_x = torch.autograd.grad(outputs=dw_x, inputs=x_hat,
                                        grad_outputs=Variable(Tensor(dw_x.size()).fill_(1.0), requires_grad=False),
                                        create_graph=True, retain_graph=True, only_inputs=True)

            grad_x = grad_x[0].view(batch.size(0), -1)
            grad_x = grad_x.norm(p=2, dim=1)
            d_loss = torch.mean(netD(x_tilde)) - torch.mean(netD(x)) + LAMBDA*torch.mean((grad_x - 1)**2) 
            gradient_penal=LAMBDA*torch.mean((grad_x - 1)**2)
            wgan_loss=torch.mean(netD(x_tilde)) - torch.mean(netD(x))
            print ("before back discriminator|||||||||||||||", time.time())
            d_loss.backward()
            print ("after back discriminator==============", time.time())
            for p in netD.parameters():
                p.register_hook(lambda grad: torch.clamp(grad, -clip_value, clip_value))
            print ("after gradient clipping|||||||||||", time.time())
            for p in list(filter(lambda p: p.grad is not None, netD.parameters())):
                print("Discriminator",p.grad.data.norm(2).item())
            optimizerD.step()
        ###########################
        # (2) Update G network
        ###########################
        optimizerG.zero_grad()
        print ("before generator iter||||||||||||", time.time())
        noise = Variable(Tensor(batch.size(0), nz, 1, 1,1).normal_(0, 1))
        imgs_fake = netG(noise)
        g_loss = -torch.mean(netD(imgs_fake)) ##minimiser
        print ("before back generator==========", time.time())
        g_loss.backward()
        print ("after back generator||||||||||||", time.time())
        for p in netG.parameters():
            p.register_hook(lambda grad: torch.clamp(grad, -clip_value, clip_value))
        for p in list(filter(lambda p: p.grad is not None, netG.parameters())):
            print("Generator",p.grad.data.norm(2).item())
        print ("befor weight update generateur===========", time.time())
        optimizerG.step()
       ## TensorBoard
        writer.add_scalar('G_loss', g_loss.item(), i + epoch * len(dataloader))
        writer.add_scalar('D_loss', d_loss.item(), i + epoch * len(dataloader))
        writer.add_scalar('gradient_penal', gradient_penal.item(), i + epoch * len(dataloader))
        writer.add_scalar('wgan_loss', wgan_loss.item(), i + epoch * len(dataloader))
        writer.add_scalar('fake_loss', fake_loss.item(), i + epoch * len(dataloader))
        writer.add_scalar('real_loss', real_loss.item(), i + epoch * len(dataloader))
        f.write('[%d/%d][%d/%d] Loss_D: %.4f Gradient_p: %.4f WGAN_loss: %.4f Loss_G: %.4f'
            % (epoch, opt.niter, i, len(dataloader),
                d_loss.data, gradient_penal.data, wgan_loss.data, g_loss.data))
        f.write('\n')
        f.close()
</code></pre>

username: luanpham
cooked: <p>Try <a href="https://github.com/tqdm/tqdm" rel="nofollow noopener">tqdm</a>.</p>
<p>Example</p>
<pre><code class="lang-auto">        for i, (images, targets) in tqdm(
                enumerate(self._train_loader),
                total=len(self._train_loader),
                leave=False
            ):
 
</code></pre>
<p>it will show something like this,<br>
<code> 89%|#########################################################       | 534/599 [07:56&lt;00:57,  1.13it/s]</code></p>

username: ptrblck
cooked: <p>Thanks for the code!</p>
<ul>
<li>
<code>autograd.detect_anomaly()</code> should be quite expensive, so if your code runs fine, I would recommend to remove this call.</li>
<li>
<code>Variable(batch.type(Tensor))</code>: <code>Variables</code> are deprecated since <code>0.4.0</code>, so you can just use tensors in newer versions. Also, if you want to convert the <code>dtype</code>, you should use <code>to()</code> or directly call the type as e.g. <code>float()</code>.</li>
<li>I haven’t debugged your code, but do you really need to retain the graph in <code>grad_x</code>? I’m a bit worried it might be stored unnecessarily.</li>
<li>Since you are running the code on a GPU and CUDA calls are asynchronous, you should call <code>torch.cuda.synchronize()</code> before starting and stopping the timer. Otherwise your profiling might yield wrong results (e.g. kernel launch times)</li>
</ul>

username: wiem_chakroun
cooked: <p>thanks for your response!<br>
This have accelerated the training process.</p>