Visualize training history from a model

username: Fahad_Jahangir
cooked: <p>I want to generate accuracy/loss vs epoch graph from a trained model. Is it possible to do so? an example image is attached.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d7742efa9e41ac7fc5a9275b345f7f1f78774934.png" data-download-href="https://discuss.pytorch.org/uploads/default/d7742efa9e41ac7fc5a9275b345f7f1f78774934" title="acc"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d7742efa9e41ac7fc5a9275b345f7f1f78774934_2_662x500.png" alt="acc" data-base62-sha1="uJZzQuCEVxtV6DE994EYWkbSTOY" width="662" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d7742efa9e41ac7fc5a9275b345f7f1f78774934_2_662x500.png, https://discuss.pytorch.org/uploads/default/original/2X/d/d7742efa9e41ac7fc5a9275b345f7f1f78774934.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/d/d7742efa9e41ac7fc5a9275b345f7f1f78774934.png 2x" data-dominant-color="FAFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">acc</span><span class="informations">861×650 33 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>You could plot the accuracy e.g. using tensorboard as described <a href="https://pytorch.org/docs/stable/tensorboard.html" rel="nofollow noopener">here</a>.</p>

username: Fahad_Jahangir
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> is it possible to generate graph with already trained pytorch model?</p>

username: ptrblck
cooked: <p>If you have saved the accuracy values for each epoch, this should be possible.<br>
Otherwise, you would need to rerun the training and plot the metrics for each epoch.</p>

username: theodorosP
cooked: <p>Here is the code in python to do so:</p>
<p>from keras.callbacks import History<br>
history = model.fit(X_test, y_train, epochs = 40, batch_size = 5, verbose = 1)<br>
accuracy = history.history[“accuracy”]<br>
epochs = range(1, len(accuracy) + 1)<br>
import matplotlib.pyplot as plt<br>
plt.plot(epochs, accuracy)<br>
plt.show()</p>

username: SM2023
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>I hope you are doing well. Sorry, I am trying to fine tuning the GPT-2 on my system. I just become a bit obsessive about where I need to put  optimizer.zero_grad() , it the code correct.? Many thnaks</p>
<pre><code class="lang-auto">
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(device)

print("***** Running training *****")
print("  Total_num_training_step = {}".format(total_num_training_steps))
print("  Num Epochs = {}".format(num_train_epochs))
print(f"  Train_batch_size per device = {train_batch_size}")
print(f"  Valid_batch_size per device = {valid_batch_size}")


model.to('cuda')

train_loss_1=[]
eval_loss_1=[]
perplexity_1=[]

for epoch in range(num_train_epochs):
    
    print(f"Start epoch{epoch+1} of {num_train_epochs}")
    train_loss=0
    
    epoch_iterator = tqdm(train_dataloader,desc='Iteration')
    
    model.train()
    
    for _, inputs in enumerate(epoch_iterator):  
        
        optimizer.zero_grad()

        d1,d2,d3=inputs
       # print(len(d1))
    
        d1=d1.to('cuda')
        d2=d2.to('cuda')
        d3=d3.to('cuda')
        output = model(input_ids=d1, attention_mask=d2,labels=d3)
        
        batch_loss=output[0]
        
        batch_loss.backward()
        
        optimizer.step()
        
        scheduler.step()
        
        train_loss+=batch_loss.item()
        
        epoch_iterator.set_description('(batch loss=%g)' % batch_loss.item())
        
        del batch_loss
     #   print(training_steps_per_epoch)
        
    train_loss_1.append(train_loss/training_steps_per_epoch)
    
    print(f'Average train loss per example={train_loss/training_steps_per_epoch} in epoch{epoch+1}')    
    print(f'Starting evaluate after epoch {epoch+1}')
    
    eval_loss=[]    
    model.eval()    
    for inputs in tqdm(valid_dataloader, desc="eval"):
        d1,d2,d3=inputs
        d1=d1.to('cuda')        
        d2=d2.to('cuda')
        d3=d3.to('cuda')
        with torch.no_grad():
            output = model(input_ids=d1, attention_mask=d2,labels=d3)
            batch_loss=output[0]
        eval_loss+=[batch_loss.cpu().item()]
        del batch_loss
        
    eval_loss=np.mean(eval_loss)
    
    perplexity=math.exp(eval_loss)
    
    eval_loss_1.append(eval_loss)
    perplexity_1.aapend(perplexity)
    
    print(f'Average valid loss per example={eval_loss} in epoch{epoch+1}')    
    print(f'Perplextiy for valid dataset in epoch{epoch+1} is {perplexity}')
</code></pre>

username: ptrblck
cooked: <p>Yes, your code look correct since you are zeroing out the gradients before calculating them in the <code>backward</code> operation and before updating the parameters in the <code>step()</code> call.</p>

username: SM2023
cooked: <p>is it important where put model.train(), can I put it at the beginning before starting for epochs?</p>

username: ptrblck
cooked: <p>Yes, you should call <code>model.train()</code> before the training loop starts. Your code looks also correct in this manner.</p>

username: SM2023
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Sorry, If I want to use training and validating in this way, how I can send my model  and datasets to the cuda for the process? should I send the tokenizer, data_collator as well? which of them should be passed? I really appreciate your idea.</p>
<pre><code class="lang-auto">training_args = TrainingArguments(
   output_dir=output_dir,
   learning_rate=learning_rate,
   per_device_train_batch_size=per_device_train_batch_size,
   per_device_eval_batch_size=per_device_eval_batch_size,
   num_train_epochs=num_train_epochs,
   weight_decay=weight_decay,
   save_strategy="epoch"
)

trainer = Trainer(
   model=model,
   args=training_args,
   train_dataset=tokenized_train,
   eval_dataset=tokenized_test,
   tokenizer=tokenizer,
   data_collator=data_collator,
   compute_metrics=compute_metrics,
)
</code></pre>

username: ptrblck
cooked: <p>I don’t know where the <code>Trainer</code> class is defined and would guess it’s coming from HuggingFace, Lightning, or another higher-level API.<br>
In plain PyTorch you would move the model and input/target tensors to the device explicitly via:</p>
<pre><code class="lang-python">device = "cuda"
model.to(device)
data = data.to(device)
...
</code></pre>
<p>I don’t know if the <code>Trainer</code> class is supposed to transfer the data to the GPU for you or not so you might need to read the docs of this class in the corresponding library.</p>