Torch.jit.trace only for inference? not training?

BongHoe_Koo: <p>Hi PyTorch Team,</p>
<p>i am currently struggling to figure out what is the problem.<br>
i made custom vgg16 model in python environment and export model with using torch.jit.trace (No training)<br>
and then i tried to train this model in c++ but it seems that accuracy is not the same as python environment</p>
<p>why?</p>
<p>※ Also i found specific model can’t train in c++ (another custom model works well but only custom vgg16 model accuracy seems weird, of course i used exact same training loop in c++ )</p>
<p>Python result :<br>
Epoch: 0001 cost = 0.988920689 acc = 0.840617180<br>
Epoch: 0002 cost = 0.837770224 acc = 0.981513083</p>
<p>Process finished with exit code -1</p>
<p>C++ result :</p>
<p>gpu enabled<br>
current epoch = 0<br>
average accuracy : 0.264888<br>
average cost : 1.38596<br>
current epoch = 1<br>
average accuracy : 0.268539<br>
average cost : 1.38558<br>
current epoch = 2<br>
average accuracy : 0.267135<br>
average cost : 1.38528<br>
current epoch = 3</p>
<p>Vgg16 Model (Model Structure)</p><aside class="onebox githubblob" data-onebox-src="https://github.com/gellston/DeepLearningStudy/blob/main/torch/model/VGG16FC.py">
  <header class="source">

      <a href="https://github.com/gellston/DeepLearningStudy/blob/main/torch/model/VGG16FC.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/gellston/DeepLearningStudy/blob/main/torch/model/VGG16FC.py" target="_blank" rel="noopener nofollow ugc">gellston/DeepLearningStudy/blob/main/torch/model/VGG16FC.py</a></h4>


      <pre><code class="lang-py">import torch

class VGG16FC(torch.nn.Module):

    def __init__(self, class_num=5):
        super(VGG16FC, self).__init__()
        self.drop_rate = 0.3
        self.class_num = class_num

        self.layer1 = torch.nn.Sequential(torch.nn.Conv2d(3, 64, kernel_size=3, stride=1, padding='same'),
                                          torch.nn.BatchNorm2d(64),
                                          torch.nn.ReLU(),
                                          torch.nn.Conv2d(64, 64, kernel_size=3, stride=1, padding='same'),
                                          torch.nn.BatchNorm2d(64),
                                          torch.nn.ReLU(),
                                          torch.nn.MaxPool2d(kernel_size=2, stride=2))

        self.layer2 = torch.nn.Sequential(torch.nn.Conv2d(64, 128, kernel_size=3, stride=1, padding='same'),
                                          torch.nn.BatchNorm2d(128),
                                          torch.nn.ReLU(),
</code></pre>



  This file has been truncated. <a href="https://github.com/gellston/DeepLearningStudy/blob/main/torch/model/VGG16FC.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>TrainingScript(Include save model)</p><aside class="onebox githubblob" data-onebox-src="https://github.com/gellston/DeepLearningStudy/blob/main/torch/torch_fiat_classification_food.py">
  <header class="source">

      <a href="https://github.com/gellston/DeepLearningStudy/blob/main/torch/torch_fiat_classification_food.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/gellston/DeepLearningStudy/blob/main/torch/torch_fiat_classification_food.py" target="_blank" rel="noopener nofollow ugc">gellston/DeepLearningStudy/blob/main/torch/torch_fiat_classification_food.py</a></h4>


      <pre><code class="lang-py">import torch
import torch.nn as nn
import random
import cv2
import numpy as np

from torchsummary import summary
from torch.utils.data import DataLoader

from model.VGG16FC import VGG16FC
from util.FIATClassificationDataset import FIATClassificationDataset


USE_CUDA = torch.cuda.is_available() # GPU를 사용가능하면 True, 아니라면 False를 리턴
device = torch.device("cuda" if USE_CUDA else "cpu") # GPU 사용 가능하면 사용하고 아니면 CPU 사용
print("다음 기기로 학습합니다:", device)


# for reproducibility
random.seed(777)
</code></pre>



  This file has been truncated. <a href="https://github.com/gellston/DeepLearningStudy/blob/main/torch/torch_fiat_classification_food.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>C++ repository is private access now . so if you guys want to see c++ code also<br>
then i can give source code privately</p>
<p>Thanks.</p>

ptrblck: <p><code>torch.jit.trace</code> will use the provided input to record all functions as they were executed and is thus unable to use any data-dependent control flow etc.<br>
In your particular case all dropout layers would be “fixed” which is most likely one of the reasons the training fails using the traced model. <code>torch.jit.script</code> on the other hand should be able to track these operations as well.</p>

BongHoe_Koo: <p>Thanks for the good comments</p>
<p>i just figured out ,  i trained at least one epoch before save model<br>
then training fail disappeared</p>
<p>My guess is that through 1 epoch training,<br>
torch.jit.trace was able to trace the training process.<br>
Is this guess correct?</p>
<p>Also, is 1 epoch training before saving the model likely to be an issue in the future? (potential issue?)<br>
If there are no potion issues, I have no problem using this method now.</p>
<p>Thanks.</p>

ptrblck: <p>No, I don’t think training the model for one epoch should change anything and cannot explain why it seems to work now. <code>trace</code> would still keep the layers in their traced “state”, i.e. it would still keep the same dropout mask etc., which would still be concerning.</p>

BongHoe_Koo: <p>ummmm If you’re right, I won’t be using dropouts in the future. However, unlike the dropout issue, the model saved after learning 1 epoch seems to be learning smoothly in C++. I will post related screenshots.</p>

ptrblck: <p>The fix would be to use <code>torch.jit.script</code> instead of <code>torch.jit.trace</code> so I wouldn’t disable dropout in the future.</p>

BongHoe_Koo: <p>I think I misunderstood what you said earlier. I’m sorry.<br>
Is there a way to save the model using torch.jit.script?</p>

ptrblck: <p>Yes, just replace <code>torch.jit.trace</code> with <code>torch.jit.script</code> in your Python script and save the model afterwards. <a href="https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html#using-scripting-to-convert-modules">This section of the tutorial</a> might be interesting to take a look at.</p>

BongHoe_Koo: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/a/bab0fe9fb7e3e8d6707f705808b9fb8c7066b12f.png" data-download-href="https://discuss.pytorch.org/uploads/default/bab0fe9fb7e3e8d6707f705808b9fb8c7066b12f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/bab0fe9fb7e3e8d6707f705808b9fb8c7066b12f.png" alt="image" data-base62-sha1="qDy1a1KNZzR12gJy0vX3p4SE7Qr" width="690" height="389" data-dominant-color="252525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">828×467 17.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> its working!!</p>