Model training failing

username: JosefLintz
cooked: <p>I’m trying to create a “simple” model that takes a standard (3-channel) 36x36 image and outputs a probability that the image contains a certain object of interest.</p>
<p>I’ve found a simple model (I don’t remember where, can’t credit), I adjusted the parameters to fit our needs. Since we’re planning on using C++, I’m using <code>tracing </code> to load this model</p>
<p>This is my Pytorch model (Python):</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
       self.conv1 = torch.nn.Conv2d(3, 32, kernel_size=3, stride=1)
        self.conv2 = torch.nn.Conv2d(32, 64, kernel_size=3, stride=1)
        self.pool = torch.nn.MaxPool2d(kernel_size=2, stride=2)
        self.fc1 = torch.nn.Linear(64 * 7 * 7, 128)
        self.fc2 = torch.nn.Linear(128, 1)
        self.sigmoid = torch.nn.Sigmoid()

    def forward(self, x):
        x = self.pool(torch.relu(self.conv1(x)))
        x = self.pool(torch.relu(self.conv2(x)))
        x = x.view(-1, 64 * 7 * 7)
        x = torch.relu(self.fc1(x))
        x = self.sigmoid(self.fc2(x))
        return x.squeeze(1)
</code></pre>
<p>Since the model works on a “non-standard Pytorch” dataset, I had to write a custom dataset class:<br>
(I’ll only include the overloaded <code>get</code> function, since it’s really the only “moving part”)<br>
The dataset expects a “main” directory to contain two sub-directories “Positive” and “Negative”, each containing the appropriate images.</p>
<pre><code class="lang-auto">torch::data::Example&lt;&gt; get(const std::size_t index)
{
    constexpr auto labels = 2;
        
    const auto indexAdjusted = index / labels;
        
      
    const auto labelIndex = indexAdjusted % labels;
        
    const std::string_view labelName = labelIndex == 0 ?
    negativeDirectoryName : positiveDirectoryName;
        
    // `_bacteriaImageDirectory` is an `std::filesystem::path` to the "Main" directory
    const auto directory = _bacteriaImageDirectory / labelName;
        
    auto directoryIterator = std::filesystem::directory_iterator(directory);
    std::advance(directoryIterator, indexAdjusted);
       
    auto image = cv::imread(directoryIterator-&gt;path().string());
    cv::cvtColor(image, image, cv::ColorConversionCodes::COLOR_BGR2RGB);
        
        
    auto tensorImage = torch::from_blob(image.data, { 3, image.cols, image.rows }, torch::kByte)
    .to(torch::kFloat32) / 255.0f;
        
    const auto tensorLabel = torch::tensor(static_cast&lt;float&gt;(labelIndex), torch::kFloat32);
        
    return { tensorImage.clone(), tensorLabel };
};
</code></pre>
<p>This is my training loop, currently I run the training on 2160 images:</p>
<pre><code class="lang-auto">// Since `model` is a loaded "jit" module, I have to manually convert the module parameters 
// to a valid type that can be accepted by `torch::optim::Adam`
std::vector&lt;torch::Tensor&gt; modelParameters;
modelParameters.reserve(model.parameters().size());

std::transform(model.parameters().begin(), model.parameters().end(),
std::back_insert_iterator(modelParameters),
[](const auto&amp; p)
{
    return p;
});

auto optimizer = torch::optim::Adam(modelParameters);

for(const auto&amp; [image, label] : *dataLoader)
{
    optimizer.zero_grad();
    
    const auto output = model.forward({image}).toTensor();
    
    const auto prediction = output.item&lt;float&gt;();
    
    auto loss = torch::nn::functional::binary_cross_entropy(output, label);
    
    loss.backward();
    optimizer.step();
    
    
    std::clog  
    &lt;&lt; "Loss: " &lt;&lt; loss.item&lt;double&gt;() 
    &lt;&lt; ", Prediction: " &lt;&lt; prediction 
    &lt;&lt; ", Label: " &lt;&lt; static_cast&lt;std::uint32_t&gt;(label.item&lt;float&gt;()) 
    &lt;&lt; '\n';
};
</code></pre>
<p>Attached are plotted results of the training. I don’t have much experience working with neural networks, but I don’t think training is supposed to look like that</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc57c84601ea05f003af227d1bec7e8dfc487bd9.png" data-download-href="https://discuss.pytorch.org/uploads/default/cc57c84601ea05f003af227d1bec7e8dfc487bd9" title="train-results"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc57c84601ea05f003af227d1bec7e8dfc487bd9_2_690x396.png" alt="train-results" data-base62-sha1="t9HtyvY87d9tm0GxQywORM97Wm5" width="690" height="396" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc57c84601ea05f003af227d1bec7e8dfc487bd9_2_690x396.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc57c84601ea05f003af227d1bec7e8dfc487bd9_2_1035x594.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc57c84601ea05f003af227d1bec7e8dfc487bd9.png 2x" data-dominant-color="F9FAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">train-results</span><span class="informations">1043×599 54.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What am I missing? I’ve been stuck here for a while, any help is very appreciated</p>

username: JosefLintz
cooked: <p>I have somewhat of an update.</p>
<p>I changed <code>torch::nn::functional::binary_cross_entropy</code> to <code>torch::nn::BCELoss</code>.<br>
I added 5 epochs. Before training I set the model to “Train” mode with <code>model.train()</code>.*</p>
<p>My training loop now looks like so:</p>
<pre><code class="lang-auto">auto criterion = torch::nn::BCELoss();
model.train();

auto optimizer = torch::optim::Adam(modelParameters, torch::optim::AdamOptions{});

for (std::size_t i = 0; i &lt; 5; ++i) 
{
    std::clog &lt;&lt; "Epoch " &lt;&lt; i + 1 &lt;&lt; '/' &lt;&lt; 5 &lt;&lt; '\n';
    
    for(const auto&amp; [image, label] : *dataLoader)
    {
        optimizer.zero_grad();
        
        const auto output = model.forward({image}).toTensor();
        
        const auto prediction = output.item&lt;float&gt;();
        
        auto loss = criterion-&gt;forward(output, label);
        
        loss.backward();
        optimizer.step();
        
        std::clog  
        &lt;&lt; "Loss: " &lt;&lt; loss.item&lt;double&gt;() 
        &lt;&lt; ", Prediction: " &lt;&lt; prediction 
        &lt;&lt; ", Label: " &lt;&lt; static_cast&lt;std::uint32_t&gt;(label.item&lt;float&gt;()) 
        &lt;&lt; '\n';
    };
};
</code></pre>
<p>It’s mostly the same, other than the few changes stated above.<br>
This has given me some improvement in the results.</p>
<p>Positive images hover around 70%-75%, Negative images results are typically &lt;10%.<br>
(The plotted loss graph still looks horrendous)</p>
<p>My question is: Is this right? Although I see improvements I don’t exactly understand why.<br>
Is it because I “added” more training data?<br>
Is the change from <code>torch::nn::functional::binary_cross_entropy</code> to <code>torch::nn::BCELoss</code> <em>that</em> drastic?<br>
Or is <code>model.train()</code> that important?</p>