Need Help Troubleshooting CNN Implementation

username: Martijn152
cooked: <p>Hello!</p>
<p>I’ve been working on an implementation of a paper titled <em>“Deep Learning-Based Crack Damage Detection Using Convolutional Neural Networks”</em> by Cha et al. for a couple of days now. In the paper they find some very high accuracy rates, but I have not managed to come even close to those. I’d like some help troubleshooting since I am still pretty new to pytorch.</p>
<p>The proposed CNN is described by this diagram and table:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/4/14925d819dc1d3c83ea40c0ec729ceddbd1a7908.png" alt="screenshot" data-base62-sha1="2VZ8oYE7MphjwKOspNT2IG3AcNG" width="545" height="396"></p>
<p>The paper uses a data set of 40.000 images, of which 32.000 training and 8.000 validation. Of these, the ratio between images with cracks and without cracks is 1:1. Images are cropped to 256x256.<br>
Some other given values are:</p>
<ul>
<li>Weight decay: 0.0001</li>
<li>Momentum: 0.9</li>
<li>Minibatch size: 100 (out of 40K)</li>
<li>Dropout rate: 0.5</li>
<li>Learning rate: Starts at 0.01 and logarithmically decreases (I haven’t been able to implement this)</li>
</ul>
<p>SGD is used for the optimizer.</p>
<p>My network looks like the following:</p>
<pre><code class="lang-auto">class Net(nn.Module):


    def __init__(self):
        super(Net, self).__init__()
        self.C1 = nn.Conv2d(in_channels=3, out_channels=24, kernel_size=20, stride=2)
        self.P1 = nn.MaxPool2d(kernel_size=7, stride=2)
        self.C2 = nn.Conv2d(in_channels=24, out_channels=48, kernel_size=15, stride=2)
        self.P2 = nn.MaxPool2d(kernel_size=4, stride=2)
        self.C3 = nn.Conv2d(in_channels=48, out_channels=96, kernel_size=10, stride=2)
        self.C4 = nn.Conv2d(in_channels=96, out_channels=2, kernel_size=1, stride=1)

        # The final layer is a softmax layer
        self.Softmax = nn.Softmax(dim=1)

        self.BN1 = nn.BatchNorm2d(24)
        self.BN2 = nn.BatchNorm2d(48)
        self.BN3 = nn.BatchNorm2d(96)
        
        self.ReLU = nn.ReLU()
        
        self.Dropout = nn.Dropout2d(0.5)


    def forward(self, x):
        # Going to take it step by step here to be able to check the values between layers
        # This is the order of layers specified in table 1.
        x = self.C1(x)
        x = self.BN1(x)
        x = self.P1(x)
        x = self.C2(x)
        x = self.BN2(x)
        x = self.P2(x)
        x = self.C3(x)
        x = self.BN3(x)
        x = self.Dropout(x)
        x = self.ReLU(x)
        x = self.C4(x)
        x = self.Softmax(x)
        return x
</code></pre>
<p>And another relevant snippet:</p>
<pre><code class="lang-auto">optimizer = optim.SGD(net.parameters(), lr=0.005, momentum=0.9, weight_decay=0.0001)
</code></pre>
<p>The results I am getting are around 77% accuracy at the highest versus the paper getting in the high 90s. This accuracy decreases after reaching that highest point, probably due to overfitting.<br>
I also suspect my data set is a bit more varied than the one used in the paper, although the one in the paper is not explicitly shown anywhere. It is also only about 2.300 photos versus the 40.000 of the paper.</p>
<p>Any thoughts on what might be going wrong? Or do I have unrealistic expectations for getting higher accuracies?</p>
<p>Thanks in advance!</p>

username: ibro45
cooked: <p>You dataset is much smaller than theirs, I would say that that’s why you can’t reach comparable results</p>

username: Martijn152
cooked: <p>I appreciate the suggestion, and I do think it definitely makes a difference, but the paper gives some interesting numbers on the success rates with smaller data sets as well:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74e601729bda756b42b9866c50aef6390ae4c552.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/74e601729bda756b42b9866c50aef6390ae4c552" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74e601729bda756b42b9866c50aef6390ae4c552_2_690x469.jpeg" alt="image" data-base62-sha1="gG8aCV4k6m6JX66T79WCbByPUwa" width="690" height="469" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74e601729bda756b42b9866c50aef6390ae4c552_2_690x469.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/7/4/74e601729bda756b42b9866c50aef6390ae4c552.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/4/74e601729bda756b42b9866c50aef6390ae4c552.jpeg 2x" data-dominant-color="E3E3E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">990×674 135 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
So I’d think it would be possible with a smaller data set to at least get a bit higher.</p>

username: ibro45
cooked: <p>Nice that they included such a plot!</p>
<p>I looked at it more closely, everything seems to be alright except for one detail, the convolutional layers that are followed by batch normalization should have <code>bias=False</code>, explained briefly <a href="https://discuss.pytorch.org/t/any-purpose-to-set-bias-false-in-densenet-torchvision/22067/2">here</a>.</p>

username: ibro45
cooked: <p>Also, try implementing the logarithmically decreasing lr with <a href="https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.ExponentialLR" rel="nofollow noopener">https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.ExponentialLR</a>. It will most likely help you get closer to where you want to be</p>