Simple CNN for object counting only works with batch size 1

username: Enric_Moreu
cooked: <p>Hello!</p>
<p>For the last 2 weeks I’ve been stuck trying to count balls from a synthetic dataset I generated.<br>
When I set a batch size higher than 1, the network predicts the average value all the time.<br>
Otherwise, the network works great in train/val</p>
<ul>
<li>Dataset: 5000 images with grey background and blue balls of the same size around. There are between 1 and 10 balls per image.</li>
<li>Optimizer: Adam with learning rate of 1e-4</li>
<li>Normalization: for the normalization of the images I used:</li>
</ul>
<pre><code class="lang-auto">transforms.Compose([
        transforms.Resize((128,128)),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5],
                             [0.5, 0.5, 0.5])
    ])
</code></pre>
<ul>
<li>Loss: L1 (Mean absolute error)</li>
<li>Network:</li>
</ul>
<pre><code class="lang-auto">class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(3, 6, 5),
            nn.ReLU(),
            nn.MaxPool2d(2, 2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(6, 15, 5),
            nn.ReLU(),
            nn.MaxPool2d(2, 2))
        self.dropout = nn.Dropout(0.2)
        self.fc1 = nn.Linear(12615, 1000)
        self.fc2 = nn.Linear(1000, 1)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.dropout(out)
        out = self.fc1(out)
        out = self.dropout(out)
        out = self.fc2(out)
        return out
</code></pre>
<p>I am bit frustrated because this is the easiest example of object counting and I can not make it work properly.</p>
<p>How can I avoid predicting the average value (5) when raising the batch size?</p>
<p>PD: I tried many things: cyclic LR, Batchnorm2d, deeper networks, all the possible optimizers…</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Are you using the summed loss or the mean (which is the default)?<br>
Also, did you try to use <code>MSELoss</code> instead of <code>L1Loss</code>?</p>
<p>Since you have a fixed number of balls (1 to 10), you could also see this use case as a multi-class classification and use <code>nn.CrossEntropyLoss</code> instead. However, extrapolating wouldn’t work, if that’s in your use case.</p>

username: Enric_Moreu
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for the quick reply!</p>
<p>Can you explain how to use a summed loss?<br>
I think that in my case I just average all the losses:</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    print('Epoch {}/{}'.format(epoch, num_epochs - 1))
    print('-' * 10)

    for phase in ['train', 'val']:
        if phase == 'train':
            model.train()
        else:
            model.eval()

        running_loss = 0.0
        successes = 0.0
        total = 0.0

        for inputs, labels in tqdm(dataloaders[phase]):
            inputs = inputs.to(device)
            labels = labels.to(device)
            
            # Set gradients to zero
            optimizer.zero_grad()

            with torch.set_grad_enabled(phase == 'train'):
                outputs = model(inputs)

                # Counting L1 loss
                loss =  criterion(outputs, labels.float())
</code></pre>
<p>I tried <code>MSELoss</code> with the same results.<br>
I also considered <code>nn.CrossEntropyLoss</code> but it will apply the same penalty to an error of 7 (instead of 8) than an error of 1 (instead of 8). I agree it may work for this example, but it won’t work if I scale the difficulty of the task.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>You would have to explicitly set it in the creation of the criterion via <code>reduction='sum'</code>.<br>
Weird behavior, indeed.<br>
Could you add a <code>F.relu</code> between the last two linear layers?<br>
Also, how large are your images?</p>

username: Enric_Moreu
cooked: <p>The results with <code>reduction='sum'</code> and <code>reduction='mean'</code> are slightly the same (I had to adapt the way I was displaying the loss). The success rate still stuck at 10% (predicting only 5s)<br>
The <code>F.relu</code> didn’t provide any significant improvement.<br>
The images are 128x128, it’s just a dataset I made for testing.</p>
<p>What else can I try? How can I debug where is the problem? Honestly, I am running out of ideas.</p>
<p>Thanks again</p>

username: ptrblck
cooked: <p>I would try to scale down the problem size and try to overfit a smaller data sample.<br>
E.g. use only 10 batches of the desired batch size and try to make your model converge.</p>
<p>Also, another maybe “unusual” value is the high number of input features to the first linear layer.<br>
I would try to lower this number by either resizing the images or adding another conv/pool layer.</p>

username: Enric_Moreu
cooked: <p>I am using only 44 samples (and batch size 4) but I still can’t overfit the model when BS &gt; 1<br>
I resized the images to 64x64 and now the input to the FC is smaller:</p>
<pre><code class="lang-auto">class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(3, 6, 5),
            nn.ReLU(),
            nn.MaxPool2d(2, 2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(6, 15, 5),
            nn.ReLU(),
            nn.MaxPool2d(2, 2))
        self.dropout = nn.Dropout(0.2)
        self.fc1 = nn.Linear(2535, 512)          
        self.fc2 = nn.Linear(512, 1)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.dropout(out)
        out = self.fc1(out)
        out = F.relu(out)
        out = self.dropout(out)
        out = self.fc2(out)
        return out
</code></pre>
<p>Unfortunately, the model keeps predicting the average value.</p>
<p>Do you know any examples of object counting for computer vision?</p>
<p>All the object counting examples I found (Surprisingly, 99% of Pytorch examples are classifiers) on the internet use a detector (such as YOLO) or density maps (for crowd counting). Maybe density maps is the way to go.<br>
Still disappointing that I can’t solve such an easy task of counting balls with a Pytorch CNN.</p>
<p>Thanks for your help <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: Enric_Moreu
cooked: <p>I published the clean code <a href="https://github.com/enric1994/cnn_counting" rel="nofollow noopener">here</a> as a showcase of the issue.</p>

username: ptrblck
cooked: <p>Thanks for the executable code, that was really helpful.</p>
<p>You are accidentally broadcasting the loss, since you have a mismatch in the output and target tensors.<br>
While your output has the shape <code>[batch_size, 1]</code>, the target has <code>[batch_size]</code>.</p>
<p>This yields to a broadcasting as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python"># your code with the broadcasting
output = torch.randn(4, 1)
target = torch.randn(4)

criterion = nn.L1Loss(reduction="none")

loss = criterion(output, target)
print(loss) # you only want the diagonal
&gt; tensor([[1.0231, 2.3743, 2.4857, 2.3248],
        [1.5896, 0.2385, 0.1270, 0.2879],
        [1.7572, 0.4061, 0.2946, 0.4555],
        [1.2650, 0.0862, 0.1976, 0.0368]])

# fixed
target = target.unsqueeze(1)
loss = criterion(output, target)
print(loss) 
&gt; tensor([[1.0231],
        [0.2385],
        [0.2946],
        [0.0368]])
</code></pre>
<p>You should also get a warning such as:</p>
<blockquote>
<p>UserWarning: Using a target size (torch.Size([4])) that is different to the input size (torch.Size([4, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.</p>
</blockquote>
<p>Use this line of code to calculate the loss and it should work:</p>
<pre data-code-wrap="python"><code class="lang-python">loss =  criterion(outputs, labels.float().unsqueeze(1))
</code></pre>
<aside class="quote no-group" data-username="Enric_Moreu" data-post="7" data-topic="82450">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/enric_moreu/48/24605_2.png" class="avatar"> Enric_Moreu:</div>
<blockquote>
<p>Still disappointing that I can’t solve such an easy task of counting balls with a Pytorch CNN.</p>
</blockquote>
</aside>
<p>We were all in the same situation, so please don’t be disappointed. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

username: Enric_Moreu
cooked: <p>Thanks a million! I knew it was a silly mistake!</p>