Problem using nn.Sequential

eagle.py: <p>While trying to train a model for the CIFAR10 datset I encountered a problem using nn.Sequential.</p>
<p>When I train the following model implementation everything goes as expected. The loss drops and the accuracy increases:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        print("After conv_block_1:", x.shape)

        x = self.pool(F.relu(self.conv2(x)))
        print("After conv_block_2:", x.shape)

        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
</code></pre>
<p>However when i try to implement the same model using nn.Sequential, the loss stays the same after every epoch:</p>
<pre><code class="lang-auto">class NetV3(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv_block = nn.Sequential(
            nn.Conv2d(3, 6, 5),
            nn.ReLU(),
            nn.MaxPool2d(2, 2),
            nn.Conv2d(6, 16, 5),
            nn.ReLU(),
            nn.MaxPool2d(2, 2)
        )
        
        self.classifier = nn.Sequential(
            nn.Linear(16 * 5 * 5, 120),
            nn.ReLU(),
            nn.Linear(120, 84),
            nn.ReLU(),
            nn.Linear(84, 10)
        )

    def forward(self, x):
        x = self.conv_block(x)
        print("After conv_block:", x.shape) 
        x = torch.flatten(x, 1)
        x = self.classifier(x)
        return x
</code></pre>
<p>the training loop i used is the following:</p>
<pre><code class="lang-auto">for epoch in range(3):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.3f}')
            running_loss = 0.0

print('Finished Training')
</code></pre>

KFrank: <p>Hi Ektoras!</p>
<aside class="quote no-group quote-modified" data-username="eagle.py" data-post="1" data-topic="219516" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/7c8e57/48.png" class="avatar"> eagle.py:</div>
<blockquote>
<p>However when i try to implement the same model using nn.Sequential, the loss stays the same after every epoch:</p>
<pre><code class="lang-auto">        if i % 2000 == 1999:    # print every 2000 mini-batches
            print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.3f}')
</code></pre>
</blockquote>
</aside>
<p>I don’t see any difference between your <code>Net</code> and <code>NetV3</code> models.  Unless I’m missing<br>
something, they should be the same.</p>
<p>First, start by printing out <code>loss</code> (which hides less information than <code>running_loss</code>) and<br>
<code>running_loss</code> after every batch (not just starting with the 2000th batch).</p>
<p>Also try using something like <code>torch.manual_seed (20001999)</code> to make sure that your<br>
two versions of <code>Net</code> have their layers initialized identically.  Also, if your <code>trainloader</code><br>
uses any randomness (which it normally would), make sure that your two runs use the<br>
same <code>trainloader</code> random sequence.</p>
<p>It could simply be that your two training runs don’t match one another (because of the<br>
initialization and <code>trainloader</code> randomness) and that the <code>NetV3</code> run happens to plateau,<br>
while the <code>Net</code> training run doesn’t.</p>
<p>If you initialize your two models identically and pass them the same data, their outputs<br>
should also be identical (or possibly differ by floating-point round-off error).  Verify that<br>
this is the case with a single forward pass.  Then check that your first backward pass and<br>
optimization step also yield the same results, and so on.</p>
<p>Best.</p>
<p>K. Frank</p>