VGG isn't training on CIFAR

Flock1: <p>I am trying to train <code>VGG1</code> on <code>CIFAR-10</code> dataset but for some reason, it’s not training. I have tried to make sure of all the major issues like how not to add <code>softmax</code> activation after the linear layer. Even cross-checked the outputs. This is how I have defined the network:</p>
<pre><code class="lang-auto">cifar_class = torchvision.models.vgg11(pretrained=False)
cifar_class.classifier.add_module('7', nn.Linear(in_features=1000, out_features=10,bias=True))
cifar_class.features.add_module('0', nn.Conv2d(in_channels=1, out_channels=64, kernel_size=3, stride=1, padding=1, bias=True))
cifar_class = cifar_class.to(device)
</code></pre>
<p>I want to use grayscale images and I am loading the dataset like this:</p>
<pre><code class="lang-auto">training_data = datasets.CIFAR10(root="data", train=True, download=True,
                                  transform=transforms.Compose([
                                      transforms.ToTensor(),
                                      transforms.Grayscale(num_output_channels=1)
                                  ]))

validation_data = datasets.CIFAR10(root="data", train=False, download=True,
                                  transform=transforms.Compose([
                                    transforms.ToTensor(),
                                    transforms.Grayscale(num_output_channels=1)
                                  ]))
</code></pre>
<p>And finally, the training process:</p>
<pre><code class="lang-auto">for epoch in range(10):  # loop over the dataset multiple times

    running_loss = 0.0
    running_corrects = 0
    for i,data in enumerate(tqdm(training_loader)):
        # get the inputs; data is a list of [inputs, labels
        inputs, labels = data[0].cuda(), data[1].cuda()

        # zero the parameter gradients
        optimizer.zero_grad()

        outputs = cifar_class(inputs)
        _, preds = torch.max(outputs, 1)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item() * inputs.size(0)
        running_corrects += torch.sum(preds == labels.data)
    print('[%d, %5d] loss: %.3f acc: %.3f' %
              (epoch + 1, i + 1, running_loss / 50000, running_corrects.double() / 50000))
    running_loss = 0.0
    running_corrects = 0

print('Finished Training')
</code></pre>
<p>What am I doing wrong?</p>

Andrei_Cristea: <p>Hello Flock!</p>
<p>Could you include your training loader, optimizer, and loss function definitions, for replication?</p>
<p>Thanks!<br>
Andrei</p>

Flock1: <p>Here’s the code:</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(cifar_class.parameters(), lr=0.001)
exp_lr_scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)
</code></pre>
<p>Here is the loader:</p>
<pre><code class="lang-auto">training_loader = DataLoader(training_data, 
                             batch_size=16, 
                             shuffle=True,
                             pin_memory=True)

validation_loader = DataLoader(validation_data,
                               batch_size=16,
                               shuffle=True,
                               pin_memory=True)
</code></pre>

Andrei_Cristea: <p>Seems like you forgot to Resize your input to what VGG normally expects (224 x 224). There are some other small issues, e.g. you didn’t Normalize your input, but from quickly playing around with it, the lack of Resize seems to be the main culprit.</p>
<p>Here I am (over)fitting on just the first few batches, with <code>shuffle=False</code>, and you can clearly see the difference in loss convergence.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c43c2f6c1c4943ec5f5aebee9e898d08e30648be.png" alt="image" data-base62-sha1="rZYwTelXk0PtQyTCZsLWidrNQN8" width="307" height="207"></p>
<p>Btw, the jumpy loss (for “with resize”) is expected since there are some <code>Dropout</code> layers in there (which normally get turned off for validation) so it’ll never settle on exactly zero.</p>
<p>This is how you resize:</p>
<pre><code class="lang-auto">transform=transforms.Compose([
    transforms.Resize(224),  # this!
    transforms.ToTensor(),
    # transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),  you may want to add this too
    transforms.Grayscale(num_output_channels=1),
]))
</code></pre>
<p>Hope this helps!</p>

Flock1: <p>Hmmm. I don’t know how will resizing help, considering the size of CIFAR images is only <code>32X32</code>. The normalizing part I’ll try. I used <a href="https://github.com/chengyangfu/pytorch-vgg-cifar10" rel="noopener nofollow ugc">this</a> as a reference and I don’t see re-sizing but I do see normalizing.</p>
<p>What do you think will happen if <code>32X32</code> is resized to <code>224X224</code>?</p>

Andrei_Cristea: <p>I think the kernels of the convolutions are sized according to some assumption about what area of the total input they cover. When the total input is much smaller, the kernels may be too large to capture the kind of features they ought to. You don’t add any <em>information</em> by upsampling from 32 to 224, but it’s possible that the “resolution” of the convolutions is inappropriate unless the input is 224. I find it pretty convincing that toggling that one knob was enough to make the whole thing work, in the example I showed you.</p>
<p>(note another difference between your training and the one from your reference, they use SGD with weight decay and momentum, you use Adam without weight decay. in my experiments that also helped a little)</p>

Flock1: <p>That’s interesting. I’ll try this.</p>
<aside class="quote no-group" data-username="Andrei_Cristea" data-post="6" data-topic="150547">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrei_cristea/48/47843_2.png" class="avatar"> Andrei_Cristea:</div>
<blockquote>
<p>they use SGD with weight decay and momentum, you use Adam without weight decay</p>
</blockquote>
</aside>
<p>I also thought of this. I’ll change it and see if it helps. Thanks though.</p>

Flock1: <p>I want to apologize for this thread. I don’t know what happened to me that I forgot some basic deep learning troubleshooting. Changing to SGD worked</p>