ResNet with CIFAR10 only reaches 86% accuracy (expecting >90%)

etrommer: <p>Hello everyone,</p>
<p>I am trying to reproduce the numbers from the original ResNet publication on CIFAR10. I am using the network implementation from here:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/akamaster/pytorch_resnet_cifar10/blob/master/resnet.py">
  <header class="source">

      <a href="https://github.com/akamaster/pytorch_resnet_cifar10/blob/master/resnet.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/akamaster/pytorch_resnet_cifar10/blob/master/resnet.py" target="_blank" rel="noopener nofollow ugc">akamaster/pytorch_resnet_cifar10/blob/master/resnet.py</a></h4>


      <pre><code class="lang-py">'''
Properly implemented ResNet-s for CIFAR10 as described in paper [1].

The implementation and structure of this file is hugely influenced by [2]
which is implemented for ImageNet and doesn't have option A for identity.
Moreover, most of the implementations on the web is copy-paste from
torchvision's resnet and has wrong number of params.

Proper ResNet-s for CIFAR10 (for fair comparision and etc.) has following
number of layers and parameters:

name      | layers | params
ResNet20  |    20  | 0.27M
ResNet32  |    32  | 0.46M
ResNet44  |    44  | 0.66M
ResNet56  |    56  | 0.85M
ResNet110 |   110  |  1.7M
ResNet1202|  1202  | 19.4m

which this implementation indeed has.
</code></pre>



  This file has been truncated. <a href="https://github.com/akamaster/pytorch_resnet_cifar10/blob/master/resnet.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>As far as I can tell, I am using the exact training parameters that are given in the paper:</p>
<blockquote>
<p>We use a weight decay of 0.0001 and momentum of 0.9,<br>
and adopt the weight initialization in [13] and BN [16] but<br>
with no dropout. These models are trained with a mini-<br>
batch size of 128 on two GPUs. We start with a learning<br>
rate of 0.1, divide it by 10 at 32k and 48k iterations, and<br>
terminate training at 64k iterations, which is determined on<br>
a 45k/5k train/val split. We follow the simple data augmen-<br>
tation in [24] for training: 4 pixels are padded on each side,<br>
and a 32×32 crop is randomly sampled from the padded<br>
image or its horizontal flip.</p>
</blockquote>
<p>Here are the relevant parts of my training script:</p>
<pre><code class="lang-auto">class ResNet(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.model = resnet.resnet20()

    def forward(self, x): 
        return self.model(x)

    def configure_optimizers(self):
        optimizer = optim.SGD(self.parameters(), lr=0.1, momentum=0.9, weight_decay=1e-4)
        scheduler = optim.lr_scheduler.MultiStepLR(optimizer, milestones=[90,135])
        return {'optimizer': optimizer, 'scheduler': scheduler}
...

class CIFAR10DataModule(pl.LightningDataModule):
    def __init__(self, **kwargs):
        ...
        self.normalize = [
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.491, 0.482, 0.447], std=[0.247, 0.243, 0.262]),
        ]
        self.augment = [
            transforms.RandomCrop(32, padding=4),
            transforms.RandomHorizontalFlip(),
        ]   
            
    def prepare_data(self):
        torchvision.datasets.CIFAR10(root=self.data_dir, train=True, download=True)
        torchvision.datasets.CIFAR10(root=self.data_dir, train=False, download=True)
            
    def setup(self, stage=None):
        if stage == 'fit' or stage is None:
            t = transforms.Compose(self.augment + self.normalize)
            cifar_full = torchvision.datasets.CIFAR10(root=self.data_dir, train=True, transform=t)
            self.df_train, self.df_val = torch.utils.data.random_split(cifar_full, [45000, 5000])
        if stage == 'test' or stage is None:
            t = transforms.Compose(self.normalize)
            self.df_test = torchvision.datasets.CIFAR10(root=self.data_dir, train=False, transform=t)

dm = CIFAR10DataModule(batch_size=128, num_workers=4)
dm.setup("fit")
lr_logger = pl.callbacks.LearningRateMonitor()
mlflow.pytorch.autolog()
model = ResNet()
trainer = pl.Trainer(gpus=1, max_epochs=180, callbacks=[lr_logger])
trainer.fit(model, dm) 
</code></pre>
<p>However the accuracy only reaches around 86%, well below the 91.25% given in the original paper.</p>
<p>There is a comment in the repository that hosts the ResNet/CIFAR10 model which indicates that this issue seemed to occur after an update of PyTorch from version 1.1 to 1.2:</p><aside class="onebox githubissue" data-onebox-src="https://github.com/akamaster/pytorch_resnet_cifar10/issues/18">
  <header class="source">

      <a href="https://github.com/akamaster/pytorch_resnet_cifar10/issues/18" target="_blank" rel="noopener nofollow ugc">github.com/akamaster/pytorch_resnet_cifar10</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/akamaster/pytorch_resnet_cifar10/issues/18" target="_blank" rel="noopener nofollow ugc">Reproduce "test" accuracy</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-03-26" data-time="19:12:22" data-timezone="UTC">07:12PM - 26 Mar 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-03-27" data-time="21:57:14" data-timezone="UTC">09:57PM - 27 Mar 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/tbachlechner" target="_blank" rel="noopener nofollow ugc">
          <img alt="tbachlechner" src="https://avatars.githubusercontent.com/u/35582223?v=4" class="onebox-avatar-inline" width="20" height="20">
          tbachlechner
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">I'm having trouble reproducing the test accuracy that you quote in the Readme. I<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">n particular running your precise code for ResNet56 I get the best validation error rate of 7.36(16)%. This differs from your quoted value of 6.61% by 5 sigma. How exactly did you determine the quoted test accuracy of your model?</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
(note that the reported numbers in the issue refer to ResNet56, but the effect is the same, just less pronounced)</p>
<p>I suspect that I am seeing the same issue and would like to understand what is causing it and how I can best fix it.</p>

etrommer: <p>Note that this is validation accuracy, not test accucary. Now that I am thinking about it, I am wondering whether the drop in accuracy I am seeing is a side effect of the augmentation. I will report a value on the test set tomorrow!</p>

etrommer: <p>I have evaluated against the test set and the effect stays the same: Training accuracy reaches around 93%, but test accuracy stagnates at around 85%.</p>

etrommer: <p>I have found the issue, and it is a very subtle one:</p>
<p>When returning a scheduler to Lightning using the dict format like I do in this line:</p>
<pre><code class="lang-auto">        return {'optimizer': optimizer, 'scheduler': scheduler}
</code></pre>
<p>the keyword for the scheduler needs to be <code>lr_scheduler</code>, otherwise it will not be picked up and the learning rate will stay high.</p>

s17861418198: <p>i want use imagerfloder get cifar10，but the accuracy reaches 85%，expecting&gt;96%</p>