Can pytorch provide some sample code on training ResNet?

BarCodeReader: <p>I recently need to train ResNet50 and do some experiments, I know there are bunch of pretrained models on github, but I feel more interested on the training process(like how to preprocess, set the LR and so on)…</p>
<p>The dataset I am using is standard CIFAR100. On github, few repo claim that they can achieve top-1 error of 22%(i.e. accuracy 78%) on ResNet50.</p>
<p>I just simply cannot get good results by using ResNet, my best record is only 66%, I read the paper, they warm up the learning rate, which I also do…I feel ResNet is really hard to train and get good results.</p>
<p>So I am wondering…it’s ok if we dont’t have any pretrained model on these kinds of datasets, but is it possible that pytorch can provide some training scirpts or preprocessing scripts for us on these datasets…we can modify and train on our own.</p>

Sunshine352: <p>You can refer to this,  <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py</a></p>

BarCodeReader: <p>this is something too basic…<img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

Sunshine352: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/pytorch/examples/tree/master/imagenet" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;v=4" class="thumbnail" width="" height="">

<h3><a href="https://github.com/pytorch/examples/tree/master/imagenet" target="_blank" rel="nofollow noopener">pytorch/examples</a></h3>

<p>A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc. - pytorch/examples</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Kushaj: <p>Review the <a href="https://arxiv.org/abs/1812.01187" rel="nofollow noopener">Bag of Tricks for Image Classification with Convolutional Neural Networks</a> for some pointers.</p>
<p><strong>Preprocessing</strong>: Zeros padding with value=4 and then randomly crop a 32x32 image. For normalization use mean=[0.491, 0.482, 0.447] and std=[0.247, 0.243, 0.261]. For data augmentation, use horizontal flip, maybe rotate. There are a lot of options here but these are the basic ones.</p>
<p><strong>Learning Rate</strong>: Assuming you are starting with random weights. Initialization the weights using He init. Do warmup for first 5 epochs (linearly increase learning rate from 0 to final value with constant step rate). Then you can use the step decy method, where the learning rate is dropped by 10 after 30 epochs. Cosine annealing is also a good alternative.<br>
I generally prefer <a href="https://nbviewer.jupyter.org/github/KushajveerSingh/Deep-Learning-Notebooks/blob/master/Leslie%20N.%20Smith%20paper%20notebooks%20%28fastai%29/main.ipynb" rel="nofollow noopener">cyclic learning</a>.</p>
<p>Use batch size = 32. You can use higher but it would be sufficient. Use SGD+momentum as the optimizer. You can also test AdamW.</p>
<p>There are many more techniques like Multi-Sample Dropout, Mish activation function, LookAhead Optimizer with RAdam, multi sample data augmentations.</p>
<p>The implementation is easy, just start implementing one step and it would be fine.</p>

BarCodeReader: <p>Thanks for the suggestion.</p>
<p>I did the preprocessing you mention, also warm up the learning for first 5 epoch, I even tried warm up 20 epochs. Batch size for me is 128 since I feel it is more stable than 64 and 32. optimizer for me is also the same…SGD+momentum… for training scheme is step decay with factor 0.2 on steps of [60,120,160] with initial LR of 0.1</p>
<p>so …ya still cannot reach 70% and I tried this for few weeks already…<img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
<p>ya i should try to initialize the weight differently. see if it will help.</p>

Kushaj: <p>That will not help much, if you used He init.</p>

BarCodeReader: <p>it helped <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"> thanks for your remind. now the model works well.</p>

JTiC: <p>You meant when use “He init”, your model reached 70% from 66%?</p>

BarCodeReader: <p>not that dramatic…I update all my conda package to latest, and introduce some data augmentation and it reach 68.3%, then I try ‘He init’ it gives me 70.2% as top-1 accuracy, which is quite close to 22-28% top-1 error reported by almost all github repos…<br>
the benefit of "he init’’ to me is like the model converges more stable, because my previous training the model is quite unstable and even with warm up the accuracy gets stuck…</p>