Pytorch net from: Striving for Simplicity: The All Convolutional Net

Brando_Miranda: <p>Is there an implementation of:</p>
<p><a href="https://arxiv.org/abs/1412.6806" class="onebox" target="_blank" rel="nofollow noopener">https://arxiv.org/abs/1412.6806</a></p>
<p>in pytorch?</p>

SimonW: <p>I’ve used this, and it works fine on Cifar10</p>
<pre><code class="lang-auto">class AllConvNet(nn.Module):

    def __init__(self, dropout=True, nc=3, num_classes=10):
        super(AllConvNet, self).__init__()
        self.dropout = dropout
        self.conv1 = nn.Conv2d(nc, 96, 3, padding=1)
        self.conv2 = nn.Conv2d(96, 96, 3, padding=1)
        self.conv3 = nn.Conv2d(96, 96, 3, padding=1, stride=2)
        self.conv4 = nn.Conv2d(96, 192, 3, padding=1)
        self.conv5 = nn.Conv2d(192, 192, 3, padding=1)
        self.conv6 = nn.Conv2d(192, 192, 3, padding=1, stride=2)
        self.conv7 = nn.Conv2d(192, 192, 3, padding=1)
        self.conv8 = nn.Conv2d(192, 192, 1)
        self.class_conv = nn.Conv2d(192, num_classes, 1)


    def forward(self, x):
        if self.dropout:
            x = F.dropout(x, .2)
        conv1_out = F.relu(self.conv1(x))
        conv2_out = F.relu(self.conv2(conv1_out))
        conv3_out = F.relu(self.conv3(conv2_out))
        if self.dropout:
            conv3_out = F.dropout(conv3_out, .5)
        conv4_out = F.relu(self.conv4(conv3_out))
        conv5_out = F.relu(self.conv5(conv4_out))
        conv6_out = F.relu(self.conv6(conv5_out))
        if self.dropout:
            conv6_out = F.dropout(conv6_out, .5)
        conv7_out = F.relu(self.conv7(conv6_out))
        conv8_out = F.relu(self.conv8(conv7_out))

        class_out = F.relu(self.class_conv(conv8_out))
        pool_out = class_out.reshape(class_out.size(0), class_out.size(1), -1).mean(-1)
        return pool_out
</code></pre>
<p>it was written based on <a href="https://github.com/StefOe/all-conv-pytorch/blob/master/allconv.py">https://github.com/StefOe/all-conv-pytorch/blob/master/allconv.py</a></p>

Brando_Miranda: <p>Beautiful Simon, Thanks you so much for sharing! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=5" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>Did you get like 95% accuracy on Cifar10? Do u remember ur accuracies/errors?</p>
<p>Do you remember how long it took to run btw?</p>

SimonW: <p>I used it in some sort of an unconventional way, without dropout, trained for very few epochs w/o lr decay, etc. But I remember getting 90% even with these conditions. I didn’t track the time, but it didn’t take too long iirc.</p>

Brando_Miranda: <p>How would you suggest to use BN on this? After every single layer? Seems like a lot of batch norm…</p>

Brando_Miranda: <p>interesting, I’m not sure what Im doing wrong, but I don’t get it to go too far away from chance…I think the only thing Im not the same as the hyperparams of from the website you shared is the batch size being size 32, Im using larger 2**10 I think but that shouldn’t matter too much…right?</p>

SimonW: <p>I’m not sure how to apply BN on this, but it may not be necessary as it is not too deep.</p>
<p>Did you make sure that you normalize the image correctly? Maybe tune the lr?</p>

Brando_Miranda: <p>Ok, I will try a few more learning rates why not…</p>
<p>I am using the preprocessing from the standard pytorch cifar10 tutorial…the paper/that code is not using that, its using instead:</p>
<pre><code class="lang-auto">train_mean = trainset.train_data.mean(axis=(0,1,2))/255  # [0.49139968  0.48215841  0.44653091]
train_std = trainset.train_data.std(axis=(0,1,2))/255  # [0.24703223  0.24348513  0.26158784]
transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(train_mean, train_std),
])

transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(train_mean, train_std),
])
</code></pre>
<p>while I am using (from the cifar 10 tutorial):</p>
<pre><code class="lang-auto">    transform = []
    ''' converts (HxWxC) in range [0,255] to [0.0,1.0] '''
    to_tensor = transforms.ToTensor()
    transform.append(to_tensor)
    ''' Given meeans (M1,...,Mn) and std: (S1,..,Sn) for n channels, input[channel] = (input[channel] - mean[channel]) / std[channel] '''
    if standardize:
        gaussian_normalize = transforms.Normalize( (0.5, 0.5, 0.5), (0.5, 0.5, 0.5) )
        transform.append(gaussian_normalize)
    ''' transform them to Tensors of normalized range [-1, 1]. '''
    transform = transforms.Compose(transform)
</code></pre>
<p>I guess since its not the same data pre-processing maybe I need to play around with the lr and scheduler more on my own…I will try that! thanks for the ideas.</p>

Brando_Miranda: <p>ok now I need to try their pre-processing but I find it really worrying that its so sensitive to everything! The pre-processing doesn’t even seem that different to me, 0.5 vs 0.2 etc ? bad signs for re-producibility… <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=5" title=":confused:" class="emoji" alt=":confused:"></p>
<p>Thanks for the help though.</p>

SimonW: <p>I won’t say it is bad sign for reproducibility. Arch, normalization method, lr, optimization algorithm, weight init, they all matter. With a different set of normalization constants, you will definitely need to tune other things too to make it work. Unfortunately that is the current state of DL.</p>

Brando_Miranda: <p>this is really weird, I think I’ve used all the hyperparams and preprocessing the link StefOe provided but I can’t get lower than 0.2 error <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=5" title=":frowning:" class="emoji" alt=":frowning:"> I wonder what I’m doing wrong…</p>

Brando_Miranda: <aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/51528464/how-can-one-get-state-of-the-art-results-on-cifar10-with-the-all-convolution-net" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1601580/charlie-parker" target="_blank" rel="nofollow noopener">
    <img alt="Charlie Parker" src="https://i.stack.imgur.com/2B6rV.png?s=128&amp;g=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/51528464/how-can-one-get-state-of-the-art-results-on-cifar10-with-the-all-convolution-net" target="_blank" rel="nofollow noopener">How can one get state of the art results on Cifar10 with the All Convolution Net?</a>
</h4>

<div class="tags">
  <strong>machine-learning, neural-network, conv-neural-network, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/1601580/charlie-parker" target="_blank" rel="nofollow noopener">
    Charlie Parker
  </a>
  on <a href="https://stackoverflow.com/questions/51528464/how-can-one-get-state-of-the-art-results-on-cifar10-with-the-all-convolution-net" target="_blank" rel="nofollow noopener">10:19PM - 25 Jul 18</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ttoosi: <p>Any updates? I am trying to train AllConvNet on CIFAR10, no success. It is strange.</p>

Originofamonia: <p>Hello,</p>
<p>I tried to train this network. However, the loss quickly converge to 2.3026, then the loss doesn’t descent. Any ideas on how to solve it?<br>
When I tried to debug it, I found the output of the network is all zeros. The output is [batch_size, 10], which is all zeros.</p>
<p>Thanks!</p>