How to use residual learning applied to fully connected networks?

Wesley_Neill: <p>Is there any reason why skip connections would not provide the same benefits to fully connected layers as it does for convolutional?</p>
<p>I’ve read the ResNet paper and it says that the applications should extend to “non-vision” problems, so I decided to give it a try for a tabular data project I’m working on.</p>
<p><strong>Try 1:</strong> My first try was to only skip connections when the input to a block matched the output size of the block (the block has <code>depth - 1</code> number of layers with <code>in_dim</code> nodes plus a layer with <code>out_dim</code> nodes :</p>
<pre><code class="lang-auto">class ResBlock(nn.Module):
    def __init__(self, depth, in_dim, out_dim, act='relu', act_first=True):
        super().__init__()
        self.residual = nn.Identity()
        self.block = block(depth, in_dim, out_dim, act)
        self.ada_pool = nn.AdaptiveAvgPool1d(out_dim)
        self.activate = get_act(act)
        self.apply_shortcut = (in_dim == out_dim)
        
    def forward(self, x):
        if self.apply_shortcut:
            residual = self.residual(x)
     
            x = self.block(x)
            return self.activate(x + residual)
        return self.activate(self.block(x))
</code></pre>
<p>The accompanying loss curve:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/8/2876a9206d786e1e18d4036ab7938b1288dc65a5.png" alt="image" data-base62-sha1="5LXklb7m5pBA7h3gqiQ026DV7BH" width="378" height="278"></p>
<p><strong>Try 2:</strong> I thought to myself <strong>“Great, it’s doing something!”</strong>, so then I decided to reset and go for 30 epochs. This training only made it 3-5 epochs and then the training and validation loss curves exploded by several orders of magnitude.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bdebd341d8f0c5820ec227bc01781e2c0f3f2722.png" alt="image" data-base62-sha1="r67uHgorVj4BpZgUTahg15nkKJk" width="370" height="278"></p>
<p><strong>Try 3:</strong> Next, I decided to try to implement the paper’s idea of reducing the input size to match the output when they don’t match: <code>y = F(x, {Wi}) + Mx</code>. I chose average pooling in place of matrix <code>M</code> to accomplish this, and my loss curve became.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b29a44679baec3acafedcbde3a8d15e7eb62022.png" alt="image" data-base62-sha1="fi0iZiCWP1gMI7oBsQfFtIOnIwa" width="384" height="278"></p>
<p>The only difference in my code is that I added average pooling so I could use shortcut connections when input and output sizes are different:</p>
<pre><code class="lang-auto">class ResBlock(nn.Module):
    def __init__(self, depth, in_dim, out_dim, act='relu', act_first=True):
        super().__init__()
        self.residual = nn.Identity()
        self.block = block(depth, in_dim, out_dim, act)
        # squeeze/pad input  to output size
        self.ada_pool = nn.AdaptiveAvgPool1d(out_dim) 
        self.activate = get_act(act)
        self.apply_pool = (in_dim != out_dim)
        
    def forward(self, x):
        # if in and out dims are different apply the padding/squeezing:
        if self.apply_pool:
            residual = self.ada_pool(self.residual(x).unsqueeze(0)).squeeze(0)
        else: residual = self.residual(x)
            
        x = self.block(x)
        return self.activate(x + residual)
)
</code></pre>
<p>So, has anyone else tried something like this? This is my first try at something a little more advanced as far as building a network goes.</p>
<p>Any tips, tricks, ideas, theory, is appreciated!</p>

googlebot: <p>Yes, such networks should work. Your issue is perhaps due to gradient explosion, or something that may not be directly related to a residual summation. In particular, relu without [batch] normalization can cause this (note that successive block outputs can’t decrease in that case).</p>

Wesley_Neill: <p>I’ll try adding batch norm and see what happens. I thought that I read in the ResNet paper<br>
that shortcutting connections is a remedy to exploding/vanishing gradients, so I didn’t add batch norm. Maybe I mis-read. If it works out, I’ll report back.</p>

Wesley_Neill: <p>Here is my result of applying batch norm:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d5e2e46706481296b77266a96323d005c083238.png" alt="image" data-base62-sha1="b2qxT2ENAQo916Jx9VdSRko5LCM" width="372" height="278"></p>
<p>I think I’ll try SGD instead of Adam, but after that I’m at a loss as to what else to experiment with. I’m just getting much better and consistent results with a vanilla feed forward network. I only wish I knew if that was on my implementation or on the data.</p>

googlebot: <p>You can try using blocks that initially output zeros. To do that, initialize <em>final</em> linear layer of a block with zeros (<a href="https://github.com/bayesiains/nsf/blob/master/nn/resnet.py#L35" rel="noopener nofollow ugc">example implementation</a>).</p>
<p>Untrained blocks in original resnet offset outputs by Normal(0,1) I believe. In any case, non-residual final layer should be used, to calibrate output ranges.</p>