Why would residual networks over non-linearities prevent exploding gradients?

BramVanroy: <p>We often sees residual connections in today’s networks, be it in ResNet or in Transformer.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/0c5cd8c2b9cdf473e30bbb1b49ca80ed442813df/torch/nn/modules/transformer.py#L282-L300" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/0c5cd8c2b9cdf473e30bbb1b49ca80ed442813df/torch/nn/modules/transformer.py#L282-L300" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/0c5cd8c2b9cdf473e30bbb1b49ca80ed442813df/torch/nn/modules/transformer.py#L282-L300</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="282" style="counter-reset: li-counter 281 ;">
<li>def forward(self, src: Tensor, src_mask: Optional[Tensor] = None, src_key_padding_mask: Optional[Tensor] = None) -&gt; Tensor:</li>
<li>    r"""Pass the input through the encoder layer.</li>
<li>
</li><li>    Args:</li>
<li>        src: the sequence to the encoder layer (required).</li>
<li>        src_mask: the mask for the src sequence (optional).</li>
<li>        src_key_padding_mask: the mask for the src keys per batch (optional).</li>
<li>
</li><li>    Shape:</li>
<li>        see the docs in Transformer class.</li>
<li>    """</li>
<li>    src2 = self.self_attn(src, src, src, attn_mask=src_mask,</li>
<li>                          key_padding_mask=src_key_padding_mask)[0]</li>
<li>    src = src + self.dropout1(src2)</li>
<li>    src = self.norm1(src)</li>
<li>    src2 = self.linear2(self.dropout(self.activation(self.linear1(src))))</li>
<li>    src = src + self.dropout2(src2)</li>
<li>    src = self.norm2(src)</li>
<li>    return src</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>The code above can be visualised as the encoder part here:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://pytorch.org/tutorials/_images/transformer_architecture.jpg" title="" rel="noopener nofollow ugc"><img src="https://pytorch.org/tutorials/_images/transformer_architecture.jpg" alt="" width="366" height="500"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename"></span><span class="informations">672×916</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg></div></a></div></p>
<p>From what I have read, residual connections help prevent exploding/vanishing gradients because skip connections can “skip passed the non-linear activation functions”, which means that gradients enjoy the same benefit. I do not understand what that means.</p>
<p>In the backward pass, why would PyTorch skip the non-linearities? How does it know to skip those? So, in this fictional example:</p>
<pre><code class="lang-python">src2 = self.activation(self.linear1(src))
src = src + src2
</code></pre>
<p>what is the advantage of using the skip connection? Doesn’t the backward pass flow through <em>all</em> operations?</p>
<p>I feel that I am missing an important part of the puzzle, but I can’t figure out which one.</p>

googlebot: <p>In <code>srcOut = src1 + src2</code></p>
<p>The key is that [loss] gradient w.r.t. srcOut is passed to both summands unchanged. As a result, any block or partial sum could in theory learn to produce the best srcOut. Thus, later blocks learn residuals.</p>
<p>Contrast this with function composition: srcOut = fc2(act(fc1(src))). Here you’ll have a <strong>chain</strong> of intermediate results, and a chain of multiplications will be applied to the initial gradient dLoss/dSrcOut.</p>