Intuition for residual connections in Transformer layers?

username: BramVanroy
cooked: <p>I was reading through the implementation of <a href="https://pytorch.org/docs/master/_modules/torch/nn/modules/transformer.html#TransformerEncoderLayer" rel="noopener nofollow ugc">TransformerEncoderLayer</a> where we find the following:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/0c5cd8c2b9cdf473e30bbb1b49ca80ed442813df/torch/nn/modules/transformer.py#L282-L300" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/0c5cd8c2b9cdf473e30bbb1b49ca80ed442813df/torch/nn/modules/transformer.py#L282-L300" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/0c5cd8c2b9cdf473e30bbb1b49ca80ed442813df/torch/nn/modules/transformer.py#L282-L300</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="282" style="counter-reset: li-counter 281 ;">
<li>def forward(self, src: Tensor, src_mask: Optional[Tensor] = None, src_key_padding_mask: Optional[Tensor] = None) -&gt; Tensor:</li>
<li>    r"""Pass the input through the encoder layer.</li>
<li>
</li><li>    Args:</li>
<li>        src: the sequence to the encoder layer (required).</li>
<li>        src_mask: the mask for the src sequence (optional).</li>
<li>        src_key_padding_mask: the mask for the src keys per batch (optional).</li>
<li>
</li><li>    Shape:</li>
<li>        see the docs in Transformer class.</li>
<li>    """</li>
<li>    src2 = self.self_attn(src, src, src, attn_mask=src_mask,</li>
<li>                          key_padding_mask=src_key_padding_mask)[0]</li>
<li>    src = src + self.dropout1(src2)</li>
<li>    src = self.norm1(src)</li>
<li>    src2 = self.linear2(self.dropout(self.activation(self.linear1(src))))</li>
<li>    src = src + self.dropout2(src2)</li>
<li>    src = self.norm2(src)</li>
<li>    return src</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>As you can see, we sum the output of self attention with the original input as a residual connection. And even further, we sum that value with the output of the feedforward network. Why is that? Why do we always sum the input to the output of some component (self-attention and feedforward network in this case), rather than just taking the output? Is this a form of regularization?</p>

username: googlebot
cooked: <p>looks like residual layers (i.e. y = x + f(x) is trained instead of y = f(x))</p>

username: BramVanroy
cooked: <p>Yes, you are right of course. I am wondering what the intuition behind that is in this case. I only know residual connections from ResNet and friends, so I am not sure how to interpret their importance in the Transformer case (typical for text).</p>

username: googlebot
cooked: <p>The original attention paper uses residual connections, without much explanations. I’d say it is the same motivation anywhere, combating vanishing gradients and hopefully easier training as less shifting is needed.</p>

username: Abhilash_Srivastava
cooked: <p>I guess pretty much what they do in CV:<br>
Skip connections or residual connections are used to allow gradients to flow through a network directly, without passing through non-linear activation functions. Non-linear activation functions, by nature of being non-linear, cause the gradients to explode or vanish (depending on the weights). Skip connections form conceptually a ‘bus’ which flows right the way through the network, and in reverse, the gradients can flow backwards along it too.</p>