Value of [CLS] Token for Transformer Encoders

blueeagle: <p>Hi everyone,</p>
<p>i want to use a transformer encoder for sequence classification. Following the idea of BERT, I want to prepend a [CLS] token to the input sequence. After forwardpassing my sequence through the transformer encoder, I plan to use the encoding of the [CLS] token as representation for the whole sequence.</p>
<p>As far as I understand, it doesn’t make a difference with what value I initialize this [CLS] token, as the transformer will (hopefully) learn to not attend that value of the [CLS] token anyways, as it doesn’t contain any information about the sequence.</p>
<p>Is there any reason why I shouldn’t initialize this token randomly during each forwardpass?<br>
This would lead to the value for the [CLS] token being different during each forwardpass however as far as I think it shouldn’t change anything.</p>
<p>Here is a minimal example of what I am planning to do:</p>
<pre><code class="lang-auto">class ClassificationTransformer(nn.Module):
    def __init__(self, embedding_dim, n_head, depth):
        super(ClassificationTransformer, self).__init__()
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=embedding_dim,
            nhead=n_head,
            batch_first=True
        )
        self.transformer_encoder = nn.TransformerEncoder(
            encoder_layer,
            num_layers=depth
        )

    def forward(self, x):
        # assume x of shape [B, T, F]
        cls_token = torch.randn((x.shape[0], 1, x.shape[-1]), device=x.device)
        tokens = torch.column_stack((cls_token, x)) # tokens is of shape [B, 1+T, F]
        encodings = self.transformer_encoder(x)
        representation = encodings[:,0,:]
        return representation
</code></pre>
<p>Thanks for helping! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

tom: <aside class="quote no-group" data-username="blueeagle" data-post="1" data-topic="164441">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blueeagle/48/47881_2.png" class="avatar"> blueeagle:</div>
<blockquote>
<p>Is there any reason why I shouldn’t initialize this token randomly during each forwardpass?</p>
</blockquote>
</aside>
<p>The transformer generates query, key and value from the input. While you have a point that value might not be needed, you <em>do</em> want at least the query part to work.</p>
<p>Also, I’d probably ask the question the other way round and not use randomness unless I have a reason to do so.</p>
<p>Best regards</p>
<p>Thomas</p>

blueeagle: <p>Hi Thomas,</p>
<p>thanks a lot for your answer.<br>
What would you propose to use instead as [CLS] token? I thought that using a tensor of all zeros could maybe lead to bad computational characteristics.<br>
Do you think setting the value for the [CLS] token randomly, but only one time (e.g. in the <strong>init</strong> method) so that it is fixed during training, would be a better option?</p>

tom: <p>The typical thing (GPT, vision transformer, …) is to make it a learned parameter, e.g. here:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/MathInf/toroidal/blob/bff09f725627e4629d464008dc7c5f9d6322ebad/toroidal/models.py#L18">
  <header class="source">

      <a href="https://github.com/MathInf/toroidal/blob/bff09f725627e4629d464008dc7c5f9d6322ebad/toroidal/models.py#L18" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/MathInf/toroidal/blob/bff09f725627e4629d464008dc7c5f9d6322ebad/toroidal/models.py#L18" target="_blank" rel="noopener nofollow ugc">MathInf/toroidal/blob/bff09f725627e4629d464008dc7c5f9d6322ebad/toroidal/models.py#L18</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="8" style="counter-reset: li-counter 7 ;">
          <li>self.n_channels = n_channels</li>
          <li>num_patches = (input_size ** 2) // (</li>
          <li>    patch_size ** 2</li>
          <li>)  # only when this is divisible...</li>
          <li>self.patch_to_vec = torch.nn.Conv2d(</li>
          <li>    3,</li>
          <li>    n_channels,</li>
          <li>    kernel_size=(patch_size, patch_size),</li>
          <li>    stride=(patch_size, patch_size),</li>
          <li>)  # equivalent to reshape + linear</li>
          <li class="selected">self.class_token = torch.nn.Parameter(</li>
          <li>    torch.randn(1, 1, n_channels)</li>
          <li>)  # "global information"</li>
          <li>self.pos_embedding = torch.nn.Parameter(</li>
          <li>    torch.randn(1, num_patches + 1, n_channels)</li>
          <li>)</li>
          <li>
          </li>
<li>torch.nn.init.zeros_(self.patch_to_vec.bias)</li>
          <li>torch.nn.init.normal_(self.patch_to_vec.bias, std=0.02)  # scale with size?</li>
          <li>torch.nn.init.normal_(self.class_token, std=0.02)</li>
          <li>torch.nn.init.normal_(self.pos_embedding, std=0.02)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Best regards</p>
<p>Thomas</p>

blueeagle: <p>Thank you so much, this is exactly what I was looking for <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Yami: <p>Hello all～：</p>
<p>I want to ask a question. If I also want to do a sequence classification task, whether I must add the CLS token before each sentence and input the CLS token embedding into FC layer.</p>
<p>If I do not add the CLS token and use the first token of each sentence to do classification , is it ok?</p>
<pre><code class="lang-auto">class ClassificationTransformer(nn.Module):
    def __init__(self, embedding_dim, n_head, depth):
        super(ClassificationTransformer, self).__init__()
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=embedding_dim,
            nhead=n_head,
            batch_first=True
        )
        self.transformer_encoder = nn.TransformerEncoder(
            encoder_layer,
            num_layers=depth
        )

    def forward(self, x):
        encodings = self.transformer_encoder(x)
        representation = encodings[:,0,:] # use the first token of the sentence to do classification
        return representation
</code></pre>