Transformers to encode a sequence into a fixed lenght vector

ErikJ: <p>In a RNN you can use the last hidden layer as a vector representation of a sequence. I was wondering if there is any similar idea in the transformer architecture?</p>
<p>Background:<br>
I am building a model that deals with protein data. At one stage I wish to include raw protein sequence information to the model. For this I need to encode a sequence into a fixed lenght vector that contains information about the sequence on the global scope. This vector will then be concatenated to some other vector and used in downstrema processing.</p>

klory: <p>you can define your own way (\eg average pool all token vectors) or simply using the BERT default setup, the <code>[CLS]</code> token’s output vector</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/">
  <header class="source">

      <a href="http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/" target="_blank" rel="noopener nofollow ugc">jalammar.github.io</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/" target="_blank" rel="noopener nofollow ugc">A Visual Guide to Using BERT for the First Time</a></h3>

  <p>Translations: Chinese, Korean, Russian


  
  


Progress has been rapidly accelerating in machine learning models that process language over the last couple of years. This progress has left the research lab and started powering some of the leading...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea82a8f0419b7d7e77a2cc405f4b6247fe88d19b.png" data-download-href="https://discuss.pytorch.org/uploads/default/ea82a8f0419b7d7e77a2cc405f4b6247fe88d19b" title="bert-distilbert-tokenization-1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea82a8f0419b7d7e77a2cc405f4b6247fe88d19b_2_690x373.png" alt="bert-distilbert-tokenization-1" data-base62-sha1="xszFgOoipjRaxt69vdk2LZBMIZR" width="690" height="373" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea82a8f0419b7d7e77a2cc405f4b6247fe88d19b_2_690x373.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea82a8f0419b7d7e77a2cc405f4b6247fe88d19b_2_1035x559.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea82a8f0419b7d7e77a2cc405f4b6247fe88d19b_2_1380x746.png 2x" data-dominant-color="EAEDE8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">bert-distilbert-tokenization-1</span><span class="informations">1705×922 63.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ErikJ: <p>Ok interesting. So if I understand correctly the model will learn to generate this artificial vector (<code>[CLS]</code>) in such way that it will be usefull for solving the task at hand? In the linked post this would be text calssification.</p>

klory: <p>[CLS] output can be treated as the last hidden layer of LSTM</p>

mortazavi: <p>People always say “look at BERT” but what if one wants to build ones own sequence-to-vector encoder? Most tutorials on BERT are limited to uses within machine translation (which is a sequence to sequence task), and they spend enormous amount of time on just setting up the dataset and the tokenizer and the mask and a single example configuration of BERT . . . As a guide this is entirely inadequate when it comes to studying sequence-to-vector encodings of the type ErikJ has written about . . . Even a simple sequence-to-sequence example that just talks about the use of the basic APIs and how to investigate various configurations would be useful . . . In other words, the tutorials need to get rid of all data-preparation talk and just work on artificial tensor sequences  . . . It is not hard to build artificial sequences for experimentation purposes and for showing the use of the transformer APIs . . . Not all examples need to be about text sequences . . .</p>

mortazavi: <p>The suggestion here</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://desh2608.github.io/2020-01-08-transformer-asr/" target="_blank" rel="noopener nofollow ugc">desh2608.github.io</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:128/134;"><img src="https://desh2608.github.io/static/img/desh.jpg" class="thumbnail" width="128" height="134"></div>

<h3><a href="https://desh2608.github.io/2020-01-08-transformer-asr/" target="_blank" rel="noopener nofollow ugc">The Challenges of using Transformers in ASR</a></h3>

<p>Since mid 2018 and throughout 2019, one of the most important directions of research in speech recognition has been the use of self-attention networks and transformers, as evident from the numerous papers exploring the subject. In this post, I try to...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>which has may of the similar set of issues as the task suggested here implies that one may first have to reduce one’s sequence lengths – perhaps, through convolutions with a large stride – before feeding the resulting (shorter) sequences to the transformer layers.</p>
<p>One or more convolution layers before the transformer layer (in problems such as the one mentioned where sequences are long and where we’re searching for full “sentence” encoding into a vector) can also improve the overall positional encoding of the full model.</p>

ErikJ: <p>Thenk you so much, this will be most helpfull!</p>