# Extracting Sentence Embeddings from Pretrained Transformer Models
## 2 Related Work
### Learning Sentence Embeddings Directly
#### 2.3.2 To RNN- and Transformer-Based Models  
Shallow Doc2 Vec-like models soon met competition from more complicated Recurrent Neural Networks (RNNs). In a sequence-to-sequence [166] (like machine translation) setting, such a model usually has two parts: encoder and decoder. The encoder, going through each token one by one, encodes the input sequence into a fixed hidden representation, which is later used by the decoder to generate the target sequence (translation in NMT) token by token. Meanwhile, only the decoder part is needed to generate sentence representation. Words of the sentence are sequentially fed as input to the RNN, and the final hidden state is interpreted as its representation. The benefit of RNN models is that the sequential consumption of tokens allows the final representation to account for the word order and process the arbitrary number of tokens. However, the effectiveness of this mechanism is questionable. It turns out that information has a hard time propagating all the way to the final hidden state. As compensation for that, a bidirectional [167] setting was used, which connects two hidden layers of opposite directions to the same output, simultaneously getting forward and backward information. The second solution was to use an attention mechanism [4], which, during decoder token generation, takes a weighted average of the encoder hidden states from all the input tokens. But it was only helpful for sequence-to-sequence tasks, like translation, as such a mechanism removed the last hidden state bottleneck, which was required to compress a long sequence of tokens into a single vector for a whole text representation. One of the first famous recurrent models for sentence representation is the SkipThought [168]. This model is trained to predict neighboring sentences from the source one. The center sentence is encoded by a bidirectional GRU (Gated Recurrent Unit [169]) that concatenates the last hidden state of a forward GRU and the last hidden state of a backward GRU, and then decodes it into the two target sentences. This way, the encoder is trained to map a sentence to a single representative vector. The authors showed that the model is robust and performed well on all tasks considered. The main drawbacks of this model are the month-long training, huge vector size of 4800 (resulting from the concatenation of 2400 vectors from two separate models), and support of only sentence-level embeddings, as well as the need for training text with a coherent inter-sentence narrative. As recurrent GRU or LSTM [170] architectures became the standard, the most recent advances in text representation were due to how data and training objectives were chosen. Here, we have to mention a famous InferSent [12] model. The authors showed that the high-quality supervised data, although in low quantity, can hugely increase the performance. They trained an encoder based on a bi-directional LSTM architecture with max pooling, on the Stanford Natural Language Inference (SNLI) [171] dataset. The resulting model outperformed SkipThought after less than a day of training on a single GPU. Other authors of [172] modified the SkipThought model to QuickThoughts by framing the training task as a classification. Instead of generating the target sentence, the model encodes the likely candidates and chooses one of them. Authors of [173] leveraged several data sources with multiple training objectives. Their GenSen model training tasks included Neural Machine Translation, Constituency Parsing, Natural Language Inference, and SkipThought-like training. Finally, authors of [174] proposed even greater utilization of training data by constructing a discourse marker prediction task, predicting tokens such us because, and, if, etc. Such framing of the task allowed the authors to mine vast amounts of text pairs together with the connecting markers. Their DisSent model performed similarly well to InferSent on various transfer tasks.