Question: What do they evaluate in the paper?
Evidence: We use the SentEval toolkit to evaluate the quality of sentence representations from BERT activations. The evaluation encompasses a variety of downstream and probing tasks. Downstream tasks include text classification, natural language inference, paraphrase detection, and semantic similarity. Probing tasks use single sentence embedding as input, are designed to probe sentence-level linguistic phenomena, from superficial properties of sentences to syntactic information to semantic acceptability. For details about the tasks, please refer to BIBREF8 and BIBREF9. We compare the BERT embeddings against two state-of-the-art sentence embeddings, Universal Sentence Encoder BIBREF5, InferSent BIBREF2, and a baseline of averaging GloVe word embeddings.
Effect of Encoder Layer: We compare the performance of embeddings extracted from different encoder layers of a pre-trained BERT using bert-as-service BIBREF10. Since we are interested in the linguistic information encoded in the embeddings, we only add a logistic regression layer on top of the embeddings for each classification task. The results of using [CLS] token activations as embeddings are presented in Figure FIGREF1. The raw values are provided in the Appendix. In the heatmap, the raw values of metrics are normalized by the best performance of a particular task from all the models we evaluated including BERT. The tasks in the figure are grouped by task category. For example, all semantic similarity related tasks are placed at the top of the figure.
As can be seen from the figure, embeddings from top layers generally perform better than lower layers. However, for certain semantic probing tasks such as tense classification, subject, and object number classifications, middle layer embeddings perform the best. Intuitively, embeddings from top layer should be more biased towards the target of BERT pre-training tasks, while bottom layer embeddings should be close to the word embeddings. We observed a higher correlation in performance between bottom layer embeddings and GloVe embeddings than embeddings from other layers. Overall, pre-trained BERT embeddings perform well in text classification and syntactic probing tasks. The biggest limitation lies in the semantic similarity and sentence surface information probing tasks, where we observed a big gap between BERT and other state-of-the-art models.
Effect of Pooling Methods: We examined different methods of extracting BERT hidden state activations. The pooling methods we evaluated include: CLS-pooling (the hidden state corresponding to the [CLS] token), SEP-pooling (the hidden state corresponding to the [SEP] token), Mean-pooling (the average of the hidden state of the encoding layer on the time axis), and Max-pooling (the maximum of the hidden state of the encoding layer on the time axis). To eliminate the layer-wise effects, we averaged the performance of each pooling method over different layers. The results are summarized in Table TABREF2, where the score for each task category is calculated by averaging the normalized values for the tasks within each category. Although the activations of [CLS] token hidden states are often used in fine-tuning BERT for classification tasks, Mean-pooling of hidden states performs the best in all task categories among all the pooling methods.
Pre-trained vs. Fine-tuned BERT: All the models we considered in this paper benefit from supervised training on natural language inference datasets. In this section, we compare the performance of embeddings from pre-trained BERT and fine-tuned BERT. Two natural language inference datasets, MNLI BIBREF11 and SNLI, were considered in the experiment. Inspired by the fact that embeddings from different layers excel in different tasks, we also conducted experiments by concatenating embeddings from multiple layers. The results are presented in Table TABREF3, and the raw values are provided in the Appendix.
As we can see from the table, embeddings from pre-trained BERT are good at capturing sentence-level syntactic information and semantic information, but poor at semantic similarity tasks and surface information tasks. Our findings are consistent with BIBREF12 work on assessing BERT's syntactic abilities. Fine-tuning on natural language inference datasets improves the quality of sentence embedding, especially on semantic similarity tasks and entailment tasks. Combining embeddings from two layers can further boost the performance on sentence surface and syntactic information probing tasks. Experiments were also conducted by combining embeddings from multiple layers. However, there is no significant and consistent improvement over pooling just from two layers. Adding multi-layer perceptron (MLP) instead of logistic regression layer on top of the embeddings also provides no significant changes in performance, which suggests that most linguistic properties can be extracted with just a linear readout of the embeddings. Our best model is the combination of embeddings from the top and bottom layer of the BERT fine-tuned on SNLI dataset.
Universal text representations are important for many NLP tasks as modern deep learning models are becoming more and more data-hungry and computationally expensive. On one hand, most research and industry tasks face data sparsity problem due to the high cost of annotation. Universal text representations can mitigate this problem to a certain extent by performing implicit transfer learning among tasks. On the other hand, modern deep learning models with millions of parameters are expensive to train and host, while models using text representation as the building blocks can achieve similar performance with much fewer tunable parameters. The pre-computed text embeddings can also help decrease model latency dramatically at inference time.
Since the introduction of pre-trained word embeddings such as word2vec BIBREF0 and GloVe BIBREF1, a lot of efforts have been devoted to developing universal sentence embeddings. Initial attempts at learning sentence representation using unsupervised approaches did not yield satisfactory performance. Recent work BIBREF2 has shown that models trained in supervised fashion on datasets like Stanford Natural Language Inference (SNLI) corpus BIBREF3 can consistently outperform unsupervised methods like SkipThought vectors BIBREF4. More recently, Universal Sentence Encoder BIBREF5 equipped with the Transformer BIBREF6 as the encoder, co-trained on a large amount of unsupervised training data and SNLI corpus, has demonstrated surprisingly good performance with minimal amounts of supervised training data for a transfer task.
BERT BIBREF7, one of the latest models that leverage heavily on language model pre-training, has achieved state-of-the-art performance in many natural language understanding tasks ranging from sequence and sequence pair classification to question answering. The fact that pre-trained BERT can be easily fine-tuned with just one additional output layer to create a state-of-the-art model for a wide range of tasks suggests that BERT representations are potential universal text embeddings.
Passages that consist of multiple sentences are coherent units of natural languages that convey information at a pragmatic or discourse level. While there are many models for generating and evaluating sentence embeddings, there hasn't been a lot of work on passage level embedding generation and evaluation.
In this paper, we conducted an empirical study of layer-wise activations of BERT as general-purpose text embeddings. We want to understand to what extent does the BERT representation capture syntactic and semantic information. The sentence-level embeddings are evaluated on downstream and probing tasks using the SentEval toolkit BIBREF8, while the passage-level encodings are evaluated on four passage-level QA datasets (both factoid and non-factoid) under a learning-to-rank setting. Different methods of combining query embeddings with passage-level answer embeddings are examined.
Effect of Encoder Layer: We compare the performance of embeddings extracted from different encoder layers of a pre-trained BERT using bertas-service (Xiao, 2018).****In this paper, we conducted an empirical study of layer-wise activations of BERT as generalpurpose text embeddings. We want to understand to what extent does the BERT representation capture syntactic and semantic information.
Answer: The quality of sentence representations extracted from different encoder layers of a pre-trained BERT as general-purpose text embeddings.