Question: What toolkit do they use on downstream and probing tasks?
Evidence: Universal text representations are important for many NLP tasks as modern deep learning models are becoming more and more data-hungry and computationally expensive. On one hand, most research and industry tasks face data sparsity problem due to the high cost of annotation. Universal text representations can mitigate this problem to a certain extent by performing implicit transfer learning among tasks. On the other hand, modern deep learning models with millions of parameters are expensive to train and host, while models using text representation as the building blocks can achieve similar performance with much fewer tunable parameters. The pre-computed text embeddings can also help decrease model latency dramatically at inference time.
Since the introduction of pre-trained word embeddings such as word2vec BIBREF0 and GloVe BIBREF1, a lot of efforts have been devoted to developing universal sentence embeddings. Initial attempts at learning sentence representation using unsupervised approaches did not yield satisfactory performance. Recent work BIBREF2 has shown that models trained in supervised fashion on datasets like Stanford Natural Language Inference (SNLI) corpus BIBREF3 can consistently outperform unsupervised methods like SkipThought vectors BIBREF4. More recently, Universal Sentence Encoder BIBREF5 equipped with the Transformer BIBREF6 as the encoder, co-trained on a large amount of unsupervised training data and SNLI corpus, has demonstrated surprisingly good performance with minimal amounts of supervised training data for a transfer task.
BERT BIBREF7, one of the latest models that leverage heavily on language model pre-training, has achieved state-of-the-art performance in many natural language understanding tasks ranging from sequence and sequence pair classification to question answering. The fact that pre-trained BERT can be easily fine-tuned with just one additional output layer to create a state-of-the-art model for a wide range of tasks suggests that BERT representations are potential universal text embeddings.
Passages that consist of multiple sentences are coherent units of natural languages that convey information at a pragmatic or discourse level. While there are many models for generating and evaluating sentence embeddings, there hasn't been a lot of work on passage level embedding generation and evaluation.
In this paper, we conducted an empirical study of layer-wise activations of BERT as general-purpose text embeddings. We want to understand to what extent does the BERT representation capture syntactic and semantic information. The sentence-level embeddings are evaluated on downstream and probing tasks using the SentEval toolkit BIBREF8, while the passage-level encodings are evaluated on four passage-level QA datasets (both factoid and non-factoid) under a learning-to-rank setting. Different methods of combining query embeddings with passage-level answer embeddings are examined.
The sentence-level embeddings are evaluated on downstream and probing tasks using the SentEval toolkit (Conneau and Kiela, 2018), while the passage-level encodings are evaluated on four passage-level QA datasets (both factoid and nonfactoid) under a learning-to-rank setting.
Answer: The SentEval toolkit.