Definition: In this task, you will be presented with a context from an academic paper and a question separated with a 
. You have to answer the question based on the context.

Positive Example 1 -
Input: We evaluate the proposed approach on the Chinese social media text summarization task, based on the sequence-to-sequence model. Large-Scale Chinese Short Text Summarization Dataset (LCSTS) is constructed by BIBREF1 . The dataset consists of more than 2.4 million text-summary pairs in total, constructed from a famous Chinese social media microblogging service Weibo.  
 Question: Are results reported only for English data?
Output: No

Positive Example 2 -
Input: Since INLINEFORM0 is constant, we only need to minimize INLINEFORM1 , therefore the loss function becomes: DISPLAYFORM0 
 Question: How is the expectation regularization loss defined?
Output: DISPLAYFORM0

Negative Example 1 -
Input: The minimal change operation substantially changed the meaning of the sentence, and yet the embedding of the transformation lies very closely to the original sentence (average similarity of 0.930). 
 Question: Do they do any analysis of of how the modifications changed the starting set of sentences?
Output: No

Negative Example 2 -
Input: Looking at the current WinoGrande leaderboard, it appears that the previous state of the art is based on RoBERTa BIBREF2, which can be characterized as an encoder-only transformer architecture. 
 Question: What is the previous state of the art?
Output: GPT-3

Now complete the following example -
Input: We computed bag-of-words-based benchmarks using the following methods:

Classification with TF-IDF + Linear SVM (TF-IDF + SVM)

Classification with Depeche++ Emotion lexicons BIBREF12 + Linear SVM (Depeche + SVM)

Classification with NRC Emotion lexicons BIBREF13, BIBREF14 + Linear SVM (NRC + SVM)

Combination of TF-IDF and NRC Emotion lexicons (TF-NRC + SVM)

Benchmarks ::: Doc2Vec + SVM
We also used simple classification models with learned embeddings. We trained a Doc2Vec model BIBREF15 using the dataset and used the embedding document vectors as features for a linear SVM classifier.

Benchmarks ::: Hierarchical RNN
For this benchmark, we considered a Hierarchical RNN, following BIBREF16. We used two BiLSTMs BIBREF17 with 256 units each to model sentences and documents. The tokens of a sentence were processed independently of other sentence tokens. For each direction in the token-level BiLSTM, the last outputs were concatenated and fed into the sentence-level BiLSTM as inputs.

The outputs of the BiLSTM were connected to 2 dense layers with 256 ReLU units and a Softmax layer. We initialized tokens with publicly available embeddings trained with GloVe BIBREF18. Sentence boundaries were provided by SpaCy. Dropout was applied to the dense hidden layers during training.

Benchmarks ::: Bi-directional RNN and Self-Attention (BiRNN + Self-Attention)
One challenge with RNN-based solutions for text classification is finding the best way to combine word-level representations into higher-level representations.

Self-attention BIBREF19, BIBREF20, BIBREF21 has been adapted to text classification, providing improved interpretability and performance. We used BIBREF20 as the basis of this benchmark.

The benchmark used a layered Bi-directional RNN (60 units) with GRU cells and a dense layer. Both self-attention layers were 60 units in size and cross-entropy was used as the cost function.

Note that we have omitted the orthogonal regularizer term, since this dataset is relatively small compared to the traditional datasets used for training such a model. We did not observe any significant performance gain while using the regularizer term in our experiments.

Benchmarks ::: ELMo embedding and Bi-directional RNN (ELMo + BiRNN)
Deep Contextualized Word Representations (ELMo) BIBREF22 have shown recent success in a number of NLP tasks. The unsupervised nature of the language model allows it to utilize a large amount of available unlabelled data in order to learn better representations of words.

We used the pre-trained ELMo model (v2) available on Tensorhub for this benchmark. We fed the word embeddings of ELMo as input into a one layer Bi-directional RNN (16 units) with GRU cells (with dropout) and a dense layer. Cross-entropy was used as the cost function.

Benchmarks ::: Fine-tuned BERT
Bidirectional Encoder Representations from Transformers (BERT) BIBREF11 has achieved state-of-the-art results on several NLP tasks, including sentence classification.

We used the fine-tuning procedure outlined in the original work to adapt the pre-trained uncased BERT$_\textrm {{\scriptsize LARGE}}$ to a multi-class passage classification task. This technique achieved the best result among our benchmarks, with an average micro-F1 score of 60.4%. 
 Question: What are the baseline benchmarks?
Output: 



Answer: TF-IDF + SVM Depeche + SVM NRC + SVM TF-NRC + SVM Doc2Vec + SVM  Hierarchical RNN BiRNN + Self-Attention ELMo + BiRNN  Fine-tuned BERT