In this task, you will be presented with a context from an academic paper and a question separated with a 
. You have to answer the question based on the context.

 Our street view environment was integrated into ParlAI BIBREF6 and used to collect a large-scale dataset on Mechanical Turk involving human perception, action and communication. 
 Question: What data did they use?
 dataset on Mechanical Turk involving human perception, action and communication

The second, Continuous Bag-of-Words (CBOW), the model is given the sequence of words without the middle one and attempts to predict this omitted word.  
 Question: Which word embeddings are analysed?
Continuous Bag-of-Words (CBOW)

We computed bag-of-words-based benchmarks using the following methods:

Classification with TF-IDF + Linear SVM (TF-IDF + SVM)

Classification with Depeche++ Emotion lexicons BIBREF12 + Linear SVM (Depeche + SVM)

Classification with NRC Emotion lexicons BIBREF13, BIBREF14 + Linear SVM (NRC + SVM)

Combination of TF-IDF and NRC Emotion lexicons (TF-NRC + SVM)

Benchmarks ::: Doc2Vec + SVM
We also used simple classification models with learned embeddings. We trained a Doc2Vec model BIBREF15 using the dataset and used the embedding document vectors as features for a linear SVM classifier.

Benchmarks ::: Hierarchical RNN
For this benchmark, we considered a Hierarchical RNN, following BIBREF16. We used two BiLSTMs BIBREF17 with 256 units each to model sentences and documents. The tokens of a sentence were processed independently of other sentence tokens. For each direction in the token-level BiLSTM, the last outputs were concatenated and fed into the sentence-level BiLSTM as inputs.

The outputs of the BiLSTM were connected to 2 dense layers with 256 ReLU units and a Softmax layer. We initialized tokens with publicly available embeddings trained with GloVe BIBREF18. Sentence boundaries were provided by SpaCy. Dropout was applied to the dense hidden layers during training.

Benchmarks ::: Bi-directional RNN and Self-Attention (BiRNN + Self-Attention)
One challenge with RNN-based solutions for text classification is finding the best way to combine word-level representations into higher-level representations.

Self-attention BIBREF19, BIBREF20, BIBREF21 has been adapted to text classification, providing improved interpretability and performance. We used BIBREF20 as the basis of this benchmark.

The benchmark used a layered Bi-directional RNN (60 units) with GRU cells and a dense layer. Both self-attention layers were 60 units in size and cross-entropy was used as the cost function.

Note that we have omitted the orthogonal regularizer term, since this dataset is relatively small compared to the traditional datasets used for training such a model. We did not observe any significant performance gain while using the regularizer term in our experiments.

Benchmarks ::: ELMo embedding and Bi-directional RNN (ELMo + BiRNN)
Deep Contextualized Word Representations (ELMo) BIBREF22 have shown recent success in a number of NLP tasks. The unsupervised nature of the language model allows it to utilize a large amount of available unlabelled data in order to learn better representations of words.

We used the pre-trained ELMo model (v2) available on Tensorhub for this benchmark. We fed the word embeddings of ELMo as input into a one layer Bi-directional RNN (16 units) with GRU cells (with dropout) and a dense layer. Cross-entropy was used as the cost function.

Benchmarks ::: Fine-tuned BERT
Bidirectional Encoder Representations from Transformers (BERT) BIBREF11 has achieved state-of-the-art results on several NLP tasks, including sentence classification.

We used the fine-tuning procedure outlined in the original work to adapt the pre-trained uncased BERT$_\textrm {{\scriptsize LARGE}}$ to a multi-class passage classification task. This technique achieved the best result among our benchmarks, with an average micro-F1 score of 60.4%. 
 Question: What are the baseline benchmarks?

TF-IDF + SVM Depeche + SVM NRC + SVM TF-NRC + SVM Doc2Vec + SVM  Hierarchical RNN BiRNN + Self-Attention ELMo + BiRNN  Fine-tuned BERT