In this task, you will be presented with a context from an academic paper and you have to write an answerable question based on the context. Your questions can be extractive, abstractive, or yes-no questions.

Our model BiLSTM+CNN(grapheme-level) and BiLSTM+CNN(G)+POS outperforms all other model experimented in OurNepali and ILPRL dataset respectively.
What is the best model?

We first describe our models individually and then the ensembling technique that we employ. In the following, MN denotes Memory Networks to encode conversational history, RCNN signify R-CNN for object level representations of an image, Wt represents additional linear layer in the decoder, and LF a late fusion mechanism as defined in BIBREF0.

Models ::: LF-RCNN
Late fusion encoder BIBREF0 with concatenated history. We use two-layered LSTMs with 512 hidden units for embedding questions and history. The object-level features are weighed using only question embeddings. The word embeddings from Glove vectors are frozen and are not fine-tuned. Figure FIGREF6 gives an overview of the architecture.

Models ::: MN-RCNN
Memory network encoder BIBREF0 with bi-directional GRUs and word embeddings fine-tuned. Object-level features are weighed by question and caption embedding. The rest of the scheme is same as above. (Figure FIGREF6)

Models ::: MN-RCNN-Wt
Same as above but with an additional linear layer applied to the dot product of candidate answer and encoder output, and gated using tanh function. Compare Figure FIGREF6 with Figure FIGREF6
Which three discriminative models did they use?

The method is evaluated on the BERTbase model, which has 12 layers, 12 self-attention heads with a hidden size of 768.

To accelerate the training speed, two-phase training BIBREF1 is adopted. The first phase uses a maximal sentence length of 128, and 512 for the second phase. The numbers of training steps of two phases are 50K and 40K for the BERTBase model. We used AdamW BIBREF13 optimizer with a learning rate of 1e-4, a $\beta _1$ of 0.9, a $\beta _2$ of 0.999 and a L2 weight decay rate of $0.01$. The first 10% of the total steps are used for learning rate warming up, followed by the linear decay schema. We used a dropout probability of 0.1 on all layers. The data used for pre-training is the same as BERT, i.e., English Wikipedia (2500M words) and BookCorpus (800M words) BIBREF14.

Do they train their model starting from a checkpoint?