In this task, you will be presented with a context from an academic paper and a question separated with a 
. You have to answer the question based on the context.
--------
Question: We design the word-level refine decoder because this process is similar to the cloze task in BERT's pre-train process, therefore by using the ability of the contextual language model the decoder can generate more fluent and natural sequences. 
 Question: Why masking words in the decoder is helpful?

Answer: ecause this process is similar to the cloze task in BERT's pre-train process, therefore by using the ability of the contextual language model the decoder can generate more fluent and natural sequences.


Question: Embedding: We developed different variations of our models with a simple lookup table embeddings learned from scratch and using high-performance contextual embeddings, which are ELMo BIBREF11, BERT BIBREF16 and ClinicalBERT BIBREF13 (trained and provided by the authors). 
 Question: What embeddings are used?

Answer:  simple lookup table embeddings learned from scratch using high-performance contextual embeddings, which are ELMo BIBREF11, BERT BIBREF16 and ClinicalBERT BIBREF13


Question: Turning to semantics, using visualizations of the activations created by different pieces of text, we show suggestive evidence that BERT distinguishes word senses at a very fine level.  We apply attention probes to the task of identifying the existence and type of dependency relation between two words. 
 Question: How were the feature representations evaluated?

Answer:
attention probes using visualizations of the activations created by different pieces of text