Task: Question Understanding
Input (English): In our final system, after pre-training the forward and backward LMs separately, we remove the top layer softmax and concatenate the forward and backward LM embeddings to form bidirectional LM embeddings, i.e., INLINEFORM0 .  
 Question: how are the bidirectional lms obtained?
Output (English): Abstractive