Task: Question Answering
Input (English): In our final system, after pre-training the forward and backward LMs separately, we remove the top layer softmax and concatenate the forward and backward LM embeddings to form bidirectional LM embeddings, i.e., INLINEFORM0 .  
 Question: how are the bidirectional lms obtained?
Output (English): They pre-train forward and backward LMs separately, remove top layer softmax, and concatenate to obtain the bidirectional LMs.