Question: how are the bidirectional lms obtained?
Answer: They pre-train forward and backward LMs separately, remove top layer softmax, and concatenate to obtain the bidirectional LMs.