Question: What is the language model pre-trained on?
Answer: Wikipedea Corpus and BooksCorpus