Question: What did they pretrain the model on?
Answer: hree years of online news articles from June 2016 to June 2019