Question: did they use other pretrained language models besides bert?
Answer: Yes