Question: What improvement does the MOE model make over the SOTA on language modelling?
Answer: Perpexity is improved from 34.7 to 28.0.