XLNet has a similar architecture to BERT. However, the major difference comes in it's approach to pre-training. BERT is an Autoencoding (AE) based model, while XLNet is an Auto-Regressive (AR). This difference materializes in the MLM task, where randomly masked language tokens are to be predicted by the model.