Before revision:  While previous work shows syntactic, semantic and word sense knowledge in BERT , little work has been done on investigating how BERT solves CommonsenseQA tasks.In particular, it is an interesting research question whether BERT relies on shallow syntactic patterns or deeper commonsense knowledge for disambiguation. We propose two attention-based methods to analyze commonsense knowledge inside BERT , and the contribution of such knowledge for the model prediction.
After revision:  While previous work shows syntactic, semantic and word sense knowledge in BERT , and the contribution of such knowledge for the model prediction.