Before revision: The success of pre-trained contextualized language models such as BERT motivates a line of work that investigates linguistic knowledge inside such models in order to explain the huge improvement in downstream tasks . While previous work shows syntactic, semantic and word sense knowledge in BERT , little work has been done on investigating how BERT solves CommonsenseQA tasks.
After revision: The success of pre-trained models can rely on spurious associations (e.g., data bias) rather than key cues in solving sentiment classification and other problems. We quantitatively investigate the presence of structural commonsense cues in BERT when solving commonsense tasks , little work has been done on investigating how BERT solves CommonsenseQA tasks.