Before revision: Recently, contextualized word embeddings outperform static word embeddings on many NLP tasks. However, we still do not know much about the mechanism inside these representations. Do they have any common patterns? If so, where do these patterns come from? We find that almost all the contextualized word vectors of BERT and RoBERTa have a commonpattern.
After revision: In this work, we demonstrate that the contextualized word vectors of BERT and RoBERTa have a commonpattern.