text:   Self-attention",4,"['Our @emnlp2020 paper “LUKE: Deep Contextualized Entity Representations with Entity-aware\nSelf-attention” is now available on arXiv! We present new pretrained contextualized representations that achieve SOTA on five datasets including SQuAD and CoNLL-2003.\n<LINK>', 'LUKE is based on bidirectional Transformer, treats words and entities in a text as independent tokens, and outputs contextualized representations of them. The representations can be used to address downstream tasks similarly to BERT. https://t.co/s7KZCxBi5D', 'LUKE is trained using a novel pretraining task that involves predicting randomly masked words (equivalent to BERT’s masked language model) and entities in an entity-annotated corpus obtained from Wikipedia.', 'LUKE also uses a new *entity-aware* self-attention mechanism that considers the types of tokens (words or entities) when computing attention scores.\nThe source code and pretrained models are available at https://t.co/k5koMphw7n. The documentation will be available soon!😃']",20,10,965