SlowGuess/ACL_2020_OCR huggingface dataset:
Type Head	Type Tail	Sent Len	Arg Dist	Arg Ord	Ent Exist	PosL Head	PosR Head	PosL Tail	PosR Tail	Tree Dep	SDP Dep	GR Head	GR Tail	F1 score
Majority vote	66.4	33.5	14.5	14.8	54.7	51.0	22.8	23.0	26.9	20.0	23.7	28.4	58.4	75.2	-
Length	66.4	33.5	100.0	13.8	54.8	59.4	18.6	24.7	26.9	20.1	30.5	29.6	58.4	75.2	-
ArgDist	66.4	33.5	16.5	100.0	54.7	77.5	14.9	23.0	26.9	19.8	23.8	35.3	58.4	75.2	-
BoE	77.7	47.6	61.1	22.6	97.3	66.5	33.7	41.5	32.5	36.3	29.8	31.0	66.3	77.4	39.4
CNN	94.0	85.8	47.6	88.1	98.8	84.5	70.7	76.1	84.0	86.5	28.5	44.0	78.0	88.6	55.9
+ELMo	97.0	90.2	48.7	91.7	99.1	84.3	76.1	81.2	86.6	90.1	28.3	45.0	82.8	91.9	58.8
+BERT↓	95.9	88.8	44.7	46.0	93.8	79.9	64.7	74.4	80.8	88.4	29.4	41.0	77.7	90.0	59.7
+BERT↑	96.1	88.8	48.0	43.7	91.9	80.0	56.9	70.3	80.1	87.5	28.0	41.3	75.0	89.6	61.0
CNN⊗	84.2	60.9	46.4	58.3	94.3	81.5	44.3	50.9	54.4	63.9	27.7	40.0	68.5	82.0	59.5
+ELMo	82.8	69.8	47.4	75.6	98.1	82.9	54.2	60.2	65.4	77.3	28.7	42.4	71.9	85.0	61.7
+BERT↓	87.6	80.3	50.9	29.3	83.2	72.4	39.3	46.1	67.7	80.7	30.1	36.9	67.1	87.4	65.3
+BERT↑	87.2	79.3	50.6	25.3	78.3	69.8	39.6	42.9	59.9	77.5	30.3	35.1	65.6	86.9	66.1
Bi-LSTM	93.4	81.2	42.0	47.9	99.4	79.2	41.2	50.8	50.6	68.4	28.7	41.7	69.3	85.2	55.3
+ELMo	96.4	89.6	27.9	47.0	97.9	80.9	47.8	52.5	67.2	72.6	25.2	42.8	72.1	90.0	61.8
+BERT↓	96.0	87.3	31.0	45.5	99.1	78.8	46.1	55.6	61.7	71.3	26.6	42.7	72.2	87.7	62.5
+BERT↑	96.0	87.7	28.6	45.3	97.7	80.4	48.0	50.9	61.4	67.4	25.1	42.3	70.8	87.0	63.1
Bi-LSTM⊗	81.9	71.4	27.6	35.6	90.6	73.2	36.1	40.5	59.3	66.4	25.7	38.4	64.6	85.3	62.9
+ELMo	82.8	50.7	30.6	19.7	73.4	65.0	32.0	35.9	37.9	41.8	28.0	32.2	63.0	79.5	64.1
+BERT↓	82.3	77.9	34.1	25.6	87.6	68.4	32.5	36.7	61.5	64.7	27.6	35.1	66.6	86.0	65.4
+BERT↑	81.7	79.6	30.2	21.3	81.1	67.0	30.6	33.8	55.9	55.1	27.3	34.2	64.1	84.9	66.1
GCN	93.0	81.9	18.8	35.5	86.0	74.4	48.6	48.8	51.2	52.3	24.0	49.9	74.2	85.9	57.4
+ELMo	96.3	86.2	18.7	29.3	77.5	74.0	50.4	52.0	48.9	51.7	23.2	47.4	77.1	86.9	62.1
+BERT↓	96.0	85.2	20.7	31.2	83.6	74.2	48.6	52.4	47.4	50.4	23.9	48.7	74.4	85.3	62.9
+BERT↑	96.3	85.7	21.4	32.9	84.3	75.3	50.1	54.6	48.6	52.5	24.5	49.2	76.3	85.8	61.5
GCN⊗	87.6	67.4	18.1	33.1	81.6	72.8	36.8	51.1	44.8	48.8	24.1	47.3	73.2	83.0	63.7
+ELMo	92.7	68.6	18.6	26.4	76.8	71.4	41.9	50.4	43.6	45.1	23.8	47.1	76.3	83.9	65.4
+BERT↓	93.5	71.5	22.0	33.3	88.5	73.8	44.9	50.6	44.7	47.7	24.4	49.1	72.6	82.3	66.3
+BERT↑	93.4	72.0	23.7	33.2	90.4	73.9	42.8	50.1	44.0	48.3	24.9	48.0	72.9	83.0	65.9
S-Att.	89.9	81.8	22.7	32.8	75.7	78.1	34.1	38.9	40.8	44.8	26.1	38.2	60.7	81.1	57.6
+ELMo	96.6	87.8	24.9	30.6	74.1	79.1	36.0	41.4	39.2	44.1	26.4	37.9	64.1	83.4	64.7
+BERT↓	96.2	87.0	25.9	31.4	75.6	76.5	35.3	40.8	39.8	44.4	25.4	39.1	61.8	81.3	63.9
+BERT↑	96.5	87.3	26.1	32.6	76.8	78.0	34.7	40.9	40.0	44.0	25.7	38.1	62.2	81.7	63.8
S-Att.⊗	79.5	56.5	29.0	44.3	91.2	79.5	29.6	43.0	36.1	60.3	26.1	39.6	64.7	79.5	65.9
+ELMo	78.2	44.4	25.1	31.5	72.3	77.1	31.6	37.5	34.4	34.8	26.2	36.7	62.1	75.9	66.6
+BERT↓	82.4	66.9	36.2	33.2	74.9	76.8	32.0	37.6	38.0	41.3	27.4	37.6	63.0	79.8	66.7
+BERT↑	80.0	69.0	31.9	32.8	78.6	76.6	30.3	34.2	37.5	39.2	27.0	38.2	60.4	79.9	66.9
Table 2: TACRED probing task accuracies and model F1 scores on the test set.  $\uparrow$  and  $\downarrow$  indicate the cased and uncased version of BERT,  $\otimes$  models with entity masking. Probing task classification is performed by a logistic regression on the representations  $s_j$  of all sentences in the dataset.
Baseline performances are reported in the top section of Table 2 and Table 3. Length and ArgDist are both linear classifiers, which use sentence length and distance between head and tail argument as the only feature. BoE computes a representation of the input sentence by summing over the embeddings of all tokens it contains. Generally, there is a large gap between top baseline performance and that of a trained encoder. While SentLength and ArgDist are trivially solved by the respective linear classifier, BoE shows surprisingly good performance on SentLen and ArgOrd, and a clear improvement over the other baselines for named entity- and part-of-speech-related probing tasks.
Encoder Architecture For most probing tasks, except SentLen and ArgOrd, a proper encoder clearly outperforms bag-of-embeddings (BoE), which is coherent with the findings of Adi et al. (2017) and Conneau et al. (2018). Similarly, the results indicate that the prior imposed by the encoder architecture preconditions the information encoded in the learned embeddings. Models with a local or recency bias (CNN, BiLSTM) perform well on probing tasks with local focus, such as PosHead{L,R} and PosTail{L,R} and distance related tasks (ArgDist, ArgOrd). Similarly, models with access to dependency information (GCN) perform well on tree related tasks (SDPTreeDepth). Due to the graph pruning step (Zhang et al., 2018), the GCN is left with a limited view of the depen
Type Head	Type Tail	Sent Len	Arg Dist	PosL Head	PosR Head	PosL Tail	PosR Tail	Tree Dep	SDP Dep	GR Head	GR Tail	F1 score
Majority vote	22.0	21.3	25.7	42.1	62.1	39.3	38.3	34.0	25.4	67.2	37.3	80.9	-
Length	25.8	24.7	100.0	42.1	62.1	39.1	38.3	46.3	44.3	67.2	40.6	80.9	-
ArgDist	23.6	22.3	25.7	100.0	62.1	43.7	37.9	35.3	26.2	67.8	45.4	80.9	-
BoE	58.5	58.0	82.4	84.8	65.1	66.1	49.2	72.5	44.1	69.8	65.4	83.6	55.7
CNN	76.1	76.2	34.9	87.5	66.0	85.8	74.2	73.1	34.1	72.1	70.3	89.1	80.2
+ ELMo	81.3	81.8	38.1	88.5	70.0	89.0	79.5	76.4	35.5	71.8	75.1	90.9	84.4
+ BERT ↓	83.9	84.1	55.9	90.2	74.0	89.3	81.2	84.6	41.3	73.1	76.8	90.6	86.3
+ BERT ↑	83.4	83.7	54.3	90.4	74.4	89.4	82.0	82.8	42.0	73.0	78.3	90.8	86.0
Bi-LSTM	77.1	77.0	50.5	74.9	63.8	75.9	61.8	68.5	41.3	70.3	69.2	87.7	80.1
+ ELMo	81.5	81.8	41.1	66.6	62.8	71.8	59.3	64.5	37.5	70.1	70.0	87.6	83.7
+ BERT ↓	83.6	83.7	41.8	61.5	62.7	68.9	57.9	63.0	37.1	70.8	67.4	86.7	85.6
+ BERT ↑	82.5	82.8	41.8	66.0	63.1	70.8	58.6	64.3	37.7	71.0	68.9	87.5	85.1
GCN	75.4	75.5	35.0	81.5	68.5	87.5	71.2	55.5	35.5	80.3	76.3	91.7	79.6
+ ELMo	80.7	80.8	32.2	68.1	68.3	83.4	65.8	53.2	34.4	75.8	80.0	91.1	84.2
+ BERT ↓	82.5	83.0	42.5	66.5	73.6	84.7	69.2	66.3	38.9	77.2	82.1	91.0	85.7
+ BERT ↑	81.5	81.9	42.7	67.3	73.8	85.1	69.6	67.8	39.6	77.6	84.2	91.9	84.3
S-Att.	77.4	77.6	34.2	50.0	62.1	56.2	49.8	47.1	35.9	67.9	54.2	84.1	80.2
+ ELMo	80.7	81.3	33.1	46.2	62.0	53.9	49.1	45.7	34.7	68.1	54.9	84.4	83.6
+ BERT ↓	83.4	83.3	31.0	45.3	62.1	51.8	48.4	44.7	33.0	67.8	53.3	83.6	85.6
+ BERT ↑	82.8	82.8	30.6	46.1	62.1	52.7	48.2	44.4	33.6	67.9	54.6	84.1	84.9
Table 3: SemEval probing task accuracies and model F1 scores on the test set.  $\uparrow$  and  $\downarrow$  indicate the cased and uncased version of BERT. Probing task classification is performed by a logistic regression on the representations  $s_j$  of all sentences in the dataset.
dency tree, which explains the low performance on TreeDepth. Surprisingly, while Self-Attention exhibits superior performance on the RE task, it consistently performs lower on the probing tasks compared to the other encoding architectures. This could indicate Self-Attention encodes "deeper" linguistic information into the sentence representation, not covered by the current set of probing tasks.
Probing Tasks Compared to the baselines, all proper encoders exhibit consistently high performance on TypeHead and TypeTail, clearly highlighting the importance of entity type information to RE. In contrast, encoders trained on the downstream task perform worse on SentLen, which intuitively makes sense, since sentence length is mostly irrelevant for RE. This is consistent with Conneau et al. (2018), who found SentLen performance to decrease for models trained on more complex downstream tasks, e.g. neural machine translation, strengthen the assumption that, as a model captures deeper linguistic properties it will tend to forget about this superficial feature. With the exception of the CNN, all encoders consistently show low performance on the argument distance (ArgDist) task. A similar performance pattern can be observed for ArgOrd, where models that are biased towards locality (CNN and BiLSTM) perform bet
ter, while models that are able to efficiently model long range dependencies, such as GCN and S-Att., show lower performance. The superior RE task performance of the latter indicates that their bias may allow them to learn "deeper" linguistic features. The balanced performance of CNN, BiLSTM and GCN encoders across part-of-speech related tasks (PosHeadL, PosHeadR, PosTailL, PosTailR) highlights the importance of part-of-speech-related features to RE, again with the exception of S-Att., which performs just slightly above baselines. On TreeDepth and SDPTreeDepth (with GCN as the exception), average performance in many cases ranges just slightly above baseline performance, suggesting that TreeDepth requires more nuanced syntactic information, which the models fail to acquire. The good performance on grammatical role tasks (GRHead, GRTail) once more emphasizes the relevance of this feature to RE, with the GCN exhibiting the best performance on average. This is unsurprising, because the GCN focuses on token-level information along the dependency path connecting the arguments, and hence seems to be able to capture grammatical relations among tokens more readily than the other encoders (even though the GCN also does not have access to the dependency labels themselves).
Entity Masking Perhaps most interestingly, masking entity mentions with their respective named entity and grammatical role information considerably lowers the performance on entity type related tasks (TypeHead and TypeTail). This indicates that masking forces the encoder's focus away from the entity mentions, which is confirmed by the performance decrease in probing tasks with a focus on argument position and distance, e.g. ArgDist, ArgOrd, and SentLen. CNN and BiLSTM encoders exhibit the greatest decrease in performance, suggesting a severe overfitting to specific entity mentions when no masking is applied. In comparison, the GCN shows less tendency to overfit. Surprisingly, with entity masking the self-attentive encoder (S-Attn.) increases its focus on entity mentions and their surroundings as suggested by the performance increase on the distance and argument related probing tasks.
Word Representations Adding contextualized word representations computed by ELMo or BERT greatly increases performance on probing tasks with a focus on named entity and part-of-speech information. This indicates that contextualized word representations encode useful syntactic and semantic features relevant to RE, which is coherent with the findings of Peters et al. (2018) and Radford et al. (2018), who both highlight the effectiveness of linguistic features encoded in contextualized word representations (deep language representations) for downstream tasks. The improved performance on syntactic and semantic abilities is also reflected in an overall improvement in RE task performance. Compared to ELMo, encoders with BERT generally exhibit an overall better and more balanced performance on the probing tasks. This is also reflected in a superior RE performance, suggesting that a bidirectional language model encodes linguistic properties of the input more effectively. Somewhat surprisingly, BERT without casing performs equally or better on the probing tasks focused on entity and part-of-speech information, compared to the cased version. While this intuitively makes sense for SemEval, as the dataset focuses on semantic relations between concepts, it is surprising for TACRED, which contains relations between proper entities, e.g. person and company names, with casing information more important to identify the entity type.
Probing vs. Relation Extraction One interesting observation is that encoders that perform better on probing tasks do not necessarily perform better on the downstream RE task. For example, CNN+ELMo scores highest for most of the probing tasks, but has an 8.1 lower F1 score than the best model on this dataset, S-Att.+BERT cased with masking. Similarly, all variants of the self-attentive encoder (S-Att.) show superior performance on RE but consistently come up last on the probing tasks, occasionally performing just above the baselines. Conneau et al. (2018) observed a similar phenomena for encoders trained on neural machine translation.
Relation Extraction The relation extraction task performance $^{7}$  on the TACRED dataset ranges between 55.3 (Bi-LSTM) and 57.6 F1 (S-Att.), with performance improving to around 58.8 - 64.7 F1 when adding pre-learned, contextualized word representations. As observed in previous work (Zhang et al., 2017), masking helps the encoders to generalize better, with gains of around 4 - 8 F1 when compared to the vanilla models. This is mainly due to better recall, which indicates that without masking, models may overfit, e.g. by memorizing specific entity names. The best-performing model achieves a score of 66.9 F1 (S-Att.+ BERT cased and masking).
On the SemEval dataset performance of the vanilla models is around 80.0 F1. Adding contextualized word representations significantly improves the performance of all models, by 3.5 - 6 F1. The best-performing model on this dataset is a CNN with uncased BERT embeddings with an F1-score of 86.3, which is comparable to state-of-the-art models (Wang et al., 2016; Cai et al., 2016).
Shi et al. (2016) introduced probing tasks to probe syntactic properties captured in encoders trained on neural machine translation. Adi et al. (2017) extended this concept of "auxiliary prediction tasks", proposing SentLen, word count and word order tasks to probe general sentence encoders, such as bag-of-vectors, auto-encoder and skip-thought. Conneau et al. (2018) considered 10 probing tasks, including SentLen and TreeDepth, and an extended set of encoders such as Seq2Tree and encoders
See Appendix for more details on RE task performance, training, and model hyperparameters
trained on NMT and NLI for general text classification. Their setup, however, is not directly applicable to relation extraction, because the RE task requires not only the input sentence, but also the entity arguments. We therefore extend their framework to accommodate the RE setting. Another difference to their work is that while their probing tasks focus on linguistic properties of general sentence encoders, we specifically focus on relation extraction. To that end, we extend the evaluation to relation extraction by introducing a set of 14 probing tasks, including SentLen and TreeDepth, specifically designed to probe linguistic properties relevant to relation extraction.
We introduced a set of probing tasks to study the linguistic features captured in sentence encoder representations trained on relation extraction. We conducted a comprehensive evaluation of common RE encoder architectures, and studied the effect of explicitly and implicitly provided semantic and syntactic knowledge, uncovering interesting properties about the architecture and input features. For example, we found self-attentive encoders to be well suited for the RE on sentences of different complexity, though they consistently perform lower on probing tasks; hinting that these architectures capture "deeper" linguistic features. We also showed that the bias induced by different architectures clearly affects the learned properties, as suggested by probing task performance, e.g. for distance and dependency related probing tasks.
In future work, we want to extend the probing tasks to also cover specific linguistic patterns such as appositions, and also investigate a model's ability of generalizing to specific entity types, e.g. company and person names.
We would like to thank all reviewers for their helpful comments and feedback. This work has been supported by the German Federal Ministry of Education and Research as part of the projects DEEPLEE (01IW17001) and BBDC2 (01IS18025E), and by the German Federal Ministry for Economic Affairs and Energy as part of the project PLASS (01MD19003E).
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In Proceedings of ICLR Conference Track, Toulon, France.
Christoph Alt, Marc Hübner, and Leonhard Hennig. 2019a. Fine-tuning pre-trained transformer language models to distantly supervised relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1388-1398, Florence, Italy. Association for Computational Linguistics.
Christoph Alt, Marc Hubner, and Leonhard Hennig. 2019b. Improving relation extraction by pre-trained language representations. In Proceedings of AKBC 2019, pages 1-18, Amherst, Massachusetts.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861-872, Vancouver, Canada. Association for Computational Linguistics.
Razvan C. Bunescu and Raymond J. Mooney. 2005. A Shortest Path Dependency Kernel for Relation Extraction. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT '05, pages 724-731, Stroudsburg, PA, USA. Association for Computational Linguistics.
Rui Cai, Xiaodong Zhang, and Houfeng Wang. 2016. Bidirectional Recurrent Convolutional Neural Network for Relation Classification. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 756-765, Berlin, Germany. Association for Computational Linguistics.
Alexis Conneau and Douwe Kiela. 2018. Senteval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). European Language Resource Association.
Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. What you can cram into a single  $\$ \& !\#$ * vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. Computing Research Repository (CoRR), abs/1810.04805.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew E. Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. Allennlp: A deep semantic natural language processing platform. Computing Research Repository (CoRR), abs/1803.07640.
Mario Giulianielli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 240-248, Brussels, Belgium. Association for Computational Linguistics.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Seaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 33-38. Association for Computational Linguistics.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke S. Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. ArXiv, abs/1907.10529.
Thomas N. Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. Computing Research Repository (CoRR), abs/1609.02907.
Sebastian Krause, Hong Li, Hans Uszkoreit, and Feiyu Xu. 2012. Large-Scale Learning of Relation-Extraction Rules with Distant Supervision from the Web. In Philippe Cudr-Mauroux, Jeff Heflin, Evren Sirin, Tania Tudorache, Jrme Euzenat, Manfred Hauswirth, Josiane Xavier Parreira, Jim Hendler, Guus Schreiber, Abraham Bernstein, and Eva Blomqvist, editors, The Semantic Web ISWC 2012, number 7649 in Lecture Notes in Computer Science, pages 263-278. Springer Berlin Heidelberg.
Joohong Lee, Sangwoo Seo, and Yong Suk Choi. 2019. Semantic Relation Classification via Bidirectional LSTM Networks with Entity-aware Attention using Latent Entity Typing. arXiv:1901.08163 [cs]. ArXiv: 1901.08163.
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural Relation Extraction with Selective Attention over Instances. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2124-2133, Berlin, Germany. Association for Computational Linguistics.
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language
Processing Toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55-60, Baltimore, Maryland. Association for Computational Linguistics.
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant Supervision for Relation Extraction Without Labeled Data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2, ACL '09, pages 1003-1011, Stroudsburg, PA, USA. Association for Computational Linguistics.
Thien Huu Nguyen and Ralph Grishman. 2015. Relation extraction: Perspective from convolutional neural networks. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 39-48. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54, Hong Kong, China. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2015. Classifying Relations by Ranking with Convolutional Neural Networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 626-634, Beijing, China. Association for Computational Linguistics.
Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural mt learn source syntax? In Proceedings of the 2016 Conference on Empirical Meth