SlowGuess/ACL_2020_OCR huggingface dataset:
Notably, as the contextualized representations become stronger, the performance of MTL+FT becomes more favorable. 5 out of 15 tasks (NER, RE, OpenIE, SRL, ORL) observe statistically significant improvements (p-value  $< 0.05$  with paired bootstrap re-sampling) with SpanBERT, a contextualized embedding pre-trained with span-based training objectives, while only one task degrades (ABSA), indicating its superiority in reconciling spans from different tasks. The GLAD benchmark provides a holistic testbed for evaluating natural language analysis capability.
Task Relatedness Analysis To further investigate how different tasks interact with each other, we choose five source tasks (i.e., tasks used to improve other tasks, e.g., POS, NER, Consti., Dep., and SRL) that have been widely used in MTL (Hashimoto et al., 2017; Strubell et al., 2018) and six target tasks (i.e., tasks to be improved, e.g., OpenIE, NER, RE, ABSA, ORL, and SRL) to perform pairwise multi-task learning.
We hypothesize that although language modeling pre-training is theoretically orthogonal to MTL (Swayamdipta et al., 2018), in practice their benefits tend to overlap. To analyze these two factors separately, we start with a weak representation GloVe to study task relatedness, then move to BERT to demonstrate how much we can still improve with MTL given strong and contextualized representations. As shown in Table 6 (GloVe), tasks are not equally useful to each other. Notably, (1) for OpenIE and ORL, multi-task learning with SRL improves the performance significantly, while other tasks lead to less or no improvements. (2) Dependency parsing and SRL are generic source tasks that are beneficial to most of the target tasks. This unified SpanRel makes it easy to perform MTL and decide beneficial source tasks.
Next, we demonstrate that our framework also provides a platform for analysis of similarities and differences between different tasks. Inspired by the intuition that the attention coefficients are somewhat indicative of a model's internal focus (Li et al., 2016; Vig, 2019; Clark et al., 2019), we hypothesize that the similarity or difference between attention mechanisms may be correlated with similarity between tasks, or even the success or failure of MTL. To test this hypothesis, we extract the attention maps of two BERT-based SpanRel models (trained on a source  $t'$  and a target task  $t$  separately) over sentences  $\mathcal{X}_t$  from the target task, and compute
${}^{3}$  Span-based  ${\mathrm{F}}_{1}$  is used as the evaluation metric in SemEval-2010 Task 8 and SemEval-2014 Task 4 as opposed to macro  ${\mathrm{F}}_{1}$  and accuracy reported in the original papers because we aim at end-to-end extractions.
Category	Task	Metric	Dataset	GloVe	ELMo	BERTbase	SpanBERTbase
STL	MTL	+FT	STL	MTL	+FT	STL	MTL	+FT	STL	MTL	+FT
IE	NER	F1	CoNLL03	88.4	86.2↓	87.5↓	91.9	91.6	91.6	91.0	88.6↓	90.2↓	91.3	90.4↓	91.2
WLP	77.6	71.5↓	76.5↓	79.2	77.4	78.2↓	78.1	78.2	78.5	77.9	78.6↑	78.5↑
RE	F1	SemEval10	50.7	15.2↓	33.0↓	61.8	30.6↓	42.9↓	61.7	55.1↓	59.8↓	62.1	54.6↓	61.8
WLP	64.9	38.5↓	53.9↓	65.5	52.0↓	55.1↓	64.7	65.9↑	66.5↑	64.1	67.2↑	67.2↑
Coref	Avg F1	OntoNotes	56.3	50.3↓	53.0↓	62.2	62.9↑	63.3↑	66.2	65.5↓	65.8	70.0	68.9↓	69.7
OpenIE	F1	OIE2016	28.3	6.8↓	19.6↓	35.2	30.0↓	32.9↓	36.7	37.1	38.5↑	36.5	37.3↑	38.6↑
SRL	F1	OntoNotes	78.0	77.9	78.6↑	82.4	82.3	82.4	83.3	82.9	83.4	83.1	83.3	83.8↑
Parsing	Dep.	LAS	PTB	92.9	93.2	93.5↑	94.7	94.9	94.9	94.9	94.8	95.0	95.1	95.1	95.1
OntoNotes	90.4	90.5	90.5	92.3	93.2↑	92.8↑	94.1	93.8	94.0	94.2	94.1	94.2
Consti.	Evalb F1	PTB	93.4	-	93.8	95.3	-	95.3	95.5	-	95.2	95.8	-	95.5
OntoNotes	91.0	-	91.5↑	93.2	-	93.7↑	93.6	-	93.8	94.3	-	94.2
Sentiment	ABSA	F1	SemEval14	63.5	48.5↓	59.0↓	69.2	57.0↓	59.0↓	70.8	63.1↓	67.0↓	70.0	63.5↓	69.5↓
ORL	F1	MPQA 3.0	38.2	18.4↓	31.6↓	42.9	24.7↓	32.4↓	44.5	38.1↓	45.6↑	45.2	40.2↓	47.5↑
POS	Accuracy	PTB	96.8	96.8	96.8	97.7	97.7	97.8	97.6	97.3	97.3	97.6	97.6	97.6
OntoNotes	97.0	97.0	97.1	98.2	98.2	98.3	97.7	97.8	97.8	98.3	98.3	98.3
Table 5: Comparison between STL and MTL+fine-tuning across all tasks. blue↑ indicates results better than STL, red↓ indicates worse, and black means almost the same (i.e., a difference within 0.5). Constituency parsing requires more memory than other tasks so we restrict its span length to 10 in MTL, and thus do not report results.
their similarity using the Frobenius norm:
$$
\left. \operatorname {s i m} _ {k} (t, t ^ {\prime}) = - \frac {1}{| \mathcal {X} _ {t} |} \sum_ {\mathbf {x} \in \mathcal {X} _ {t}} \left\| A _ {k} ^ {t} (\mathbf {x}) - A _ {k} ^ {t ^ {\prime}} (\mathbf {x}) \right\| _ {F}, \right.
$$
where  $A_{k}^{t}(\mathbf{x})$  is the attention map extracted from the  $k$ -th head by running the model trained from task  $t$  on sentence  $\mathbf{x}$ . We select OpenIE as the target task because it shows the largest performance variation when paired with different source tasks (34.0 - 38.8) in Table 6. We visualize the attention similarity of all heads in BERT (12 layers × 12 heads) between two mutually harmful tasks (OpenIE/POS on the left) and between two mutually helpful tasks (OpenIE/SRL on the right) in Figure 2a. A common trend is that heads in higher layers exhibit more divergence, probably because they are closer to the prediction layer, thus easier to be affected by the end task. Overall, it can be seen that OpenIE/POS has much more attention divergence than OpenIE/SRL. A notable difference is that almost all heads in the last two layers of the OpenIE/POS models differ significantly, while some heads in the last two layers of the OpenIE/SRL models still behave similarly, providing evidence that failure of MTL can be attributed to the fact that dissimilar tasks require different attention patterns. We further compute average attention similarities for all source tasks in Figure 2b, and we can see that there is a strong correlation (Pearson correlation of 0.97) between the attentions similarity and the
(a) Attention similarity between OpenIE/POS (left), and between OpenIE/SRL (right) for all heads.
(b) Correlation between attention similarity and MTL performance.
Figure 2: Attention-based task relatedness analysis.
performance of pairwise MTL, supporting our hypothesis that attention pattern similarities can be used to predict improvements of MTL.
MTL under Different Settings We analyze how token representations and sizes of the target dataset affect the performance of MTL. Comparing BERT and GloVe in Table 6, the improvements become smaller or vanish as the token representation becomes stronger, e.g., improvement on OpenIE with SRL reduces from 5.8 to 1.6. This is expected because both large-scale pre-training and MTL aim to learn general representations and their benefits tend to overlap in practice. Interestingly, some helpful source tasks become harmful when we shift from GloVe to BERT, such as OpenIE paired with POS. We conjecture that the gains of MTL might have already been achieved by BERT, but the task-specific characteristics of POS hurt the performance of OpenIE. We did not observe many tasks benefitting from MTL for the GloVe-based model in Table 5
Source Target	GloVe	BERTbase
STL	POS	NER	Consti.	Dep.	SRL	STL	POS	NER	Consti.	Dep.	SRL
OpenIE	28.3	29.9↑	27.0↓	31.2↑	32.9↑	34.1↑	36.7	34.0↓	34.3↓	35.2↓	37.8↑	38.3↑
NER (WLP)	77.6	77.8	78.3↑	77.9	78.6↑	78.1↑	78.1	78.0	78.1	78.1	77.7	78.8↑
RE (WLP)	64.9	65.5↑	65.6↑	64.9	66.5↑	65.9↑	64.7	64.4	64.7	64.3	64.9	65.3↑
RE (SemEval10)	50.7	52.3↑	52.8↑	49.6↓	52.9↑	52.8↑	61.7	61.9	60.2↓	59.2↓	62.1	59.9↓
ABSA	63.5	63.4	62.8↓	59.8↓	63.5	60.2↓	70.8	68.9↓	71.4↑	70.4	69.9↓	69.6↓
ORL	38.2	35.7↓	37.9	36.1↓	38.6	41.0↑	44.5	45.8↑	44.2	44.8	45.1↑	46.6↑
SRL (10k)	68.8	69.6↑	68.9	70.7↑	71.3↑	-	78.7	79.4↑	79.5↑	79.6↑	79.8↑	-
Table 6: Performance of pairwise multi-task learning with GloVe and  $\mathrm{BERT}_{\mathrm{base}}$ . blue↑ indicates results better than STL, red↓ indicates worse, and black means almost the same (i.e., a difference within 0.5). We show the performance after fine-tuning. Dataset of source tasks POS, Consti., Dep. is PTB and dataset of NER is CoNLL-2003.
Figure 3: MTL Performance of SRL wrt. the data size.
because it is trained on all tasks (instead of two), which is beyond its limited model capacity. The improvements of MTL shrink as the size of the SRL datasets increases, as shown in Figure 3, indicating that MTL is useful when the target data is sparse.
Time Complexity Analysis Time complexities of span and relation prediction are  $\mathcal{O}(l\cdot n)$  and  $\mathcal{O}(K^2) = \mathcal{O}(\tau^2\cdot n^2)$  respectively for a sentence of  $n$  tokens (Section 3). The time complexity of BERT is  $\mathcal{O}(L\cdot n^2)$ , dominated by its  $L$  self-attention layers. Since the pruning threshold  $\tau$  is usually less than 1, the computational overhead introduced by the span-relation output layer is much less than BERT. In practice, we observe that the training/testing time is mainly spent by BERT. For SRL, one of the most computation-intensive tasks with long spans and dense span/relation annotations,  $85.5\%$  of the time is spent by BERT. For POS, a less heavy task, the time spent by BERT increases to  $98.5\%$ . Another option for span prediction is to formulate it as a sequence labeling task, as in previous works (Lample et al., 2016; He et al., 2017), where time complexity is  $\mathcal{O}(n)$ . Although slower than token-based labeling models, span-based models offer the advantages of being able to model overlapping spans and use span-level information for label prediction (Lee et al., 2017).
General Architectures for NLP There has been a rising interest in developing general architectures for different NLP tasks, with the most prominent examples being sequence labeling framework (Collobert et al., 2011; Ma and Hovy, 2016) used for tagging tasks and sequence-to-sequence framework (Sutskever et al., 2014) used for generation tasks. Moreover, researchers typically pick related tasks,
motivated by either linguistic insights or empirical results, and create a general framework to perform MTL, several of which are summarized in Table 1. For example, Swayamdipta et al. (2018) and Strubell et al. (2018) use constituency and dependency parsing to improve SRL. Luan et al. (2018, 2019); Wadden et al. (2019) use a span-based model to jointly solve three information-extraction-related tasks (NER, RE, and Coref.). Li et al. (2019) formulate both nested NER and flat NER as a machine reading comprehension task. Compared to existing works, we aim to create an output representation that can solve nearly every natural language analysis task in one fell swoop, allowing us to cover a far broader range of tasks with a single model.
In addition, NLP has seen a recent burgeoning of contextualized representations pre-trained on large corpora (e.g., ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019)). These methods focus on learning generic input representations, but are agnostic to the output representation, requiring different predictors for different tasks. In contrast, we present a methodology to formulate the output of different tasks in a unified format. Thus our work is orthogonal to those on contextualized embeddings. Indeed, in Section 4.3, we demonstrate that the SpanRel model can benefit from stronger contextualized representation models, and even provide a testbed for their use in natural language analysis.
Benchmarks for Evaluating Natural Language Understanding Due to the rapid development of NLP models, large-scale benchmarks, such as SentEval (Conneau and Kiela, 2018), GLUE (Wang et al., 2019b), and SuperGLUE (Wang et al., 2019a) have been proposed to facilitate fast and holistic evaluation of models' understanding ability. They
mainly focus on sentence-level tasks, such as natural language inference, while our GLAD benchmark focuses on token/phrase-level analysis tasks with diverse coverage of different linguistic structures. New tasks and datasets can be conveniently added to our benchmark as long as they are in the BRAT standoff format, which is one of the most commonly used data format in the NLP community, e.g., it has been used in the BioNLP shared tasks (Kim et al., 2009) and the Universal Dependency project (McDonald et al., 2013).
We provide the simple insight that a large number of natural language analysis tasks can be represented in a single format consisting of spans and relations between spans. As a result, these tasks can be solved in a single modeling framework that first extracts spans and predicts their labels, then predicts relations between spans. We attempted 10 tasks with this SpanRel model and show that this generic task-independent model can achieve competitive performance as state-of-the-art methods tailored for each tasks. We merge 8 datasets into our GLAD benchmark for evaluating future models for natural language analysis. Future directions include (1) devising hierarchical span representations that can handle spans of different length and diverse content more effectively and efficiently; (2) robust multitask learning or meta-learning algorithms that can reconcile very different tasks.
This work was supported by gifts from Bosch Research. We would like to thank Hiroaki Hayashi, Bohan Li, Pengcheng Yin, Hao Zhu, Paul Michel, and Antonios Anastasopoulos for their insightful comments and suggestions.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, LAW-ID@ACL 2013, August 8-9, 2013, Sofia, Bulgaria, pages 178-186.
Michele Banko, Michael J. Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In *IJCAI*
2007, Proceedings of the 20th International Joint Conference on Artificial Intelligence, Hyderabad, India, January 6-12, 2007, pages 2670-2676.
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ales Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT@ACL 2014, June 26-27, 2014, Baltimore, Maryland, USA, pages 12-58.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc Le. 2018. Semi-supervised sequence modeling with cross-view training. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1914-1925, Brussels, Belgium. Association for Computational Linguistics.
Michael Collins. 1997. Three generative, lexicalised models for statistical parsing. In 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics, pages 16-23, Madrid, Spain. Association for Computational Linguistics.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch. J. Mach. Learn. Res., 12:2493-2537.
Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Miyazaki, Japan. European Languages Resources Association (ELRA).
Lingjia Deng and Janyce Wiebe. 2015. MPQA 3.0: An entity/event-level sentiment corpus. In NAACL HLT 2015, The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, May 31 - June 5, 2015, pages 1323-1328.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),
pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Kalpit Dixit and Yaser Al-Onaizan. 2019. Span-level model for relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5308-5314, Florence, Italy. Association for Computational Linguistics.
Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Comput. Linguist., 28(3):245-288.
Jiang Guo, Wanxiang Che, Haifeng Wang, Ting Liu, and Jun Xu. 2016. A unified architecture for semantic role labeling and relation classification. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1264-1274, Osaka, Japan. The COLING 2016 Organizing Committee.
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple NLP tasks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 1923-1933.
Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018. Jointly predicting predicates and arguments in neural semantic role labeling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 364-369, Melbourne, Australia. Association for Computational Linguistics.
Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017. Deep semantic role labeling: What works and what's next. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 473-483, Vancouver, Canada. Association for Computational Linguistics.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Seaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval@ACL 2010, Uppsala University, Uppsala, Sweden, July 15-16, 2010, pages 33-38.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. CoRR, abs/1907.10529.
Jin-Dong Kim, Tomoko Ohta, Sampo Pyysalo, Yoshi-nobu Kano, and Jun'ichi Tsujii. 2009. Overview of bionlp'09 shared task on event extraction. In Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing: Shared
Task, BioNLP '09, pages 1-9, Stroudsburg, PA, USA. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Nikita Kitaev, Steven Cao, and Dan Klein. 2019. Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3499–3505, Florence, Italy. Association for Computational Linguistics.
Sandra Kübler, Ryan McDonald, and Joakim Nivre. 2009. Dependency parsing. Synthesis Lectures on Human Language Technologies, 1(1):1-127.
Chaitanya Kulkarni, Wei Xu, Alan Ritter, and Raghu Machiraju. 2018. An annotated corpus for machine reading of instructions in wet lab protocols. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 97-106, New Orleans, Louisiana. Association for Computational Linguistics.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In *NAACL HLT* 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 260-270.
Wuwei Lan and Wei Xu. 2018. Neural network models for paraphrase identification, semantic textual similarity, natural language inference, and question answering. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3890-3902, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 188-197.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Understanding neural networks through representation erasure. CoRR, abs/1612.08220.
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2019. A unified MRC framework for named entity recognition. CoRR, abs/1910.11476.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. In Proceedings
of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 4487-4496.
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In Proceedings of the SIGDIAL 2015 Conference, The 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2-4 September 2015, Prague, Czech Republic, pages 285-294.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 3219-3232.
Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3036-3046, Minneapolis, Minnesota. Association for Computational Linguistics.
Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-cnns-crf. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers.
Ana Marasović and Anette Frank. 2018. SRL4ORL: Improving opinion role labeling using multi-task learning with semantic role labeling. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 583-594, New Orleans, Louisiana. Association for Computational Linguistics.
Mitchell P. Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The penn treebank: Annotating predicate argument structure. In Human Language Technology, Proceedings of a Workshop held at Plainsboro, New Jerey, USA, March 8-11, 1994.
Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Tackström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013. Universal dependency annotation for multilingual parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 92–97, Sofia, Bulgaria. Association for Computational Linguistics.