Title: GENERALIZING NATURAL LANGUAGE ANALYSIS THROUGH SPAN-RELATION REPRESENTATIONS

Abstract: A large number of natural language processing tasks exist to analyze syntax, semantics, and information content of human language. These seemingly very different tasks are usually solved by specially designed architectures. In this paper, we provide the simple insight that a great variety of tasks can be represented in a single unified format consisting of labeling spans and relations between spans, thus a single task-independent model can be used across different tasks. We perform extensive experiments to test this insight on 10 disparate tasks as broad as dependency parsing (syntax), semantic role labeling (semantics), relation extraction (information content), aspect based sentiment analysis (sentiment), and many others, achieving comparable performance as state-of-the-art specialized models. We further demonstrate benefits in multi-task learning. We convert these datasets into a unified format to build a benchmark, which provides a holistic testbed for evaluating future models for generalized natural language analysis.

1. INTRODUCTION:

A large number of natural language processing (NLP) tasks exist to analyze various aspects of human language, including syntax (e.g., constituency and dependency parsing), semantics (e.g., semantic role labeling), information content (e.g., named entity recognition and relation extraction), or sentiment (e.g. sentiment analysis). At first glance, these tasks are seemingly very different in both the structure of their output and the variety of information that they try to capture. To handle these different characteristics, researchers usually use specially designed neural network architectures. In this paper we ask the simple questions: are the task-specific architectures really necessary? Or with the appropriate representational methodology, can we devise a single model that can perform -and achieve state-of-the-art performance on -a large number of natural language analysis tasks? Interestingly, in the domain of efficient human annotation interfaces, it is already standard to use unified representations for a wide variety of NLP tasks. On the right we show one example of the annotation interface BRAT (Stenetorp et al., 2012) , which has been used for annotating data for tasks as broad as part-of-speech tagging, named entity recognition, relation extraction, and many others. Notably, this interface has a single unified format that consists of spans (e.g. the span of an entity), labels on the spans (e.g. the variety of entity such as "person" or "location"), and labeled relations between the spans (e.g. "born-in"). These labeled relations can form a tree or graph structure (e.g., dependency tree), expressing the linguistic structure of sentences. We detail this BRAT format and how it can be used to represent a wide number of natural language analysis tasks in Section 2.

The simple hypothesis behind our paper is: if humans can perform natural language analysis in a single unified format, then perhaps machines can as well. Fortunately, there already exist NLP models that perform span prediction and prediction of relations between pairs of spans, such as the end-to-end neural coreference model of Lee et al. (2017) . We extend this model with minor architectural modifications (which are not our core contributions) and pre-trained contextualized (Peters et al., 2018) BERT (Devlin et al., 2019) BERT baseline (Shi & Lin, 2019) SpanBERT (Joshi et al., 2019) Single Model for Different Tasks ney et al., 2019a;b) , to the best of our knowledge this is the first work that uses such a unified model to actually perform analysis. Moreover, despite it simplicity we demonstrate that such a model can achieve comparable performance with special-purpose state-of-the-art models on the tasks above (Table 1 ). We also demonstrate that this framework allows us to easily perform multi-task learning among different tasks, leading to improvements when there are related tasks to be learned from or data is sparse. In summary, our contributions are:

• We provide the simple insight that a great variety of natural language analysis tasks can be represented and solved in a single unified format, i.e., span-relation representations. This insight may seem obvious in hindsight, but it has not been examined, particularly to this scale, by previous work on model-building for NLP. • We perform extensive experiments to test this insight on 10 disparate tasks, achieving comparable empirical results as the state-of-the-art, using a single task-independent modeling framework. • We further use this framework to perform an analysis of the benefits from multi-task learning across all of the tasks above, gleaning various insights about task relatedness and how multi-task learning performs with different token representations. • Upon acceptance of the paper, we will release our General Language Analysis Datasets (GLAD) benchmark with 8 datasets covering 10 tasks in the BRAT format, and provide a leaderboard to facilitate future work on generalized models for NLP. Compared to the full sentence-level tasks in the GLUE leaderboard (Wang et al., 2019a; b) , we cover a wide variety of natural language analysis tasks that require analyzing of the finer grained text units (e.g., words, phrases, clauses).

2. SPAN-RELATION REPRESENTATIONS:

In this section, we explain how the BRAT format can be used to represent a large number of tasks.

There are two fundamental types of annotations: span annotations and relation annotations. Given a sentence x = [w 1 , w 2 , ..., w n ] of n tokens, a span annotation (s i , l i ) consists of a contiguous span of tokens s i = [w bi , w bi+1 , ..., w ei ] and its label l i (l i ∈ L), where b i /e i are the start/end 2a and Table 2b . These tasks fall in two categories: span-oriented tasks, where the goal is to predict labeled spans (e.g., named entities in NER) and relation-oriented tasks, where the goal is to predict relations between two spans (e.g., relation between two entities in RE).

• Span-oriented Tasks (Table 2a )

-Named Entity Recognition (Sang & Meulder, 2003) NER is traditionally considered as a sequence labeling task. We model named entities as spans over one or more tokens. -Constituency Parsing (Collins, 1997) Constituency parsing aims to produce a syntactic parse tree for each sentence. Each node in the tree is an individual span, and spans are nested. -Part-of-speech Tagging (Ratnaparkhi, 1996; Toutanova et al., 2003) POS tagging is another sequence labeling task, where every single token is an individual span with a POS tag. -Aspect-based Sentiment Analysis (Pontiki et al., 2014) ABSA is a task that consists of identifying certain spans as aspect terms and predicting their associated sentiments. • Relation-oriented Tasks (Table 2b ) -Relation Extraction (Hendrickx et al., 2010) RE concerns the relation between two entities.

-Coreference (Pradhan et al., 2012) Coreference resolution is to link named, nominal, and pronominal mentions that refer to the same concept, within or beyond a single sentence. -Semantic Role Labeling (Gildea & Jurafsky, 2002 ) SRL aims to identify arguments of a predicate (verb or noun) and classify them with semantic roles in relation to the predicate. -Open Information Extraction (Banko et al., 2007; Niklaus et al., 2018) In contrast to the fixed relation types in RE, OpenIE aims to extract open-domain predicates and their arguments (usually subjects and objects) from a sentence. -Dependency Parsing (Kübler et al., 2009) Spans are single-word tokens and a relation links a word to its syntactic parent with the corresponding dependency type. -Opinion Role Labeling (Yang & Cardie, 2013) ORL detects spans that are opinion expressions, as well as holders and targets related to these opinions.

While the tasks above represent a remarkably broad swath of NLP, it is worth mentioning what we have not covered, to properly scope the work. Notably, sentence-level tasks such as text classification and natural language inference are not covered, although they can also be formulated using this span-relation representation by treating the entire sentence as a span. We chose to omit these tasks because they are already well-represented by previous work on generalized architectures (Lan & Xu, 2018) and multi-task learning (Devlin et al., 2019; Liu et al., 2019) , and thus we mainly focus on tasks using phrase-like spans. In addition, the span-relation representations described here are designed for natural language analysis, and cannot handle tasks that require generation of text, such as machine translation (Bojar et al., 2014) , dialog response generation (Lowe et al., 2015) , and summarization (Nallapati et al., 2016) . There are also a small number of analysis tasks such as semantic parsing to logical forms (Banarescu et al., 2013) where the outputs are not directly associated with spans in the input, and handling these tasks is beyond the scope of this work.

3. SPAN-RELATION MODEL:

Now that it is clear that a very large number of analysis tasks can be formulated in a single format, we turn to devising a single model that can solve these tasks. We base our model on a span-based model first designed for end-to-end coreference resolution (Lee et al., 2017) , which is then adapted for other tasks (He et al., 2018; Luan et al., 2018; 2019; Dixit & Al-Onaizan, 2019; Zhang & Zhao, 2019) . At the core of the model is a module to represent each span as a fixed-length vector, which is used to predict labels for spans or span pairs. We first briefly describe the span representation used and proven to be effective in previous works, then highlight some details we introduce to make this model generalize to a wide variety of tasks.

Span Representation Given a sentence x = [w 1 , w 2 , ..., w n ] of n tokens, a span s i = [w bi , w bi+1 , ..., w ei ] is represented by concatenating two components: a content representation z c i calculated as the weighted average across all token embeddings in the span, and a boundary representation z u i that concatenates the embeddings at the start/end positions of the span. Specifically,

EQUATION

EQUATION

where TokenRepr could be non-contextualized embeddings, such as GloVe (Pennington et al., 2014) , or contextualized embeddings, such as ELMo (Peters et al., 2018) , BERT (Devlin et al., 2019) , and SpanBERT (Joshi et al., 2019) . We refer to Lee et al. (2017) for further details.

Span and Relation Label Prediction Since we extract spans and relations in an end-to-end fashion, we introduce two additional labels NEG SPAN and NEG REL in L and R respectively. NEG SPAN indicates invalid spans (e.g., spans that are not named entities in NER) and NEG REL indicates invalid span pairs without any relation between them (i.e., no relation exists between two arguments in SRL). We first predict labels for all spans up to a length of l words using a multilayer perceptron (MLP): softmax(MLP span (z i )) ∈ ∆ |L| , where ∆ |L| is a |L|-dimensional simplex. Then we keep the top K = τ • n spans with the lowest NEG SPAN probabilities in relation prediction for efficiency, where smaller pruning threshold τ indicates more aggressive pruning. Another MLP is applied to pairs of the remaining spans to produce their relation score:

o jk = MLP rel ([z j ; z k ; z j •z k ]) ∈ R |R| . 2

Application to Disparate Tasks For most of the tasks, we can simply maximize the probability of the ground truth relation for all pairs of the remaining spans. However, some tasks might have different requirements, e.g., coreference resolution aims to cluster spans referring to the same concept and we do not care about which antecedent a span is linked to if there are multiple ones. (Collins, 1997) in constituency parsing.

decoding (Stern et al., 2017) to generate a valid parse tree. For dependency parsing, each word is linked to exactly one parent with the highest relation probability. For other tasks, we predict relations for all span pairs and use those not predicted as NEG REL to construct outputs.

Our core insight is that the above formulation is largely task-agnostic, meaning that a task can be modeled in this framework as long as it can be formulated as a span-relation prediction problem with properly defined span labels L and relation labels R. As shown in Table 1 , this unified Span-Relation (SpanRel) model makes it simple to scale to a large number of language analysis tasks, with breadth far beyond that of previous work.

Multi-task Learning:

The SpanRel model makes it easy to perform multi-task learning (MTL) by sharing all parameters except for the MLPs used for label prediction. With shared span representations, different tasks can learn from each other. However, because different tasks capture different linguistic aspects, they are not equally beneficial to each other. It is expected that jointly training on related tasks is helpful, while forcing the same model to solve unrelated tasks might even hurt the performance (Ruder, 2017) . Compared to manually choosing source tasks based on prior knowledge, which might be sub-optimal when the number of tasks is large, SpanRel offers a systematic way to examine relative benefits of source-target task pairs, as we will show in Section 4.3.

4. GLAD BENCHMARK AND RESULTS:

We first describe our General Language Analysis Datasets (GLAD) benchmark and evaluation metrics, then conduct experiments to (1) verify that SpanRel can achieve comparable performance across all tasks (Section 4.2), and (2) demonstrate its benefits in multi-task learning (Section 4.3).

4.1. EXPERIMENTAL SETTINGS:



GLAD Benchmark and Evaluation Metrics:

As summarized in Table 3 , we convert 8 widely used datasets with annotations of 10 tasks into the BRAT format and include them in the GLAD benchmark. It covers diverse domains, spans, and relations, and provides a holistic testbed for natural language analysis evaluation. The major evaluation metric is span-based F 1 (denoted as F 1 unless otherwise noted), a standard metric for SRL. Precision is the proportion of extracted spans (spans not predicted as NEG SPAN) that are consistent with the ground truth. Recall is the proportion of ground truth spans that are correctly extracted. F 1 is their harmonic mean. Span F 1 is also applicable to the case of relations, where an extracted relation (relations not predicted as NEG REL) is correct iff both head and tail spans have correct boundaries and the predicted relation label is correct. To make fair comparisons with existing works, we also compute standard metrics for different tasks, as listed in Table 3 . We refer to the corresponding papers for details. Implementation Details We attempted four token representation methods (Equation 1), namely GloVe (Pennington et al., 2014) , ELMo (Peters et al., 2018) , BERT (Devlin et al., 2019) , and Span-BERT (Joshi et al., 2019) . We use BERT base in our main results and report BERT large in Appendix A.

A three-layer BiLSTM with 256 hidden units is used (Equation 2). Both span and relation prediction MLPs have two layers with 128 hidden units. Dropout (Srivastava et al., 2014) of 0.5 is applied to all layers. For GloVe and ELMo, we use Adam (Kingma & Ba, 2015) with learning rate of 1e-3 and early stop with patience of 3. For BERT and SpanBERT, we follow standard fine-tuning and use Adam with learning rate of 5e-5, β 1 = 0.9, β 2 = 0.999, L2 weight decay of 0.01, warmup over the first 10% steps, and number of epochs tuned on development set. Task-specific hyperparameters maximal span length and pruning ratio are tuned on development set and listed in Appendix B.

4.2. COMPARISON WITH TASK-SPECIFIC SOTA METHODS:

We compare the SpanRel model with state-of-the-art task-specific models by training on data from a single task. By doing so we attempt to answer the research question "can a single model with minimal task-specific engineering achieve competitive or superior performance to other models that have been specifically engineered?" We select competitive SOTA models mainly based on settings, e.g., single-task learning and end-to-end extraction of spans and relations. To make fair comparisons, token embeddings (GloVe, ELMo, BERT) and other hyperparameters (e.g., the number of antecedents in Coref. and the maximal span length in SRL) in our method are set to match those used by SOTA models, to focus on differences brought about by the model architecture.

As shown in Table 4 , the SpanRel model achieves comparable performances as the task-specific SOTA methods (regardless of whether the token representation is contextualized or not). This indicates that the span-relation format can generically represent a large number of natural language analysis tasks and it is possible to devise a single unified model that can achieves strong performance on all of them. It provides a strong and generic baseline for natural language analysis tasks and a way to examine the usefulness of task-specific designs.

4.3. MULTI-TASK LEARNING WITH SPAN-RELATION REPRESENTATION:

To demonstrate the benefit of the SpanRel model in MTL, we perform single-task learning (STL) and MTL across all tasks using end-to-end settings. 4 Following Liu et al. (2019) , we perform MTL+finetuning and show the results in separate columns of blue↑ indicates results better than STL, red↓ indicates worse, and black means almost the same (i.e., a difference within 0.5). Constituency parsing requires more memory than other tasks so we restrict its maximal span length as 10 in MTL, thus it cannot form a valid tree.

is almost universally helpful to NLP tasks. Comparing the results of MTL+fine-tuning with STL, we found that performance with GloVe drops on 8 out of 15 tasks, most of which are tasks with relatively sparse data. It is probably because the capacity of the GloVe-based model is too small to store all the patterns required by different tasks. The results of contextualized representations are mixed, with some tasks being improved and others remaining the same or degrading. We hypothesize that this is because different tasks capture different linguistic aspects, thus are not equally helpful to each other. Reconciling these seemingly different tasks in the same model might be harmful to some tasks. Notably, as the contextualized representations become stronger, the performance of MTL+FT becomes more favorable. 5 out of 15 tasks (NER, RE, OpenIE, SRL, ORL) observe improvements with SpanBERT, a contextualized embedding pre-trained with span-based training objectives, while only one task degrades (ABSA), indicating its superiority in reconciling spans from different tasks.

The GLAD benchmark provides a holistic testbed for evaluating natural language analysis capability.

Task Relatedness Analysis To further investigate how different tasks interact with each other, we choose five source tasks (POS, NER, Consti., Dep., and SRL) that have been widely used in MTL (Hashimoto et al., 2017; Strubell et al., 2018) and six target tasks (OpenIE, NER, RE, ABSA, ORL, and SRL) to perform pairwise multi-task learning. We hypothesize that although language modeling pre-training is theoretically orthogonal to MTL (Swayamdipta et al., 2018) , in practice the benefit of pre-training tends to overlap or even overshadow the benefit of MTL. To analyze these two factors separately, we start with a weak representation GloVe to study task relatedness, then move to BERT to demonstrate how much we can still improve with MTL given strong and contextualized representations. As shown in Table 6 (GloVe), tasks are not equally useful to each other. Notably, (1) for OpenIE and ORL, multi-task learning with SRL improves the performance significantly, while other tasks lead to less or no improvements. (2) Dependency parsing and SRL are generic source tasks that are beneficial to most of the target tasks.

(3) ABSA is quite different from the source tasks and no improvement is observed with MTL. This unified SpanRel makes it easy to perform MTL and decide beneficial source tasks.

MTL under Different Settings:

We analyze how token representations and sizes of the target dataset affect the performance of MTL. Comparing BERT and GloVe in Table 6 , the improvements of MTL become smaller or vanish as the token representation becomes stronger, e.g., improvement on OpenIE with SRL reduces from 5.8 to 1.6 and improvement on SRL with Dep. reduces from 2.5 to 1.1. This is expected because both large-scale pre-training and MTL aim to learn general representations and their benefits tend to overlap in practice. Interestingly, some helpful source tasks even become harmful when we shift from GloVe to BERT, such as OpenIE paired with Consti. or POS. We conjecture that the gains of MTL might have already been achieved by BERT, but the taskspecific characteristics of Consti. and POS hurt the performance of OpenIE. The improvements of MTL shrink as we increase the size of the SRL datasets, as shown in Figure 2 , indicating that MTL is more useful when the target data is sparse. ). Compared to existing works, we aim to create an output representation that can solve nearly every natural language analysis task in one fell swoop, allowing us to cover a far broader range of tasks with a single model.

5. RELATED WORK:

In addition, NLP has seen a recent burgeoning of contextualized token embeddings pre-trained on large corpra (e.g., ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) ). These methods focus on learning generic input representations, but are agnostic to the output representation, requiring different predictors to be designed for different tasks. In contrast, we present a methodology to formulate the output of different tasks in a unified format. Thus our work is orthogonal to those on contextualized embeddings. Indeed, in Section 4.3, we demonstrate that the SpanRel model can benefit from stronger contextualized representation models, and even provide a testbed for their use in natural language analysis.

Benchmarks for Evaluating Natural Language Understanding Due to the rapid development of NLP models, large-scale benchmarks, such as SentEval (Conneau & Kiela, 2018) , GLUE (Wang et al., 2019b) , and SuperGLUE (Wang et al., 2019a) have been proposed to facilitate fast and holistic evaluation of models' natural language understanding ability. They mainly focus on sentence-level tasks, such as text classification and natural language inference, while our GLAD benchmark focuses on token/phrase-level analysis tasks with diverse coverage of different linguistic structures. With different tasks represented under the same format, a model can be easily evaluated on all our tasks, reflecting various aspects of its natural language analysis capability. New tasks and datasets can be conveniently added to our benchmark as long as they are in the BRAT standoff format, which is one of the most commonly used data format in the NLP community, e.g., it has been used in the BioNLP shared tasks (Kim et al., 2009) and the Universal Dependency project (McDonald et al., 2013) .

6. CONCLUSION:

We provide the simple insight that a large number of natural language analysis tasks can be represented in a single format consisting of spans and relations between spans. As a result, these tasks can be solved in a single modeling framework that first extracts spans and predicts their labels, then predicts relations between extracted spans. We attempted 10 tasks with this SpanRel model under this unified representation and show that this generic task-independent model can achieve competitive performance as state-of-the-art methods tailored for each tasks. We merge 8 datasets into our GLAD benchmark for evaluating future models for natural language analysis. Future directions include (1) devising hierarchical span representations that can handle spans of different length and diverse content and length more effectively and efficiently. (2) robust multitask learning or metalearning algorithms that can learn to relate and reconcile very different tasks.

A RESULTS OF BERT LARGE MODEL 

:

To contrast to work on pre-trained contextualized representations like ELMo(Peters et al.,

2018) or BERT(Devlin et al., 2019), these works learn unified features to represent the input in different tasks, whereas we propose a unified representational methodology that represents the output of different tasks. Analysis models using BERT still designed special-purpose output predictors for specific tasks or task classes.

The time complexity of span prediction is O(l • n) for a sentence of n tokens, and the time complexity of relation prediction is O(K 2 ) = O(τ 2 • n 2 ). Another option for span prediction is to formulate it as a sequence labeling task, as did in previous works on SRL(He et al., 2017) and many others(Lample et al., 2016;Stanovsky et al., 2018), and their time complexity is O(n). Although slower than token-based labeling models, span-based models offer the advantages of being able to model overlapping spans (e.g., overlapping arguments in SRL) and exploring span-level information for span prediction.

• The small version ofLee et al. (2017)'s method with 100 antencedents and no speaker features. For OpenIE and ORL, we use span-based F1 instead of syntactic-head-based F1 and binary coverage F1 used in the original papers because they are biased towards extracting long spans. † For SRL, we choose to compare withHe et al. (2018) because they also extract predicates and arguments in an end-to-end way. We followXu et al. (2019) to report accuracy of restaurant and laptop domain separately in ABSA.

Span-based F1 is used as the evaluation metric in SemEval-2010 Task 8 and SemEval-2014 Task 4 as opposed to macro F1 and accuracy reported in the original papers because we aim at end-to-end extractions.

Figures and Tables:

Figure 1: An example from BRAT, consisting of POS, NER, and RE.

Figure 2: MTL Performance of SRL wrt. the data size.

The unified span-relation model can work on multiple NLP tasks, in contrast to previous works usually designed for a subset of tasks.

Table content: <table><tr><td>Guo et al. (2016)</td></tr><tr><td>Swayamdipta et al. (2018)</td></tr><tr><td>Strubell et al. (2018)</td></tr><tr><td>Clark et al. (2018)</td></tr><tr><td>Luan et al. (2018; 2019)</td></tr><tr><td>Dixit &amp; Al-Onaizan (2019)</td></tr><tr><td>Marasović &amp; Frank (2018)</td></tr><tr><td>Hashimoto et al. (2017)</td></tr><tr><td>This Work</td></tr></table>

respectively, and L is a set of span labels. A relation annotation (s j , s k , r jk ) refers to a relation r jk (r jk ∈ R) between the head span s j and the tail span s k , where R is a set of relation types. This span-relation representation can easily express many tasks by defining different L and R, as summarized in Table

Table content: <table><tr><td>Task</td><td colspan="7">Spans annotated with labels</td><td>Task</td><td>Spans and relations annotated with labels</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>cause-effect</td></tr><tr><td>NER</td><td colspan="4">Barack Obama</td><td colspan="4">was born in Hawaii .</td><td>RE</td><td>The burst has been caused by pressure.</td></tr><tr><td/><td colspan="2">person</td><td/><td/><td/><td/><td>location</td><td>coref.</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>Coref.</td><td>I voted for Tom because he is clever.</td></tr><tr><td colspan="6">Consti. And their suspicions</td><td colspan="2">of each other</td><td>run deep .</td><td>ARG1</td></tr><tr><td/><td/><td/><td colspan="2">NP</td><td/><td/><td>NP</td><td>ADVP</td><td>ARG0</td><td>ARG2</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td>PP</td><td>VP</td><td>SRL</td><td>We brought you the tale of two cities.</td></tr><tr><td/><td/><td/><td/><td/><td>NP</td><td/><td/><td>ARG0</td><td>ARG1</td></tr><tr><td/><td/><td/><td/><td/><td/><td colspan="2">S</td><td>OpenIE</td><td>The four lawyers climbed out from under a table.</td></tr><tr><td>POS</td><td>What</td><td>kind</td><td>of</td><td colspan="3">memory</td><td>?</td><td>det</td><td>dobj</td></tr><tr><td/><td>WP</td><td>NN</td><td>IN</td><td/><td>NN</td><td/><td/><td>amod</td><td>nsubj</td><td>advmod nummod</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>Dep.</td><td>The entire division employs about 850 workers.</td></tr><tr><td colspan="9">ABSA Great laptop that offers many great features !</td><td>holder</td><td>target</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>positive</td><td>ORL</td><td>We therefore as MDC do not accept this result.</td></tr><tr><td colspan="9">Table 2a: Span-oriented tasks. Spans are annotated</td><td>Table 2b: Relation-oriented tasks. Directed arcs indicate</td></tr><tr><td colspan="7">by underlines and their labels.</td><td/><td>the relations between spans.</td></tr></table>

Pairwise Maximize the probabilities of the ground truth relations for all pairs of the remaining spans independently: softmax(o jk ) r jk , where r jk indexes the ground truth relation. 2. Head Maximize the probability of selecting the ground truth head spans for a specific span s j : k∈head(sj ) softmax([o j1 , o j2 , ..., o jK ]) k , where head(•) returns indices of one or multiple heads and o j• is the corresponding scalar from o j• indicating how likely two spans are related.We use option 1 for all tasks except for coreference resolution which uses option 2. Note that the above loss functions only differ in how relation scores are normalized and the other parts of the model remain the same across different tasks. At test time, we follow previous inference methods to generate valid outputs. For coreference resolution, we link a span to the antecedent with highest score and build clusters(Lee et al., 2017). For constituency parsing, we use the greedy top-down

Table content: <table><tr><td>Dataset</td><td>Domain</td><td>#Sent.</td><td>Task</td><td colspan="2">#Spans #Relations</td><td>Metric</td></tr><tr><td>Wet Lab Protocols (Kulkarni et al., 2018)</td><td>biology</td><td>14,301</td><td>NER RE</td><td>60,745 60,745</td><td>-43,773</td><td>F1 F1</td></tr><tr><td>CoNLL-2003 (Sang &amp; Meulder, 2003)</td><td>news</td><td>20,744</td><td>NER</td><td>35,089</td><td>-</td><td>F1</td></tr><tr><td>SemEval-2010 Task 8 (Hendrickx et al., 2010)</td><td>misc.</td><td>10,717</td><td>RE</td><td>21,437</td><td colspan="2">10,717 Macro F1 •</td></tr><tr><td/><td/><td/><td>Coref.</td><td>194,477</td><td>1,166,513</td><td>Avg F1</td></tr><tr><td>OntoNotes 5.0 (Pradhan et al., 2013)</td><td>misc.</td><td>94,268</td><td>SRL POS Dep.</td><td>745,796 1,631,995 1,722,571</td><td>543,534 -1,628,558</td><td>F1 Accuracy LAS</td></tr><tr><td/><td/><td/><td colspan="2">Consti. 1,320,702</td><td>-</td><td>Evalb F1  †</td></tr><tr><td>Penn Treebank (Marcus et al., 1994)</td><td>speech, news</td><td colspan="2">49,208 43,948 43,948 Consti. POS Dep.</td><td>1,173,766 1,090,777 871,264</td><td>-1,046,829 -</td><td>Accuracy LAS Evalb F1  †</td></tr><tr><td>OIE2016 (Stanovsky &amp; Dagan, 2016)</td><td>news, Wiki</td><td>2,534</td><td>OpenIE</td><td>15,717</td><td>12,451</td><td>F1</td></tr><tr><td>MPQA 3.0 (Deng &amp; Wiebe, 2015)</td><td>news</td><td>3,585</td><td>ORL</td><td>13,841</td><td>9,286</td><td>F1</td></tr><tr><td>SemEval-2014 Task 4 (Pontiki et al., 2014)</td><td>reviews</td><td>4,451</td><td>ABSA</td><td>7,674</td><td colspan="2">-Accuracy •</td></tr></table>

Statistics of the GLAD benchmark consisting of 10 tasks from 8 datasets. Following He et al. (2018), we use a subset of OntoNotes 5.0 dataset based on CoNLL 2012 splits (Pradhan et al., 2012). • Previous works use gold standard spans in these evaluations.

Table content: <table/>

Comparison between the SpanRel model and the task-specific state-of-the-art models.3  FollowingLuan et al. (2019), we perform NER and RE jointly on WLP dataset. We use gold entities in SemEval-2010 Task 8, gold aspect terms in SemEval-2014 Task 4, and gold opinion expressions in MPQA 3.0 to be consistent with existing works.

Table content: <table><tr><td>Category</td><td>Task</td><td>Metric</td><td>Dataset</td><td>Setting</td><td>SOTA Model</td><td colspan="2">Previous SOTA Our Model</td></tr><tr><td/><td>NER</td><td>F 1</td><td colspan="2">CoNLL03 BERT WLP ELMo</td><td>Devlin et al. (2019) Luan et al. (2019)</td><td>92.8 79.5</td><td>92.2 79.2</td></tr><tr><td>IE</td><td>RE</td><td colspan="3">Macro F1 SemEval10 BERT, gold F 1 WLP ELMo</td><td>Wu &amp; He (2019) Luan et al. (2019)</td><td>89.3 64.1</td><td>87.4 65.5</td></tr><tr><td/><td>Coref.</td><td>Avg F 1</td><td colspan="3">OntoNotes GloVe, CharCNN Lee et al. (2017) •</td><td>62.0</td><td>61.1</td></tr><tr><td/><td>OpenIE</td><td>F 1</td><td>OIE2016</td><td>ELMo</td><td>Stanovsky et al. (2018)</td><td>31.1</td><td>35.2</td></tr><tr><td>SRL</td><td/><td>F 1</td><td colspan="2">OntoNotes ELMo</td><td>He et al. (2018)  †</td><td>82.9</td><td>82.4</td></tr><tr><td>Parsing</td><td>Dep.</td><td>LAS</td><td>PTB</td><td>ELMo</td><td>Clark et al. (2018)</td><td>94.4</td><td>94.7</td></tr><tr><td/><td>Consti.</td><td>Evalb F 1</td><td>PTB</td><td>BERT</td><td>Kitaev et al. (2019)</td><td>95.6</td><td>95.5</td></tr><tr><td>Sentiment</td><td colspan="4">ABSA Accuracy SemEval14 BERT, gold</td><td>Xu et al. (2019)</td><td>85.0/78.1</td><td>85.5/76.6</td></tr><tr><td/><td>ORL</td><td>F 1</td><td colspan="2">MPQA 3.0 GloVe, gold</td><td>Marasović &amp; Frank (2018)</td><td>56.4</td><td>55.6</td></tr><tr><td>POS</td><td/><td>Accuracy</td><td>PTB</td><td>ELMo</td><td>Clark et al. (2018)</td><td>97.7</td><td>97.7</td></tr></table>

Contextualized token representations yield significantly better results than GloVe on all tasks, indicating that pre-training on large corpora

Table content: <table><tr><td/><td/><td/><td/><td>GloVe</td><td>ELMo</td><td>BERT base</td><td>SpanBERT base</td></tr><tr><td>Category</td><td>Task</td><td>Metric</td><td>Dataset</td><td colspan="3">STL MTL +FT STL MTL +FT STL MTL +FT STL MTL +FT</td></tr><tr><td/><td>NER</td><td>F 1</td><td>CoNLL03 WLP</td><td colspan="3">88.4 86.2↓ 87.5↓ 91.9 91.6 91.6 77.6 71.5↓ 76.5↓ 79.2 77.4↓ 78.2↓ 78.1 78.2 78.5 91.0 88.6↓ 90.2↓ 91.3 90.4↓ 91.2 77.9 78.6↑ 78.5↑</td></tr><tr><td>IE</td><td>RE</td><td>F 1</td><td colspan="4">SemEval10 50.7 15.2↓ 33.0↓ 61.8 30.6↓ 42.9↓ 61.7 55.1↓ 59.8↓ 62.1 54.6↓ 61.8 WLP 64.9 38.5↓ 53.9↓ 65.5 52.0↓ 55.1↓ 64.7 65.9↑ 66.5↑ 64.1 67.2↑ 67.2↑</td></tr><tr><td/><td>Coref</td><td>Avg F 1</td><td colspan="4">OntoNotes 56.3 50.3↓ 53.0↓ 62.2 62.9↑ 63.3↑ 66.2 65.5↓ 65.8</td><td>70.0 68.9↓ 69.7</td></tr><tr><td/><td>OpenIE</td><td>F 1</td><td>OIE2016</td><td colspan="3">28.3 6.8↓ 19.6↓ 35.2 30.0↓ 32.9↓ 36.7 37.1 38.5↑ 36.5 37.3↑ 38.6↑</td></tr><tr><td>SRL</td><td/><td>F 1</td><td colspan="3">OntoNotes 78.0 77.9 78.6↑ 82.4 82.3 82.4</td><td>83.3 82.9 83.4</td><td>83.1 83.3 83.8↑</td><td>c</td></tr><tr><td>Parsing</td><td>Dep.</td><td>LAS</td><td colspan="4">PTB OntoNotes 90.4 90.5 90.5 92.9 93.2 93.5↑ 94.7 94.9 94.9 92.3 93.2↑ 92.8↑ 94.1 93.8 94.0 94.9 94.8 95.0</td><td>95.1 95.1 95.1 94.2 94.1 94.2</td></tr><tr><td/><td colspan="2">Consti. Evalb F 1</td><td colspan="4">PTB OntoNotes 91.0 -91.5↑ 93.2 -93.7↑ 93.6 -93.8 93.4 -93.8 95.3 -95.3 95.5 -95.2</td><td>95.8 -95.5 94.3 -94.2</td></tr><tr><td>Sentiment</td><td>ABSA ORL</td><td>F 1 F 1</td><td colspan="4">SemEval14 63.5 48.5↓ 59.0↓ 69.2 57.0↓ 59.0↓ 70.8 63.1↓ 67.0↓ 70.0 63.5↓ 69.5↓ MPQA 3.0 38.2 18.4↓ 31.6↓ 42.9 24.7↓ 32.4↓ 44.5 38.1↓ 45.6↑ 45.2 40.2↓ 47.5↑</td></tr><tr><td>POS</td><td/><td>Accuracy</td><td colspan="2">PTB OntoNotes 97.0 97.0 97.1 96.8 96.8 96.8</td><td>97.7 97.7 97.8 98.2 98.2 98.3</td><td>97.6 97.3 97.3 97.7 97.8 97.8</td><td>97.6 97.6 97.6 98.3 98.3 98.3</td></tr></table>

Comparison between STL and MTL+fine-tuning of the SpanRel model across all tasks.

Table content: <table/>

Performance of pairwise multi-task learning with GloVe and BERT base . blue↑ indicates results better than STL, red↓ indicates worse, and black means almost the same (i.e., a difference within 0.5). We show the performance after fine-tuning. Dataset of source tasks POS, Consti., Dep. is PTB and dataset of NER isCoNLL-2003.

Table content: <table><tr><td/><td/><td/><td>GloVe</td><td/><td>BERT base</td></tr><tr><td>Target</td><td colspan="3">Source STL POS NER Consti. Dep. SRL</td><td colspan="2">STL POS NER Consti. Dep. SRL</td></tr><tr><td colspan="2">OpenIE</td><td colspan="2">28.3 29.9↑ 27.0↓ 31.2↑ 32.9↑ 34.1↑</td><td colspan="2">36.7 34.0↓ 34.3↓ 35.2↓ 37.8↑ 38.3↑</td></tr><tr><td colspan="2">NER (WLP)</td><td>77.6</td><td>77.8 78.3↑ 77.9 78.6↑ 78.1↑</td><td>78.1</td><td>78.0 78.1 78.1 77.7 78.8↑</td></tr><tr><td colspan="2">RE (WLP)</td><td colspan="2">64.9 65.5↑ 65.6↑ 64.9 66.5↑ 65.9↑</td><td>64.7</td><td>64.4 64.7 64.3 64.9 65.3↑</td></tr><tr><td colspan="4">RE (SemEval10) 50.7 52.3↑ 52.8↑ 49.6↓ 52.9↑ 52.8↑</td><td>61.7</td><td>61.9 60.2↓ 59.2↓ 62.1 59.9↓</td></tr><tr><td colspan="2">ABSA</td><td>63.5</td><td>63.4 62.8↓ 59.8↓ 63.5 60.2↓</td><td colspan="2">70.8 68.9↓ 71.4↑ 70.4 69.9↓ 69.6↓</td></tr><tr><td colspan="2">ORL</td><td colspan="2">38.2 35.7↓ 37.9 36.1↓ 38.6 41.0↑</td><td colspan="2">44.5 45.8↑ 44.2 44.8 45.1↑ 46.6↑</td></tr><tr><td colspan="2">SRL (10k)</td><td colspan="2">68.8 69.6↑ 68.9 70.7↑ 71.3↑ -</td><td colspan="2">78.7 79.4↑ 79.5↑ 79.6↑ 79.8↑ -</td></tr></table>

General Architectures for NLP There has been a rising interest in developing general architectures for different NLP tasks, with the most prominent examples being sequence labeling framework(Collobert et al., 2011;Ma & Hovy, 2016) used for tagging tasks (e.g., NER, POS) and sequenceto-sequence framework(Sutskever et al., 2014) used for generation tasks (e.g., machine translation). Moreover, researchers typically pick related tasks, motivated by either linguistic insights or empirical results, and create a general framework to perform MTL, several of which are summarized in Table1. For example, based on the belief that semantic structure of a sentence should conform with syntactic structure,Swayamdipta et al. (2018) andStrubell et al. (2018) use constituency and dependency parsing to improve SRL.Luan et al. (2018; 2019)  use a span-based model to jointly solve three information-extraction-related tasks (NER, RE, and Coref.

Table content: <table/>

Single-task learning performance of the SpanRel model with different token representations. BERT large requires a large amount of memory so we cannot feed the entire document to the model in coreference resolution.

Table content: <table><tr><td>Category</td><td>Task</td><td>Metric</td><td>Dataset</td><td colspan="5">GloVe ELMo BERT base SpanBERT base BERT large</td></tr><tr><td/><td>NER</td><td>F 1</td><td>CoNLL03 WLP</td><td>88.4 77.6</td><td>91.9 79.2</td><td>91.0 78.1</td><td>91.3 77.9</td><td>90.9 78.3</td></tr><tr><td>IE</td><td>RE</td><td>F 1</td><td>SemEval10 WLP</td><td>50.7 64.9</td><td>61.8 65.5</td><td>61.7 64.7</td><td>62.1 64.1</td><td>64.7 65.1</td></tr><tr><td/><td>Coref</td><td>Avg F 1</td><td>OntoNotes</td><td>56.3</td><td>62.2</td><td>66.3</td><td>70.0</td><td>-</td></tr><tr><td/><td>OpenIE</td><td>F 1</td><td>OIE2016</td><td>28.3</td><td>35.2</td><td>36.7</td><td>36.5</td><td>36.5</td></tr><tr><td>SRL</td><td/><td>F 1</td><td>OntoNotes</td><td>78.0</td><td>82.4</td><td>83.3</td><td>83.1</td><td>84.4</td></tr><tr><td>Parsing</td><td>Dep.</td><td>LAS</td><td>PTB OntoNotes</td><td>92.9 90.4</td><td>94.7 92.3</td><td>94.9 94.1</td><td>95.1 94.2</td><td>95.3 94.5</td></tr><tr><td/><td colspan="2">Consti. Evalb F 1</td><td>PTB OntoNotes</td><td>93.4 91.0</td><td>95.3 93.2</td><td>95.5 93.6</td><td>95.8 94.3</td><td>95.8 93.9</td></tr><tr><td>Sentiment</td><td>ABSA</td><td>F 1</td><td>SemEval14</td><td>63.5</td><td>69.2</td><td>70.8</td><td>70.0</td><td>73.8</td></tr><tr><td/><td>ORL</td><td>F 1</td><td>MPQA 3.0</td><td>38.2</td><td>42.9</td><td>44.5</td><td>45.2</td><td>47.1</td></tr><tr><td>POS</td><td/><td>Accuracy</td><td>PTB OntoNotes</td><td>96.8 97.0</td><td>97.7 98.2</td><td>97.6 97.7</td><td>97.6 98.3</td><td>97.4 97.9</td></tr></table>

Peer review (iclr2020):
- Overview: This work presents a unified formulation of various (phrase and token level) NLP tasks. They use this formulation to devise a single model architecture that, when trained individually on 10 diverse NLP tasks, can get near SOTA performance on all of them. They additionally conduct experiments to determine when multitask learning is useful for this set of tasks, which is made easy due to the unified framework. The stated main contributions are - unified formulation of 10 diverse NLP tasks, including NER, constituency + dependency, parsing, POS tagging, sentiment, coref., relation extraction, SRL, information extraction, and more - a benchmark of pre-existing datasets for each of these tasks - experiments on their benchmark in pushing the state of the art - experiments leveraging their task formulation and benchmark to analyze the effectiveness of multitask learning under a variety of settings. - Review: The paper as-is is not as novel as the authors claim, and I recommend a weak reject. That said, I think there is a lot of interesting work and analyses in the latter half of the work, and I think with some reframing, this could be a really nice work. - I*m not convinced of the novelty of the formulating various NLP tasks as span and and relation labelling. As the authors point out (somewhat concerningly buried on a footnote on page 4), Tenney et al.[1] use present a very similar framework for these tasks, also with the aim of having uniform task format and model architecture. The authors rightly point out a key difference between their work and [1]: The latter*s work uses this unified framework in the context of analyzing pertained models (with no training), rather than training a model to perform as well as on these tasks as possible. As a result, [1] uses gold span information, whereas this paper also needs to predict those spans. While this difference in setting is fairly substantial, it doesn*t change the fact that [1] previously unified a similar suite of tasks with more or less the same framework, which I believe significantly weakens this work*s main stated claim of novelty. - I appreciate the effort to standardize analysis tasks in NLP. However, I think there*s some tension in trying to group the proposed set of tasks into a sub-sentence-level language understanding benchmark and calling it an analysis tool. In providing a training set for each task and expecting users of the benchmark to train/fine-tune on each task, we lose the ability to say a pretrained model contains information for X task (e.g. *My model has a strong sense of named entities*). On the other hand, I suppose you gain some insight into how well your pertained model can be adapted for these NLP tasks, which is also useful but I think not how analysis tasks are usually implemented. - Matching previous (single-task) SOTA on each of the 10 tasks individually doesn*t seem hugely interesting to me. It doesn*t seem too surprising that pretained LMs perform well regardless of the task format / architecture. We see this in question answering where concatenating the question to the context is as effective as QA-specific architectures [2]. - That being said, I think there is still a lot of interesting work and results here. For me, the most interesting part is the analysis for which tasks are mutually beneficial or harmful. Determining and characterizing mutually beneficial tasks is a big open question, and these experiments are useful evidence and should be leaned into more, especially over the unified task formulation aspect. There*s a lot more that could be done here, e.g. flushing out the experiments testing the various tasks under different amounts of training data or deeper analysis into why certain task pairs are helpful/harmful (genre? both are semantic-y or syntactic? lots of word overlap between the datasets? etc.). Things to improve + Questions - The paper would strongly benefit from a more in-depth explanation of the covered tasks, perhaps in an appendix. The explanations on p3 feel redundant. The examples in Tables 2a,b do a lot of work, but some of the tasks and labels still could use more explanation. - I didn*t really understand the phrase *all pairs of remaining spans*. Are those the top-K / lowest NEG_SPAN probability spans? - What do and index ? - The conclusions from Tables 5 + 6 seem to contradict one another. Table 5, as you say, seems to suggest that *as the contextualized representations become stronger, the performance of MTL+FT becomes more favorable*. On the other hand, Table 6 says that there seem to be many more mutually beneficial task pairs when using GloVe than when using BERT. Any hypotheses as to why this is ? - Figure 2 is hard to read because of the size and the legend formatting. [1] WHAT DO YOU LEARN FROM CONTEXT? PROBING FOR SENTENCE STRUCTURE IN CONTEXTUALIZED WORD REPRESENTATIONS. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, Ellie Pavlick. [2] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova