Structured Prediction as Translation between Augmented Natural Languages
Authors: Giovanni Paolini
Ben Athiwaratkun
Jason Krone
Jie Ma
Alessandro Achille
RISHITA ANUBHAI
Cicero Nogueira dos Santos
Bing Xiang
Stefano Soatto
Venue: ICLR 2021 Spotlight (2021)
Keywords: language models
few-shot learning
transfer learning
structured prediction
generative modeling
sequence to sequence
multi-task learning

Abstract: We propose a new framework, Translation between Augmented Natural Languages (TANL), to solve many structured prediction language tasks including joint entity and relation extraction, nested named entity recognition, relation classification, semantic role labeling, event extraction, coreference resolution, and dialogue state tracking. Instead of tackling the problem by training task-specific discriminative classifiers, we frame it as a translation task between augmented natural languages, from which the task-relevant information can be easily extracted. Our approach can match or outperform task-specific models on all tasks, and in particular achieves new state-of-the-art results on joint entity and relation extraction (CoNLL04, ADE, NYT, and ACE2005 datasets), relation classification (FewRel and TACRED), and semantic role labeling (CoNLL-2005 and CoNLL-2012). We accomplish this while using the same architecture and hyperparameters for all tasks, and even when training a single model to solve all tasks at the same time (multi-task learning). Finally, we show that our framework can also significantly improve the performance in a low-resource regime, thanks to better use of label semantics.

Published as a conference paper at ICLR 2021
STRUCTURED PREDICTION AS TRANSLATION BETWEEN AUGMENTED NATURAL LANGUAGES
Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero Nogueira dos Santos, Bing Xiang, Stefano Soatto
Amazon Web Services
(paoling, benathi, hi, kronej, jieman, aachille, ranubhai, cinog, bxiang, soattos| } @ama zon. com
ABSTRACT
We propose a new framework, Translation between Augmented Natural Lan- guages (TANL), to solve many structured prediction language tasks including joint entity and relation extraction, nested named entity recognition, relation clas- sification, semantic role labeling, event extraction, coreference resolution, and dialogue state tracking. Instead tackling the problem by training task-specific discriminative classifiers, we frame it as translation task between augmented
natural languages, from which the task-relevant information can be easily ex- tracted. Our approach match outperform task-specifie models on all tasks, and in particular, achieves new state-of-the results on joint entity and relation extraction (CoNLL04, ADE, NYT, and ACE2005 datasets), relation classification (FewRel and TACRED), and semantic role labeling (CoNLL-2005 and CoNLL- 2012). We accomplish this while using the same architecture and hyperparameters for all tasks and even when training a single model to solve all tasks at the same time (multi-task learning). Finally, we show that our framework can also signifi- cantly improve the performance in low-resouree regime, thanks to better use of label semantics.
INTRODUCTION
Structured prediction refers to inference tasks where the output space consists of structured objects, for instance graphs representing entities and relations between them. In the context of natural lan- guage processing (NLP), structured prediction covers a wide range of problems such as entity and relation extraction, semantic role labeling, and coreference resolution. For example, given the input sentence "Tolkien's 's epic novel The Lord of the Rings was published in 1954-1955, years after the book was completed" might seek to extract the following graphs (respectively in joint entity and relation extraction, and a coreference resolution task):
Toklen
The Lord of the Ring
Takken person
author
The Lord of the Rig book
rafora
book
Most approaches handle structured prediction by employing task-specific discriminators for the var- ious types relations or attributes, on top of pretrained transformer encoders such as BERT (Devlin) et al. 2019). Yet, this presents two limitations. First, discriminative classifier cannot easily lever- age latent knowledge that the pretrained model may already have about the meaning (semantics) of task labels such as person and author. For instance, knowing that a person can write book would greatly simplify learning the author relation in the example above. However, discriminative mod- els are usually trained without knowledge of the label semantics (their targets are class numbers), thus preventing such positive transfer. Second, since the architecture of a discriminative model is adapted the specific task. it is difficult to train single model to solve many tasks, to fine-tune a model from a task to another (transfer learning) without changing the tpsk-specific components of the discriminator. Hence, our main question is: we design framework to solve different

---

Published as a conference paper at ICLR 2021
Augmented Natural Language Translation
Sruucure Extraction
aneertt intearton intition
wee' op novel The Losd of the Hing min publds
113 penec Ia The Lend (dooh) oan wes anid yeure wfter
1964 1900, offer the book coregrntee
Hengel
TANL
Takien's opic nowel 1Ta Lostod Bioos (nsgeer
Eh book uoe come antedd
Thkkani oplc nowd Thes of was dbishhd 160 after book wis
publo 22t 1225, years after baad of the finge wa 001
Koor
lork
Figure 1: Our TANI. model translates between input and output text in augmented natural language, and the output then decoded into structured objects.
structured prediction tasks with the same architecture, while leveraging any latent knowledge that the pre-trained model may have about the label semantics?
In this paper, we propose to solve this problem with text-to-text model, by framing it as task of Translation between Augmented Natural Languages (TANL). Figure | shows how the previous example is handled within our framework, in the case of three different structured prediction tasks. The augmented languages are designed in way that makes easy to encode structured information (such relevant entities) in the input, and to decode the output text into structured information.
We show that out-of-the-box transformer models can easily learn this augmented language trans- lation task. In fact, successfully apply our framework to wide range of structured prediction problems, obtaining new state-of-the-art results on many datasets, and highly competitive results on all other datasets. We achieve this by using the same architecture and hyperparameters tasks, the only difference among tasks being the augmented natural language formats. This is in contrast with previous approaches that use task-specific discriminative models. The choice of the input and output format is crucial: by using annotations in a format that is as close as possible to nat- ural language, we allow transfer of latent knowledge that the pre-trained model has about the task, improving performance especially in low-data regime. Nested entities and an arbitrary number of relations are neatly handled by our models, while being typical sources of complications for previ- ous approaches. We implement an alignment algorithm to robustly match the structural information extracted from the output sentence with the corresponding tokens in the input sentence.
all
We also leverage our framework to train single model to solve all tasks at the same time, and show that it achieves comparable or better results with respect to training separately on each task. To the best of our knowledge, this is the first model to handle such variety of structured prediction tasks without any additional specific modules.
To summarize, our key contributions are the following.
1. We introduce TANL, a framework to solve several structure prediction tasks in a unified way, with a common architecture and without the need for task-specific modules. We cast structured prediction tasks as translation tasks, by designing augmented natural languages that allow us to encode structured information as part of the input or output. Robust alignment ensures that extracted structure is matched with the correct parts of the original sentence (Section
We apply our framework to (1) joint entity and relation extraction; (2) named entity recognition; (3) relation classification; (4) semantic role labeling; (5) coreference resolution; (6) event extrac- tion; (7) dialogue state tracking (Sections 4 and 5). In all cases we achieve least comparable results to the current state-of-theearr, and we achieve new state-of-the-art performance on joint entity and relation extraction (CoNI.L04, ADE, NYT, and ACE2005 datasets), relation classifi- cation (FewRel and TACRED), and semantic role labeling (CoNLL-2005 and CoNLL-2012).
3. We also train a single model simultaneously on all tasks (multi-task learning), obtaining compa- rable or better results as compared with single-task models (Section 5.1
4. We show that, thanks to the improved transfer of knowledge about label semantics, we can signif- icantly improve the performance in the few-shot regime over previous approaches (Section 5.2
2

---

Published as a conference paper at ICLR 2021
5. We show that, while model is purely generative (it outputs sentence, not class labels), it can be evaluated discriminatively by using the output token likelihood as proxy for the class score, resulting in more accurate predictions (Section 3 and Appendix A.3
The code is available at htttht::/gggttiii www.ttttww...mmmmm.aaa..o.. www. amazon weteeereeeeeaaaaeeeeeeeeee
2 RELATED WORK
Many classical methods for structured prediction (SP) in NLP are generalizations of traditional classification algorithms and include, among others, Conditional Random Fields (Lafferty et al.) 2001], Structured Perceptron (Collins) 2002). and Structured Support Vector Machines (Tosochan- taridis et al.) 2004). More recently, multiple efforts to integrate SP into deep learning methods have been proposed. Common approaches include placing an SP layer as the final layer of a neural net Collobert et al. 2011) and incorporating SP directly into DL. models (Dyer et al. 2013).
Current state-of-thhearr approaches for SP NLP train a task-specific classifier on top of the features learned by a pre-trained language model, such as BERT (Devlin et al. (2019). In this line of work, BERT MRC (Li al. 2019a) performs NER using two classification modules predict respectively the first and the last tokens corresponding to an entity for given input sentence. For joint entity and relation extraction, SpERT (Eberts & Ulges 2019) uses a similar approach to detect token spans corresponding to entities, followed by a relation classification module. In the case of coreference resolution, many approaches employ a higher-order coreference model (Lee et al. 2018) which learns probability distribution over all possible antecedent entity token spans.
Also related to this work are papers on ing and SP. Raffel al. (219)) describe a framework to cast problems such translation and summarization as text-to-text tasks in natural language, leveraging the transfer learning power of transformer-based language model. Other seuuence-to-eqqueeee
(seq2seq) models for multi- task learn-
approaches solve specific struc- tured prediction tasks by generating the desired output directly: see for example WDec (Nayak) & Ng 2020) for entity and relation extraction, and SimpleTOD (Hosseini-Asi
al. 2020) and
SOLOIST (Peng et al. 2020) for dialogue state tracking. Closer to us, GSL (Athiwaratkun et al., 2020 which introduced the term augmented natural language, showed early applications of the generative approach in sequence labeling tasks such as slot labeling. intent classification, and named entity recognition without nested entities. Our approach is also related to previous works that use seq2seq approaches to perform parsing (Vinyals et al., 2015 Dyer et al., 2016] Choe & Charniak] 2016 Rongali et al., 2020), with the main difference that we propose a general framework that uses augmented natural languages
way to unify multiple tasks and exploit label semantics. In some cases (e.g., relation classification), our output format resembles that of a question answering task (McCann et al. 2018). This paradigm has recently proved to be effective for some structured pre- diction tasks, such as entity and relation extraction and coreference resolution (Li et al. 2019cc Zhao ct al. 2020 Wu et al. 2020). Additional task-specific prior work is discussed in Appendix
Finally, TANL enables easy multi-task structured prediction (Section 5.1
Recent work has high-
lighted benefits of multi-task learning (Changpinyo al. 2018) and transfer learning (Vu et al., 2020) in NLP, especially in low-resource scenarios.
3 METHOD
We frame structured prediction tasks as text-to-text translation problems. Input and output follow specific augmented natural languages that are appropriate for a given task, as shown in Figure] [ In this section, we describe the format design concept and the decoding procedure we use for inference.
Augmented natural languages. We use the joint entity and relation extraction task as our guiding example for augmented natural language formats. Given a sentence, this task aims to extract a set of entities (one more consecutive tokens) and set of relations between pairs of entities. Each predicted entity and relation has to be assigned to an entity or a relation type. In all the datasets considered, the relations are asymmetric; i.e., it is important which entity comes first in the relation (the head entity) and which comes second (the tail entity). Below is the augmented natural language designed for this task (also shown in Figure
3

---

Published as a conference paper at ICLR 2021
Input: Tolkien's epic novel The Lord of the Rings was published in 1954-1955, years after the book completed.
Output:
Tolkien person J's epic novel The Lord of the Rings book I author Tolkien was published 1954- 1955, years after the book was completed.
Specifically, the desired output replicates the input sentence and augments it with patterns that can be decoded into structured objects. For this task, each group consisting of an entity and possibly some relations is enclosed by the special tokens [] A sequence of I-separated tags describes the entity type and a list of relations in the format "X = Y", where X is the relation type, and Y is another entity (the tail of the relation). Note that the objects of interest are all within the enclosed patterns l". However, we replicate all words in the input sentence, as it helps reduce ambiguity
"I "I... I when the sentence contains more than one occurrence of the same entity. also improves learning, as shown by our ablation studies (Section 5.3 and Appendix (B. In the target output sentence, entity and relation types are described in natural words (c.g. person, location) not abbreviations such as PER, LOC - to take full advantage of the latent knowledge that pre-trained model has about those words.
For certain tasks, additional information can be provided as part of the input, such as the span of relevant entities semantic role labeling or coreference resolution (see Figure T]. We detail the input/output formats for all structured prediction tasks in Section 4
Nested entities and multiple relations. Nested patterns allow us to represent hierarchies of enti- ties. In the following example from the ADE dataset, the entity "lithium toxicity" is of type disease, and has a sub-entity "lithium" of type drug. The entity "lithium toxicity" is involved in multiple relations: one of type effect with the entity "acyclovir", and another of type effect with the entity "lithium". In general, the relations in the output can occur in any order.
Input: Six days after starting acyclovir she exhibited signs lithium toxicity. Output: Six days after starting [ acyclovir drug she exhibited signs of lithium drug toxicity disease I effect = acyclovir I effect = lithium J.
Decoding structured objects. Once the model generates an output sentence in an augmented nat- ural language format, we decode the sentence to obtain the predicted structured objects, as follows.
1. We remove all special tokens and extract entity types and relations, to produce cleaned output. part of the generated sentence has an invalid format, that part is discarded.
2. We match the input sentence and the cleaned output sentence at the token levels using the dy- namic programming (DP) based Needleman- Needlemaan-uuschh alignment algorithm (Needleman & Wun- sch 1970). We then use this alignment to identify the tokens corresponding to entities in the original input sentence. This process improves the robustness against potentially imperfect gen- eration by the model, as shown by our ablation studies (Section 5.3 and Appendix BB).
3. For each relation proposed in the output, we search for the closest entity that exactly matches the predicted tail entity. If such an entity does not exist, the relation is discarded.
4. We discard entities or relations whose predicted type does not belong to the dataset-dependent list of types.
To better explain the DP alignment in step 2, consider the example below where the output contains misspelled entity word, "Aciclovir" (instead of "acyclovir"). The cleaned output containing the word "Aciclovir", tokenized as "A-cicl-o-vir", is matched to "a-cycl-o-vir" in the input, from which we deduce that refers "acyclovir".
Generated output: Six days after starting Aciclovir drug she exhibited signs of [ lithium drug toxicity disease effect= = Aciclovir effect = lithium J.
Cleaned output: Six days after starting Aciclovir she exhibited signs of lithium toxicity.
Multi-task learning. Our method naturally allows us to train single model on multiple datasets that can cover many structured prediction tasks. In this setting, we add the dataset name followed by the task separator: (for example, "ade :") prefix each input sentence.
Categorical prediction tasks. For tasks such as relation prediction, where there is limited number of valid outputs, alternative way to perform classification is compute class scores of all possible outputs and predict the class with the highest score. We demonstrate that we can use the output sequence likelihood as a proxy for such score. This method offers more robust way perform the
4

---

Published as a conference paper at ICLR 2021
evaluation in low resource scenarios where generation can be imperfect (see Appendix A.3 ) This approach is similar to the method proposed by dos Santos et al. (2020) for ranking with language models.
STRUCTURED PREDICTION TASKS
Joint entity and relation extraction. Format and details for this task are provided in Section 3
Named entity recognition (NER). This is an entity-only particular case of the previous task.
Relation classification. For this task, we are given an input sentence with head and tail entities and seek to classify the type of relation between them, choosing from a predefined set of relations. Since the head entity does not necessarily precede the tail entity in the imput sentence, we add phrase "The relationship between head] ] and [ tail iss after the original input sentence. The output repeats this phrase, followed by the relation type. In the following example, the head and tail entities are "Carmen Melis" and "soprano" which have a voice type relation.
Input: Bom in Bologna, Orlandi was student of the famous Italian soprano and voice teacher I Carmen Melis in Milan. The relationship between Carmen Melis and soprano is
Output: relationship between Carmen Melis and soprano = voice rype
Semantic role labeling (SRL). Here we are given an input sentence along with a predicate, and seek to predict a list of arguments and their types. Every argument corresponds to a span of tokens that correlates with the predicate in specific manner (e.g. subject, location, or time). The predicate marked the input, whereas arguments are marked the output and are assigned an argument type. In the following example, "sold" is the predicate of interest.
Input:
The luxury auto maker last year sold 1,214 cars in the U.S.
Output: The luxury auto maker subject ][ last year I temporal sold 1.214 cars obbjct ][ in the U.S. location
This task requires extracting (1) event triggers, each indicating the occurrence of a real-world event and (2) trigger arguments indicating the attributes associated with each trigger. In the following example, there two event triggers, "attacked" of type attack and "injured" of type injury. We perform trigger detection using the same format as in NER, as shown below. To perform argument extraction, we consider single trigger as input at time. We mark the trigger (with its type) in the input, and we use output format similar to joint entity and relation extraction. Below, we show an argument extraction example for the trigger "attacked", where two arguments need to be extracted, namely, "Two soldiers" of type targer and "yesterday" of type attack time.
Event extraction.
Trigger extraction input: Two soldiers were attacked and injured yesterday.
Trigger extraction output: Two soldiers were [ attacked attack and injured injury yesterday.
Argument extraction input: Two soldiers were attacked attack J and injured yesterday.
Argument extraction output: Two soldiers individual target attacked J were attacked and injured [ yesterday time attack rime attacked
Coreference resolution. This is the task of grouping individual text spans (mentions) referring to the same real-world entity, For each mention that is not the first occurrence of group, we reference with the first mention. In the following example, "his" refers to "Barack Obama" and is marked as his Barack Obama ] in the output.
Input: Barack Obama nominated Hillary Rodham Clinton as his secretary of state on Monday, chose her because she had foreign affairs experience as a former First Lady.
Output: Barack Obama J nominated [ Hillary Rodham Clinton as | his Barack Obama secretary of state Hillary Rodham Clinton on Monday. | He Barack Obama I chose her I Hillary Rodham Clinton because she Hillary Rodham Clinton had foreign affairs experience former I First Lady Hillary Rodham Clinton }.
Dialogue state tracking (DST). Here we are given as input history of dialogue turns, typically between a user (trying to accomplish a goal) and an agent (trying to help the user). The desired output is the dialogue state, consisting of value for each key (or slot name) from predefined list. In the input dialogue history, we add the prefixes "I user J and "[ agent ] :" to delineate user and agent turns, respectively. Our output format consists of list of all slot names with their predicted
5

---

Published as a conference paper at ICLR 2021
Relatian F1
Reletion
TAML SIERT
TANL (numenc labels) TANL (no DP alignment)
NYT)
-1
Percentage of training data
Percenlage trainig data
(a) Low-resource scenarios
(b) Ablation studies
Figure 2: Experiments on the CoNLL04 dataset. (a) Our model outperforms the previous state-of-the-art model SpERT, in low-resource scenarios. (b) Ablation studies where we
remove label semantics (numeric labels), augmented natural language format (abridged output) or dynamic programming alignment (no DP alignment), and plot the score differ- ence with the non-ablated TANL.
values. We add 'I belief I" delimiters to help the model know when to stop generating the output sequence. We tag slots that are not mentioned in the dialogue history with the value "not given" (we do not show them in the example below, for brevity).
Input: user J: am looking for cheap place to stay agent I How long? [ser ] Two Output: belief hotel price range cheap, hotel type hotel, duration two [elief
5 EXPERIMENTS
In this section, show that our TANL framework, with the augmented natural languages outlined in Section 4 can effectively solve the structured prediction tasks considered and exceeds the previous state of the art on multiple datasets.
All our experiments start from pre-trained T5- base model (Raffel et al. (019). To keep our frame- work as simple as possible, hyperparameters are the same across all experiments, except for some dataset-specific ones, such as the maximum sequence length. Details about the experimental setup, datasets, and baselines are described Appendix AA]
5.1 SINGLE-TASE AND MULTI-TASK EXPERIMENTS
We use three data settings in our experiments: (1) single dataset, (2) multiple datasets for the same task (multi-dataset), and (3) all datasets across all tasks (multi-task). Table shows the results
With the single-task setup, we achieve state-of-the-ar performance on the following datasets: ADE, NYT, and ACE2005 (joint entity and relation extraction), FewRel and TACRED (relation classifi- cation), CoNLL-2005 and CoNLL-2012 (semantic role labeling). For example, we obtain a +6.2 absolute improvement Fl score on the NYT dataset over the previous state of the art. Interest- ingly, this result is higher than the performance of models that use ground-truth entities to perform relation extraction, such REDN (Li & Tian (020), which achieves a relation score of 89.8. In coreference resolution, TANL performs similarly to previous approaches that employ BERT-bass model, except for CorefQA (Wu et al., (2000). To the best of our knowledge, ours is the first end- to-end approach to coreference resolution not requiring separate mention proposal module and not enforcing a maximum mention length.
For other datasets, obtain competitive performance within few points of the best baselines. We highlight that our approach uses a single model architecture that can be trained to perform any of the tasks without model modification. This is in stark contrast with typical discriminative models, which tend to be task-specific, as can be seen from Table [
In fact, under this unified framework, single model can be trained perform multiple all tasks at once, with the performance being on par or even better than the single-task setting. In particular, when the dataset sizes small such in ADE CoNLLO4, obtain sizable improvements and
We are grateful Wenxuan Zhou and Tianyu Gao for pointing out an inconsistency in computing results the TACRED dataset, which have been corrected in Table T

---

Published as a conference paper at ICLR 2021
Table Results on all tasks. All numbers indicate scores except noted otherwise.
Datasets marked with an asterisk (*) have nested entities.
CoNLL04
ADE*
NYT
ACE2005
Entity
Rel.
Entity
Rel.
Entity
Rel.
Entity
Rel.
Entity Relation Extr.
SpERT EEberts& Ulges
2019
88.9
71.5
89.3
78.8
DyGIE
Luan ct al. 2019)
88.4
63.2
MRC4ERE
(hha cl al 2020
88.9
71.9
85.5
62.1
RSAN
Yuan et al. 2020
84.6
TANL
89.4
71.4
90.2
80.6
94.9
90.8
88.9
63.7
TANL (multi-dataset)
89.8
72.6
90.0
80.0
94.7
90.5
88.2
62.5
TANL (multi-task)
90.3
70.0
91.2
94.7
90.7
CoNLL03
OntoNotes
GENIA
ACE2005
BERT-MRC Li al. 2019a
93.0
91.1
83.8
86.9
BERT-MRC+DSY Li 2019b)
93.3
92.1
Cloze-CNN (Baevski et al.
2019
93.5
NER
GSL (Athiwaratkun et al.
2020)
90.7
90.2
TANL
91.7
89.8
76.4
84,9
TANL (multi-datasett
92.0
89.8
75.9
84.4
TANL (multi-task)
91.7
89.4
76.4
FewRel 1.0 (validation)
5-way 1-shot
5-way 5-shot
10-way 1-shot
10-way
TACRED
5-shot
Relation Class.
BERT-EM (Soares al. 2019
70.1
88.9
82.8
BERTEM+MIH (Sares
2019
71.5
90.1
83.4
BG-SpanBERT Chen el al. 2020
71.5
BERT-PAIR (Gao et al. 2019
85.7
89.5
76.8
81.8
TANL
71.9
94.0 + 4.1
96.4 + 4.2
82.6 + 4.5
88.2 +: 5.9
TANL (multi-task)
69.1
CoNLL05 WSJ
CoNLLO5 Brown
CoNLL2012
Dep and Span (Li et al.
2019d)
86.3
76.4
83.1
BERT SRL (Shi & Lin 2019
88.8
82.0
86.5
SRL
TANL
89.3
82.0
87.7
TANL (multi-dataseet
89.4
84.3
87.6
TANL (multi-task)
89.1
84.1
87.7
ACE2005
Trigger Id.
Trigger CL.
Argument Id.
Argument CL.
Event Extr.
J3EE (Nguyen & Nguyen 2019
72.5
69.8
59.9
52.1
DyGlE++ Wadden 2019
69.7
55.4
52.5
TANL
72.9
68.4
50.1
47.6
TANL (multi-task)
71.8
68.5
48.5
48.5
CoNLL-2012 (BERT base BERT-large)
MUC
B
CEAF d4
Avg.
Coreference Res.
Higher-order c2f-coref (Lee al.
2018
80.4
70.8
67.6
73.0
SpanBERT (Joshi al. 2020)
85.3
78.1
BERT+c2f-cores (Joshi ct al. 2019
81.4
71.7
75.3
68.8
71.9
73.9
76.9
CorefQAASpanBEKK Wu ct al.
2020
86.3
88.0
77.6
82.2
75.8
79.1
79.9
83.1
TANI.
81.0
69.0
68.4
72.8
TANL (multi-task)
78.7
65.7
63.8
69.4
MultiWOZ 2.1 (Joint Accuracy)
TRADE Wu et al. 2019
45.6
DST
SimpleTODD (Hossem--As Asi et
2020
55.7
TANL
50.5
TANL (multi-task)
51.4

---

Published as a conference paper at ICLR 2021
become the new state of the (from 80.6 to 83.7 for ADE relation FI, and from 89.4 to 90.6 for CoNLL04 entity F1). The only case where our multi-task model has notably lower scores is coref- erence resolution, where the input documents maximum sequence length in the multi-task experiment (512 tokens) is smaller than the single- dataset coreference experiment (1,536 tokens for input and 2,048 for output), the input documents need to be split into smaller chunks, and this hurts the model's 's ability to connect multiple men- tions of the same entity across different chunks. From the multi-task experiment, we leave out all datasets based on ACE2005 except for event extraction due to overlap between train and test splits for different tasks. We discuss our experiments in more detail in Appendix(A]
much longer than in the other tasks. Since the
All results presented in this paper are obtained from pre-trained TS-base model. In principle, any pre-trained generative language model can be used, such as BART (Lewis et al.) 2020) or GPT-2 (Radford et al.) 2019). It would be interesting to check whether these models are as capable as T5 (or even better) learning to translate between our augmented languages. We leave this as a direction for future investigation.
5.2 LOW-RESOURCE SETTINGS
Multiple experiments suggest that TANL is data-efficient compared to other baselines. On the FewRel dataset, a benchmark for few-shot relation classification, our model outperforms the best baselines BERTEM EM and MET (Devlin et al., 2019; SSoarss et al.] 2019). where the MTB version uses a large entity-linked text corpus for pre-training. On the TACRED relation classifica- tion dataset, our model also improves upon the best baselines (from 71.5 to 71.9). While TACRED is not specifically a few-shot dataset, we observe that there are many label types that rarely ap- pear in the training set, some of them having less than 40 appearances out of approximately 70,000 training label instances. We show the occurrence statistics for all label types in the appendix (Ta- ble 3). demonstrating that the dataset is highly imbalanced. Nonetheless, we find that our model performs well, even on instances involving scarce label types. This ability distinguishes our models from other few-shot approaches such prototypical networks (Snell al. 2017) or matching net- works (Vinyals et al. 2016), which are designed only for few-shot scenarios but do not scale well on real-world data which often contains a mix of high and low-resource label types.
Our low-resource study on the joint entity and relation extraction task also confirms that our ap- proach is more data-cfficient compared to other methods. We experiment on the CoNLL04 dataset, using only 0.8% (9 sentences) to 6% (72 sentences) of the training data. Our approach outper- forms SpERT (a state-of-the-an discriminative model for joint entity and relation extraction) in this low-resource regime, whereas the performance is similar when using the full training set.
Thanks to the unified framework, we can easily train on a task, potentially with larger resources, and adapt other low-resource end tasks (transfer learning). To show this, we train model with large dataset from joint entity and relation extraction (NYT) and fine-tune it on a limited portion of the CoNLL04 dataset (Figure 2), obtaining significant increase in performance (up to +9 relation Fl).
Finally, in Appendix C we analyze how the size the training dataset affects the number of gener- ation errors of our model.
5.3 ABLATION STUDIES
We conduct ublation studies to demonstrate that label semantics, augmented natural language for- mat, and optimal alignment all contribute to the effectiveness of TANL (Figure 2b). Further details on these ablation studies can be found in Appendix
Numeric labels: To prevent the model from understanding the task through label semantics, we use numeric labels. This substantially hurts the performance, especially in low-resource setting where transfer learning is more important. Abridged output: Second, to determine the impact of the augmented natural language format outlined in Section 4 we experiment with format which does not repeat the entire input sentence. We find that this abridged format consistently hurts model performance, especially in low-resource scenarios. In other tasks, we generally find that more natural-looking format usually performs better (see Appendix AA.).. No DP alignment: We use exact word matching instead of the dynamic programming alignment described in Section 3
8

---

Published as a conference paper at ICLR 2021
6 DISCUSSION AND CONCLUSION
We have demonstrated that our unified text-to-text approach to structured prediction can handle all the considered tasks within simple framework and offers additional benefits in low-resource settings, Unlike discriminative models common in the literature, TANL is generative as translates from an input to output in augmented natural languages. These augmented languages are flexible and can be designed to handle a variety of tasks, some of which are complex and previously required sophisticated prediction modules. By streamlining all tasks to be compatible with single model, multi-task learning becomes seamless and yields state-of-the-art performance for many tasks.
Generative models, and in particular sequence- sequence-to-sequene to-seqeence models, have been used successfully in many NLP problems such as machine translation, text summarization, etc. These tasks involve mappings from one natural language input another natural language output. However, the use of sequence modeling for structured prediction has received little consideration. This is perhaps due to the perception that the generative approach is too unconstrained and that it would not be a robust way to generate precise output format that corresponds to structured objects, or that it may add an unnecessary layer of complexity with respect to discriminative models. We demonstrate that this quite the opposite. The generative approach can easily handle disparate tasks, even at the same time, by outputting specific structures appropriate for each task with little, if any, format error.
We note that one drawback of the current generative approach is that the time complexity for each token generation O(L²) where L is the sentence length. However, there have been recent advances in the attention mechanism that reduce the complexity to OoL L) as in Reformer (Kitaev et al., 2020), or to (L) as in Linformer (Wang et al. 2020). Incorporating these techniques in the future can significantly reduce computation time and allow us to tackle more complex tasks, well as improve on datasets with long input sequences such as in coreference resolution.
Based on our findings, believe that generative modeling is highly promising but has been an understudied topic in structured prediction. Our findings corroborate a recent trend where tasks typically treated with discriminative methods have been successfully solved using generative ap- proaches (Brown et al. 2020)] Izacard & Grave] 2020); [Schick & Schlitze 2020). We hope our results will foster further research in the generative direction.
REFERENCES
Ben Athiwaratkun, Nogueira Cicero dos Santos, Jason Krone, and Bing Xiang. Augmented natural language for generative sequence labeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 375-385, 2020. URLhttps:/ //op::///ww
aclweb. orq/
manlp-maing 27
Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. Cloze-driven pretraining of self-attention networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiao
jun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Lan-
guage Processing and the 9th International Joint Conference on Natural Language Process- ing, EMNLP-IJCNLP 2019, Hong Kong. China, November 3-7, 2019, pp. 5359 13399-368..1 Asso- doi: 1118655/11/11--1590 URL https:
ciation for Computational Linguistics, 2019.
7/doi. //doi.org/10. /ww./v///11..
Tom B. Brown, Benjamin Mann, Nick Ryder. Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Xavier Carreras and Lluís Mårquez. Introduction to the conll-2005 shared task: Semantic role labeling. In Ido Dagan and Daniel Gildea (eds.), Proceedings of the Ninth Conference on Compu- tational Natural Language Learning, CoNLL Ann Arbor, Michigan, USA, June 29-30, 2005, pp. 152-164. ACL, 2005, URL https:://///. / www. acl web. org/ htw.gtttooooooooo/ooooooootoottt /w00 WD0 06207
Soravit Changpinyo, Hexiang Hu, and Fei Sha. Multi-tast learning for sequence tagging: An em- pirical study. In Emily M. Bender, Leon Derczynski, and Pierre Isabelle (eds.), Proceedings of

---

Published as a conference paper at ICLR 2021
the 27th International Conference Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26. 2018, pp. 2965-2977. Association for Computational Linguistics, 2018. URLtttts:: www www.ww.ww.ccc.c org/ /ttp://ooooooyy1111111111111
Jun Chen, Robert Hoehndorf, Mohamed Elhoseiny, and Xiangliang Zhang. Efficient long- lont-distance relation extraction with dg-spanbert. CoRR, abs/2004.03636, 2020. URL https://arxiv.
org/abs/22 200aab///......... 03636
Do Kook Choe and Eugene Charniak. Parsing language modeling. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2331-2336, 2016. URL
https: www.w//wwwww.... aclweb. org wttpg//aaaooooggle-----111
Michael Collins. Discriminative training methods for hidden Markov models: Theory and experi- ments with perceptron algorithms. In Proceedings of the 2002 Conference on Empirical Methods
in Natural Language Processing (EMNLP 2002), pp. 1-8. Association for Computational Lin- guistics, July 2002. doi: 10.31151111693.111 181694..
URL https:/ /www www.wcclwebb...... org/
Wo2 1001
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. J. Mach. Learn. Res., 12(mull)24993- 2537, 2011. ISSN 1532-4435.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HIT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/v 1/13/////111//////////11 1423. URL 1ew.l11663vvv1
1423
Cicero Nogueira dos Santos, Xiaofei Ma. Ramesh Nallapati, Zhiheng Huang. and Bing Xiang. Beyond [CLS] through ranking by generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1722 1727, 2020. URL
https: /top:::/wwwww.. www. aclweb. wwt//nttologgg...o.ooooooooo 20tp:/2220...mm..2 main. 134
Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. Transition- based dependency parsing with stack long short-term memory. In Proceedings of the 53rd An- nual Meeting of the Asxociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 334-343, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.3115/v1/P15- 1033. URL ogy http:/aaahoooooooooolaaa11
https: www. ac web. o wwrgg//atttttooo
Chris Dyer, Adhigunaa Kuncoro, Miguel Ballesteros, and Noah A. Smith. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 199-209, 2016. URL
ht https: /tt:://www.... www.ww.wwwb............. org htt.g/ggnnttttg////ogoo NI 6- 1024
Markus Eberts and Adrian Ulges. Span-based joint entity and relation extraction with transformer
pre-training. CoRR, abs/1909.0777 07755, 2019. URL http: http:://waaa...ooo///////0000000 07755
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur MultiWOZ. 2.1: A consolidated multi- domain dialogue dataset with state corrections and state tracking baselines. In Proceedings of The 12th Language Resources and Evaluation Conference, pp. 422-428, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-4 URL https: www. aclweb. org. http://atccooooo////0..eeeeeeee.
Jenny Rose Finkel and Christopher D. Manning. Nested named entity recognition. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, 6-7 August 2009, Singapore, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 141-150. ACL, 2009. URL https: www. web. www.borr/eeee.. httpg//nthhlllgggoo
10

---

Published as a conference paper at ICLR 2021
Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Fewrel 2.0: Towards more challenging few-shot relation classification. In Kentaro Inui, Jing Jiang. Vin- Empirical Methods
cent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing, EMNN.P-IJCNIPP 2019, Hong Kong, China, November 3-7, 2019, pp. 6249- 6254. Association for Computational Linguistics, 2019. doi: 10.18653/11/119-166999 URL
attps://doi.org/111 1e..0.88653/vv/ wtt.://///...///////11. 1649
Pankaj Gupta, Hinrich Schütze, and Bernt Andrassy. Table filling multi-task recurrent neural net- work for joint entity and relation extraction. In Nicoletta Calzolari, Yuji Matsumoto, and Rashmi Prasad (eds.), COLING 2016, 26th International Conference on Computational Linguistics, Pro- ceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan, pp. 2537- 2547. ACL, 2016. URLpttpss:
www. www.ww.wwwwb.o.. org
Harsha Gurulingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmann- Apitius, and Luca Toldo. Development of benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports. J. Biomed. Informatics,
892,
2012. doi: 10.1016/j.jbr wttp:b.ii......000 URL https: /ttp:://doo..e http://i..ogg/111100001 1tlpg//....1111..11 jbi 2012. 04. 008
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Ellen Riloff, David Chiang, Julia Hockenmaier, and incic Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4. 2018, pp. 4809. Association for Computational Linguistics, 2018. doi: 10.18653/v 1/53/v1/d188 8- 1514. UtLhttps:: //ottps:///oi... wor./orggg. http:///oo..rrr00.1111333/11111/
Ehsan Hosseini- Asl,, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. A simple language model for task oriented dialogue. CoRR, abs/2005.00796, 2020. URL https://
arxiv. org.orgg/aao abs 2005. 00796
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering, 2020.
Heng Ji and Ralph Grishman. Refining event extraction through cross-document inference. In Proceedings of ACL-08: HLT, pp. 254 262, Columbus, Ohio, June 2008. Association for Com- putational Linguisties. URL https: www. www.ww.wwwb....... org/ http//attoooly///-10000000
Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. BERT for coreference resolution:
Baselines and analysis. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joini Conference Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 5802-5807, Association for Computational Linguistics, 2019. doi: 10.18653/v 1/D13/ 15888 TRRLhttps:
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Trans. Assoc. Comput. Linguistics, 8:64-77, 2020. URL http:://rraaaccllorg//////
php/tacl/
atticle/view/155
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. h
http:
wwarxii.orggaaaal. /11aa /1222 6980
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. (RR.https:/// httttp::///opnrevieee net. http:ew...eeetteeeeeeeeeee forum? vK
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth
11

---

Published as a conference paper at ICLR 2021
International Conference Machine Learning, ICML '01, pp. 282-289, San Francisco, CA, USA, Morgan Kaufmann Publishers Inc. ISBN 1558607781.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. End-to-end neural coreference res- olution. Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9- I, 2017, pp. 188-197. Association for Computational I Linguisties, 2017. doi: 10.18653/1//11-1100 / 1ttp://0000111331111111---111111
Kenton Lee, Luheng He, and Luke Zettlemoyer. Higher-order coreference resolution with coarse- to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 2 (Short Papers), pp. 687-692, New Orleans, Louisiana, June 2018. Association for Computational Lin- guisties. doi: 10.18653////11--20088. URL https: /www///wwwww. wwwwwb.wwwbbc... http://wwwwwwe...//aatt......... N18 2108
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-sequane pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 7871-7880. Association for Computational Linguistics, 2020. URL https:
7/ wwww. www.aaclweb.. org/ http///tttooogggooooeeem
ma main.. 703/
Cheng Li and Ye Tian. Downstream model design of pre-trained language model for relation extrac- tion task. CoRR, abs/2004.03788 2020.
abtpgr//bb//////// 03786
Qi Li, Heng Ji, and Liang Huang. Joint event extraction via structured prediction with global fea- tures. In Proceedings of 5 Annual Meeting the Association for Computational Linguisties (Volume 1: Long Papers), pp. 73 Soña, Bulgaria, August 2013. Association for Computa- tional Linguistics. URL httpstts:/// www. awwweb. org/ atthhooyooooooogooooo 1008
Xiaoya Li, Jingrong Feng. Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. A unified MRC framework for named entity recognition. CoRR, abs/1910.11476 2019a. URL http:
// wtw.://.wrr/aaa//11.111...
Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li, Dice loss for data- imbalanced NLP tasks. CoRR, abs/1911.02855 2019b. URL
.rgg abs//
1911. 02855
Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. Entity-relation extraction as multi-turn question answering. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.). Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 1340- 1350. Association for Computational Linguistics, 2019c. doi: 10.18653/v1/ p19- 1129. URL htttts:://// http///oooog/././/////o////// http:/11..165//11 11w133/311311133 105/
Zuchao Li, Shexia He, Hai Zhao, Yiqing Zhang, Zhuosheng Zhang, Xi Zhou, and Xiang Zhou. Dependency or span, end-to-end uniform semantic role labeling. In The Thirty- Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, Junuary 27 Febru- ary 1. 2019, pp. 6730 6737. AAAI Press, 2019d. doi: 10. 10.1609/aaai. 10.1699aaii.3330.33300666 333001.300166730 URL /ttps://do1. www.orgg/// / 1t////111166111 100916699/200 aaaiiaai.....
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hanmanch Hajishirzi. A gen- eral framework for information extraction using dynamic span graphs In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies,
12

---

Published as a conference paper at ICLR 2021
NATCL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 9036-3046, Association for Computational Linguistics. 2019. doi: 10.18653/v1/n19- 1308.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language 2018. URLhttp:
decathlon: Multitask learning as question answering. CoRR,
//arxiv.org abs/ 1806.00000000 08.088333 730
Tapas Nayak and Hwee Tou Ng. Effective modeling of encoder- decoder architecture for joint entity and relation extraction. In The Ahirty-Fourthh AAAI Conference on Artificial Intelligence, AAAI
2020, The Thirty-Second Innovalive Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York NY, USA, February 7-12, 2020, pp. 8528-8535. AAAI Press, 2020. URL https://
aaai. .
/idexx http:/AAI/atticccccc
Saul B Needleman and Christian Wunsch. A general method applicable to the search for similar- ities in the amino acid sequence of two proteins. Journal of molecular biology, 48(3):443-455 1970.
Trung Minh Nguyen and Thien Huu Nguyen. One for all: Neural joint modeling of entities and events. In The Thirty-This Third AAAI Conference on Artificial Intelligence. AAAI 2019, The Thirty- First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances Artificial Intelligence, EAAI 2019, Honolulu, Hawaii USA, January 27- February 1, 2019, pp. 6858. AAAI Press, 2019. doi: 10.1609/aaai. URL https: / /ooi wrg.10.gggoooooooo.o 609 aaai v33101 33016851
Tomoko Ohta, Yuka Tateisi, Jin-Dong Kim, Hideki Mima, and Junichi Tsujii. The genia corpus: An annotated research abstract corpus in molecular biology domain. In Proceedings of the second international conference on Human Language Technology Research, pp. 82-86, 2002.
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. SOLOIST: few-shot aak-oriented dialog with single pre-trained ruto- model. CORR, abs/2005.05298, 2020. URL htthhtss:/// http::///xxvvvve.e wtt.:orggaab.a/..../////
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. Conll- 2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Sameer Pradhan, Alessandro Moschitti, and Nianwen Xue (eds.). Joint Conference on Empirical Meth- ods in Natural Language Processing and Computational Natural Language Learning Pro- ceedings of the Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes, 2012, July 13, 2012, Jeju Island, Korea, pp. 1-40. ACL, 2012. httpss::: //www. awwweb. org/ ottp://atttooooy//125555111
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. Towards robust linguistic analysis using ontonotes. In Julia Hockenmaier and Sebastian Riedel (eds.), Proceedings of the Seventeenth Conference on Computational Natural Language Learning, CoNLL 2013, Sofia, Bulgaria, August 8-9, 2013, pp. 143-152. ACL, 2013. URL hwtp//ww.wwc ac www.aaclee..rr..o org/ httpa//www.o.o//aaaoooooooooaaa/
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/ 106633,,,, 2019, URL /ttp://axvvvvvv www.aawwwww..aaaaaaaaaaaaa wttg/abs...............bb.bea/ abs / 9910.1. 10683
Sebastian Riedel, Limin Yao, and Andrew McCallum. Modeling relations and their mentions without labeled text. In José L. Balcázar, Francesco Bonchi, Aristides Gionis, and Michèle Sebag (eds.), Machine Learning and Knowledge Discovery in Databases, European Confer- ence, ECML PKDD 2010, Barcelona, Spain, September 20-24, 2010, Proceedings, Part III, vol- ume 6323 of Lecture Notes in Computer Science, pp. 148 163. Springer, 2010. doi: 10. URL https: /ooi org 10. 10070.0..000000
978-3-642- 15939-81 1
642-
10
13

---

Published as a conference paper at ICLR 2021
Subendhu Rongali, Luca Soldaini, Emilio Monti, and Wael Hamza. Don't parse, generate! A sequence to sequence architecture for trie-triented semantic parsing. Proceedings of The Web Conference 2020, 2020. URL (RLLhht:://// /tttp://dx... doi org 0145/ 0000855336633..8 3380066 3380064
Dan Roth and Wen-tau Yih. A linear programming formulation for global inference in natural language tasks. In Hwee Tou Ng and Ellen Riloff (eds.), Proceedings of the Eighth Conference on Computational Natural Language Learning, CoNLL 2004, Held in cooperation with HLT. NAACL 2004, Boston, Massachusetls, USA, May 6-7, 2004, pp. 1-8. ACL, 2004. URL https: 7/ 7/www. ac www.wb.w.bbb..........
org/ antho wogy/ WO04 2401/
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Walter Daelemans and Miles Osborne (eds.), Proceedings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with HLLTNAACL 2003, Edmonton, Canada, May 31 June 1, 2003, pp. 142-147. ACL, 2003. URL hwwwwwww.. ac weeelwebbco otg/ aatg/ante antho ogy / WO3 0419/
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. CoRR, abs/2009.07118, 2020. URL httph:tttt//att httt:a:/aaxx.v.orr//b////0000
07118
Peng Shi and Jimmy Lin. Simple BERT models for relation extraction and semantic role labeling. CoRR, abs/1 has/1904.05255. 2019. URL
05255
Takashi Shibuya and Eduard H. Hovy. Nested named entity recognition via second best sequence learning and decoding. CoRR, abs/1909.02250, 2019. URL http:://axxv..rrg////
1909. 02250
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. CoRR, abs/1 703.05175, 2017.
htttt::://a.xll.orr//abbb 05175
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. Matching the blanks: Distributional similarity for relation learning. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2. 2019, Volume 1: Long Papers, pp. 2895-2905. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-. 1279. URL https://dois org. wttp://///..../////////////////. 1279
Jana Straková, Milan Straka, and Jan Hajic. Neural architectures for nested NER through lineariza- tion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguis- tics, pp. 5326-5333 Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.18653/v1/P1 1527. URL https: www. wttp://wwwwo...ttootoooeoeoeeeoo 1
Ioannis Tsochantaridis, Thomas Hofmann, Thorsten Joachims, and Yasemin Altun. Support vec- tor machine learning for interdependent and structured output spaces. In Proceedings of the Twenty-First International Conference Machine Learning, ICML '04, pp. 104, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581 138385. doi: 10. 10.1145/1015300. 1015341. URLhttts:://dot org 1tt.///1111111111333333 1015341
Oriol Vinyals, F. ukasz Kaiser, Terry Koo, Slav Petrov, llya Sutskever, and Geoffrey Hinton. Gram- mar as a foreign language. C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28, pp. 2773-2781. Cur- ran Associates, Inc., 2015, URL papp:aaeaacccaaeeeeapppeeeeee
www.inngg....rss.... cc/ http:cccpppppp/....../////
fi
Paper.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 3630-3638. Curran Associates, Inc., 2016. URL
nips. cc.pacez//ppr//
6385
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Matkerella-Mick,, Subhransu Maji. and Mohit lyyer. Exploring and predicting transferability across NLP tasks. CoRR, abs/2005.00770, 2020. URL /Rtta:/tt: //a.vvv haatpp::/arxivvoorg///////// wrgp/bs/200... 00770
14

---

Published as a conference paper at ICLR 2021
David Wadden, Ulme Wennberg, Yi Luan, and Hannanch Hajishirzi. Entity, relation, and event extraction with contextualized span representations. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing, EMNIP-IJCCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 3783-5788, As- sociation for Computational Linguistics. 2019. doi: 10.18653/1/D19-15888 URL https:
7/doi wtt.:oogg/.....///... 1ttp//00.1653//1//D/0 1585
Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. Ace 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia, 57:45, 2006.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. CoRR, abs/2006.04768, 2020. URL https:://rxiv..ogg 2006 04768
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's s transformers: ofate-of-the-ar natural language processing. CoRR, abs/1910.03771, 2019. URL
http::/arxit: org.orggaae abs /
191 910....0 03771
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pas- cale Fung. Transferable multi-domain state generator for task-oriented dialogue systems. In Pro- ceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2019.
Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. Corefqa: Coreference resolution as query- based span prediction. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL Association for Computational Linguisties, 2020.
2020, Online, July 5-10, 2020, pp. http:///thoolll//200000 acl -ma- i 6227
URL https: www ac web www.worr..tthooooooo/ooo
ant
Bowen Yu, Zhenyu Zhang. Jianlin Su, Yubin Wang, Tingwen Liu, Bin Wang, and Sujian Li. Joint ex- traction of entities and relations based on a novel decomposition strategy. CoRR, abs/1909.04273, 2019. URL http//aaxvvvv//a wwwpwiii.wvr..ooooooo..o
/ttgg//b////// abs 01119999992.2.
Yue Yuan, Xiaofei Zhou, Shirui Pan, Qiannan Zhu, Zeliang Song, and Li Guo. A relation-specific attention network for joint entity and relation extraction. In Christian Bessiere (ed.). Proceedings of the Ninhty-Nitth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 4054-4066 ijcai.org. 2020. doi: 10.24963/jcci.2220556.. URL 24963/1 jeai 2020/561
org///.....
Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao. Extracting relational facts by end-to-end neural model with copy mechanism. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguis- fies, ACL 2018, Melbourne, Australia, July 15-20. 2018, Volume I: Long Papers, pp. 506- 514. Association for Computational Linguistics, 2018. doi: 10.18653W11/P18-1044 URL https: http://ww..cllll... org anthohoohyy....... pttp://11111177771111
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. Position- aware attention and supervised data improve slot filling. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.). Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pp. 35 45. Association for Computational Linguistics, 2017. doi: 10. 10.11653/11//11-10044 URL https:
7/doi.org/10.
Tianyang Zhao, Zhao Yan, Yunbo Cao, and Zhoujun Li. Asking effective and diverse questions: A machine reading comprehension based framework for joint entity-relation extraction. In Chris- tian Bessiere (ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 3948-39544 orga 2020. doi: 2/t.9663//caii.22055446 URL https://6bi.org/10...
15

---

Published as a conference paper at ICLR 2021
A EXPERIMENTAL SETUP, DATASETS, AND BASELINES
In all experiments, we fine-tune a pre-trained T5-base model (Rafeel al. 2019), to exploit prior knowledge of the natural language. The family of TS models was specially designed for downstream text-to-text tasks, making them suitable for our needs. The T5-base model has about 220 million parameters. For comparison, both encoder and decoder are similar in size to BERT-base (Devlin ct al. 2019). We use the implementation of HuggingFace's Transformers library (Wolf al.) 2019).
To keep our framework as simple as possible, hyperparameters are the same across the majority of our experiments. We use: 8 V100 GPUs with a batch size of per GPU; the AdamW optimizer (Kingma & Ba 2015] Loshchilov & Hutter, 2019); linear learning rate decay starting from 0.0005; maximum input/output sequence length equal to 256 tokens training time (longer sequences are truncated), except for relation classification, coreference resolution, and dialogue state tracking (see below). The number of fine-tuning epochs adjusted depending on the size of the dataset, as described later. With these settings, one fine-tuning step takes approximately 0.8 seconds. This translates into 15 seconds per epoch for the (relatively small) CoNLLO4 dataset (joint entity-relation extraction) and 16 minutes per epoch for the (much larger) OntoNotes dataset (NER). At inference time, we employ beam search with 8 beams, and we adjust the maximum sequence length depending on the length of the sentences in each dataset. Note that beam search is not an essential part of our framework, as we find that greedy decoding gives almost identical results.
In the rest of this section, we describe datasets and baselines for each structured prediction task, as well as additional insights on particular experiments. Results of all experiments are given in Table Unless otherwise specified, micro-Fl scores are reported. Most experiments are run more than once, as described below, and the average result is reported. Table 5 shows input-output examples from different datasets.
For the multi-task experiment, we train for 50 epochs on 80 GPUs, with a batch size of 3 per GPU. The maximum input/output sequence length is set to 512 for all tasks.
A.1 JOINT ENTITY-RELATION EXTRACTION
Datasets.
We experiment the following datasets: CONLL04 (Roth & Yih 2004), ADE (Gu-
rulingappa et al. 2012), NYT (Riedel et al. 2010), and ACE2005 Walker et al. 2006).
The CoNLL04 dataset consists of sentences extracted from news articles, with four en- tity types (location, organization, person, other) and five relation types (work for, kill, organization based in, live in, located in). As in previous work, we use the training (922 sentences), validation (231 sentences), and test (288 sentences) split by Gupta et al. (016)) We train for 200 epochs and report our test results averaged over 10 runs.
The ADE dataset consists of 4, 272 sentences extracted from medical reports, with two entity types (drug. disease) and a single relation type (effect). This dataset has sentences with nested entities. As in previous work, we conduct 10-fold cross- vooss-valiattion and report the average macro-F1 results across all 10 splits (except for the multi-task experiment, which is carried out once and uses the first split of the ADE dataset). We train for 200 epochs.
The NYT dataset (Zeng al. 2018) based on the New York Times corpus and was auto- matically labeled with distant supervision by Riedel et al. (2010). We use the preprocessed version of Yu al. (2019). This dataset has three entity types (location, organization, person) and 24 relation types (such as place of birth, nationality, company). It consists of 56,195 sentences for training. 5,000 for validation, and 5,000 for testing. We train for 50 epochs and report our test results averaged over 5 runs.
The ACE2005 dataset is derived from the ACE2005 corpus (Walker et al. 2006) and con- sists of sentences from variety of domains, including news and online forums. We use the processing code Luan et al. (2019). After filtering out the sentences without entities, we get 7.477 sentences for training, 1789 for validation, and 1517 for testing. It has seven entity types (location, organization, person, vehicle, geographical entity, weapon, facil- ity) and six relation types (PHYS, ART, ORG-AFF, GEN-AFF, PER-SOC, PART-
16

---

Published as a conference paper at ICLR 2021
Table 2: Details about the single-dataset experiments in joint entity-relation extraction and named entity recognition.
Dataset
# Epochs
# Runs
Results
Joint entity-relation extraction
Entity F1
Relation F1
CONILO04
200
10
89.4 + 0.3
71,4 + 1.1
ADE
200
10
90.2 + 0.7
80.6 + 1.5
NYT
5
94.9 = 0.1
90.8 + 0.1
ACE2005
100
10
88.9 + 0.1
63.7 + 0.7
Named entity recognition
Entity F1
CoNLLO3
91.7 0.1
OntoNotes
20
10
89.8 + 0.1
GENIA
50
10
76.4 - 0.4
ACE2005
50
10
84.9 0.2
Relation F1
TANI
TAML (wth
TANL (MNL NYT) NYT)
2%
Percentage of training data
Percentage of training data
Figure 3: Low-resource experiments on the CoNLLO4 dataset.
The natural labels we use for the relation types are: physical, artifact, employer, affiliation, social, part of. We train for 100 epochs and report our test results averaged over 10 runs.
For all single-dataset experiments, T shows the number of training epochs, the number of runs, and the standard deviations, in addition to the average results, which are already reported in Table
Baselines. SpERT (Eberts Ulges 2019 BERT-based model which performs span classi- fication and then relation classification. Multi-turn QA (Li et al.) 2019c) casts the problem as multi-turn question answering task. ETL-Span (Yu et al. 2019) uses BiLSTM and decomposes the problem into two tagging sub-problems: head entity extraction, and tail entity and relation extrac- tion. WDec (Nayak & Ng 2020) uses an encoder-decoder architecture to directly generate a list of relation tuples. MRC4ERE (Zhao et al. (0200) improves the question answering approach by leveraging a diverse set of questions. RSAN Yuuan et al. 2020) is sequence labeling approach which utilizes relation-aware attention mechanism.
Low-resource experiments. As outlined in Section 5.2 we experiment on the CoNLL04 dataset with only a limited portion of the training set available and plot our results in Figure Comparison is made with SpERT (Eberts & Ulges 2019), a state-of thee-art discriminative model. TANL performs better than SpERT with fewer data, especially on the more complex task of relation extraction (right plot). We also show our method's 's performance with preliminary tieettunng on the NYT dataset for one epoch, which significantly improves the performance on both entity and relation extraction. To account for the small dataset size, we fine-tune CoNLLO4 for 2,000 epochs (10x the number of epochs we use to train on the full CoNLLO04 dataset). For fair comparison, we train SpERT for 20, 200, and 2000 epochs (respectively 1 10x, and 100x the number of epochs suggested in the paper), and report the best result among the three, which always obtained with 200 epochs. We plot mean and standard deviation over 10 runs (each model being fine-tune tunet the same 10 subsets of the training set and evaluated on the entire test set). For reference, the smallest training set has only 9 sentences (0.8% of the total), effectively consisting in few-shot learning scenario.
Multi-dataset experiments. We train a single model on all four datasets for 20 epochs and report the average over 10 runs. We different split of the ADE dataset in each run.
17

---

Published as a conference paper at ICLR 2021
A.2 NAMED ENTITY RECOGNITION
Datasets. We experiment on two flat NER datasets, CoNLL03 Sang & Meulder 2003 and OntoNotes (Pradhan al. 2013), and two nested NER datasets, GENTA (Ohta et al. 2002 and ACE2005 (Walker et al. 2006).
For the CoNLL.03 dataset (Sang & Meulder 2003 we use the same processing and splits Li et al. (2019a), resulting in 14,041 sentences for training, 3,250 for validation, and 3,453 for testing. This dataset has four entity types (location, organization, person, miscel- laneous). We train for 50 epochs and report our test results averaged over 10 runs.
The English OntoNotes dataset (Pradhan al., 2013) consists of 59,924 sentences for training, 8,528 for validation, and 8,262 for testing. It has 18 entity types (such as person, organization, date, percent). We train for 20 epochs and report our test results averaged over 10 runs.
The GENIA dataset (Ohta et al. 2002) consists of sentences from the molecular biology domain. As in previous work, resulting in 14,824 sentences for training, 1,855 for validation, and 1,854 for testing. There are five entity types (protein, DNA, RNA, cell line, cell type). We train for 50 epochs and report our test results averaged over 10 runs.
the processing and splits Finkel & Manning/ (2009)
The ACE2005 dataset for nested NER based on the ACE2005 corpus (Walker et al. 2006), but is different from the one used for joint entity-relation extraction. We use the same processing and splits of Li et al. (2019a), resulting in 7,299 sentences for training, 971 for validation, and 1,060 for testing. It has the same seven entity types as the ACE2005 dataset used for joint entity-relation extraction. We train for 50 epochs and report our test results averaged over 10 runs.
As for joint entity-relation extraction, Table 2 summarizes our setup and results (with standard de- viations) for the single-dataset experiments.
Baselines.
State-of -art--rrtth---tttte results on popular NER datasets are mostly detained by BERT-MRC LLi et al. 2019a) and BERT-MRC + DSC eLi al. 2019b), which formulate the problem as a machine
reading comprehension task, solved by asking multiple questions. ClozeCNN (Baevski et al. 2019) leverages a cloze-driven pre-training. Seq2seq-BERT (Straková et al. 2019) uses a seq model to output the list of entity types. Second-best learning and decoding (Shibuya & Hovy (019)) iteratively decodes nested entities starting from the outermost ones, using the Viterbi algorithm. For flat NER, our approach is similar to GSL (Athiwaratkun et al., 2020).
Multi-dataset experiments. We train single model on all four datasets for 10 epochs and report our results averaged over runs.
A.3 RELATION CLASSIFICATION
Datasets. We experiment on FewRel (Han al. 2018 and TACRED (hhang et al. 2017).
FewRel consists of 100 relations with instances for each relation. The standard evaluation for this benchmark uses few-shot N-way K-shot settings, which we follow. The entire dataset is split into train (64 relations), validation (16 relations) and test set (20 relations). We train our model on the meta-training set, which has no overlapping classes with the evaluation set. At evaluation time, given a support set and a query set on new task, we fine-tune the model on the support set to learn the new task and evaluate on the query set.
TACRED is large-scale relation classification dataset with 106,344 examples (68,164 for training. 22,671 for validation, and 15,509 for testing), covering 41 relation types. We train for epochs and report our test results averaged over runs. The maximum input sequence length is set to 300, whereas the maximum output sequence length is set to 64 during training and 128 during inference.
We compare our approach with the following two models in the literature. The first BERT-pair (Gao et al. 2019). a sequence classification model based on BERT, which learns to optimize the scores indicating the relation between query instance and other supporting instances
Baselines.
18

---

Published as a conference paper at ICLR 2021
for the same relation. The second is BERTEM EM Matching the Blanks (MTB) (Soares et al. 2019 BERTEN EM uses entity markers indicating the start and the end of the head and tail entities in the imput sentence. MTB is pre-training based additional large corpus of relation data. Nevertheless, our model is able to outperform BERTEM+MTB in certain cases, such as the 5-way 1-shot setting on FewRel.
Augmented natural language formats. We experiment with many augmented natural language formats, as shown below:
Input (chosen): Born in Bologna. Orlandi was a student of the famous Italian soprano J and voice teacher [ Carmen Mclis ] in Milan. The relationship between [
Carmen Melis ] and soprano ] is
Output (chosen): relationship between [ Carmen Melis J and soprano = voice type
Input (alternative 1): Born in Bologna, Orlandi was student of the famous Italian [ soprano and voice Carmen Mclis and soprano is
teacher I Carmen Melis in Milan. The relationship between [
Output (alternative 1): voice type
Input (alternative 2): Born in Bologna, Orlandi was a student of the famous Italian soprano tail and voice teacher Carmen Melis head in Milan.
Output (alternative 2): relationship hetween [ Carmen Melis] ] and [ soprano ]= voice rype
The alternative version has shorted output which only produces the keyword such as voice type corresponding to the predicted relation. However, we find that it does not perform as well as the chosen format. We hypothesize that it is due to the rich semanties of the sentence "relationship between [ Carmen Melis I and soprano ]", and possibly softer gradient information on the longer sequence which improves training.
The alternative 2
version annotates the head vs. tail information for the entities directly in the input, instead of using a phrase such as "relationship between Carmen Melis and soprano I" to specify that "Carmen Melis" is the head entity. However, this format also does not perform as well, possibly because the meaning of the words head and tail are not fully understood in this context. Overall, the chosen format sounds the most natural out of all options and is closer to natural language, which we use as our guiding principle to design our augmented natural language formats.
TACRED results and label sparsity. A major factor for our state-of-the-art result on the TACRED dataset the shared semantics across different labels, which particularly beneficial the case of sparse labels. In Table 3 we show the relation types in natural words, the number of training examples, which can be quite small, and the test recall (i.e., out of all ground truth relations for a given type, how many we predict correctly). We can that even though some relation types such as date of birth have as little as 64 labels in the training set (less than 0. 1% of the entire set), our model is able to correctly predict this relation type with recall 77.8%.
The ability to handle few-shot cases allows our approach to perform well in real-world data such as TACRED, where the labels can be highly imbalanced. As seen in Table 3 only a few instances such as employee of, lop members employees, title, and no relation dominate the majority of the training set (approximately 60,000 out of 68,000), where the rest can be considered scarce. Our model is different from other approaches specifically designed for few-shot scenarios in that it scales across different levels of data.
Few-shot experiments. For the FewRel dataset, we perform meta-training by training the model on the training set of FewRel for I epoch. During evaluation, we fine-tune the model on the support set for each episode for 2,500 epochs in the 1-shot cases, and for 500 epochs in the 5-shot cases.
Likelihood-baser prediction. In relation classification, we aim predict one class out of a pre-defined set of classes, so we can perform prediction by using sequence likelihoods as class scores. This helps improve the performance particularly in the case of few-shot scenarios, where the generation of label types can be imperfect since the model has seen only one or few instances of each type. With the likelihood prediction, we obtain a slight improvement across the board. For instance, we improve from an Fl score of 95.6 + 4.8 to 96.4 + 4.2 for the 5-way 5-shot case of FewRel. For TACRED, using the likelihood approach yields a smaller improvement, possibly due to the fact that the model can generate exact label types given enough training resources, unlike in the few-shot case. All our reported numbers on the FewRel and TACRED datasets are obtained by using this approach.
19

---

Published as a conference paper at ICLR 2021
Table 3: TACRED recall by relation type (in one of our runs), with number of training, validation, and test examples.
Relation type
# Train
Dev
# Test
Test recall
country of death dissolved
7
47
44.4
24
9
2
50.0
country of birth
29
21
0.0
state or province of birth
27
50.0
state or province of death
50
42
14
64.3
religion
54
54
47
48.9
date of birth
64
32
77.8
city of birth
66
34
20.0
charges
73
103
85.4
number of employees members
76
28
19
63.2
shareholders
77
56
13
0.0
city of death
82
119
28
32.1
founded
39
37
83.8
political religious affiliation
106
11
10
40.0
website
112
87
26
88.5
cause of death member of founded by
118
169
52
36.5
123
32
18
0.0
77
68
82.4
date death
135
207
54
55.6
schools attended
150
51
30
73.3
siblings
166
31
55
76.4
members
86
48.4
other family
180
81
60
46.7
children
212
100
37
67.6
state provinec of headquarters
230
71
51
80.4
spouse
259
160
66
69.7
subsidiaries
44
45.5
origin
326
211
132
62.1
state or provinces of residence
332
73
81
54.3
citics of residence
375
180
189
58.7
city of headquarters
82
70.7
age
200
96.0
parents
439
153
150
64.7
countries of residence country of headquarters
446
227
148
37.8
469
178
108
60.2
alternate names employee of top members employees title
224
89.3
1,525
376
264
70.1
1,891
535
346
84.4
2,444
920
500
86.8
no relation
55,113
17,196
12,184
93.8
20

---

Published as a conference paper at ICLR 2021
A.4 SEMANTIC ROLE LABELING
Datasets. We use CoNLL-200S Carreras & Märquez 2005) and the CoNLL-2012 English subset of OntoNotes 5.0 (Pradhan al. 2013) in our experiments. See also Carreras Märquez (2005); Pradhan et al. (2012). These tasks have highly specific label types, and their natural words might be cumbersome for training. Therefore, we use the raw label types from the original datasets as presented below.
CoNLL-2005 focuses on the semantic roles given verb predicates. The argument notation is the following. verb; A0: acceptor; Al: thing accepted: A2: accepted from; A3: attribute; AM-MOD: modal; AM-NEG: negation.
CoNLL-2012. The argument notation, taken from Pradhan et (2012), is follows.
Numbered arguments (A0-A5, AA): Arguments defining verb-specific roles. Their seman- ties depends on the verb and the verb usage sentence, verb sense. The most frequent roles are AO and AI. Commonly. AO stands for the agent, and Al corresponds to the pa- tient or theme of the proposition. However, no consistent generalization can made across different verbs or different senses of the same verb. PropBank takes the definition of verb senses from VerbNet, and for each verb and each sense defines the set of possible roles for that verb usage, called the roleset. The definition of rolesets is provided in the PropBank Frames files, made available for the shared task as an official resource to develop systems.
Adjuncts (AM-): General arguments that any verb may take optionally. The following are the 13 types of adjuncts. AM-ADV: general-purpose; AM-CAU: cause; AM-DIR:
direction; AM-DIS: discourse marker; AM-EXT: extent; AM-LOC: location; AM-MNR:
manner; AM-MOD: modal verb; AM-NEG: negation marker; AM-PNC: purpose; AM-
PRD: predication; AM-REC: reciprocal; AM-TMP: temporal.
References (R-): Arguments representing arguments realized in other parts of the sentence. The role of a reference is the same as the role of the referenced argument. The label is an R-tag prefixed to the label of the referent, c.g., R-AI.
Baselines.
We compare our results with Dependency and Span SRL Li et al., 2019d). which uses Bi-LSTM with highway connection and biaffine scorers, and BERT-SRL (Shi & Lin 2019). BERT-based model which predicts the spans based on the contextual and positional embeddings.
Multi-dataset experiments. We train single model on all datasets for 50 epochs and report our results averaged over 5 runs.
A.5 EVENT EXTRACTION
Datasets. We the ACE2005 English event data (Walker al. 2006) in experiments, fol- lowing standard event extraction literature. We use the same split as previous work (Ti & Grishman 2008 Li al., 2013) with 529 documents for training, 30 for validation, and 40 for testing. Since the majority of event triggers and their corresponding arguments are within the same sentence, we perform the event extraction task only at the sentence level. We fine-tune our model for 50 epochs on this dataset.
Baselines. We compare our method with the following two baseline models in the literature. The first is J3EE (Nguyen & Nguyen 2019). a Bi-GRU based model that jointly performs event trigger detection, event mention detection, and event argument classification. J3EE performs event trigger detection and event mention detection sequence tagging problems, and event argument classification as a classification problem, given any trigger and candidate argument pair. The second baseline is DyGIE++ (Wadden et al. 2019), BERT based multi-task learning framework for the tasks of coreference resolution. relation extraction, named entity recognition, and event extraction. DyGlE++ enumerates all possible phrases within sentence and predicts the best entity type and trigger type for each of these phrases. Argument roles are then predicted for each trigger and entity pair.
21

---

Published as a conference paper at ICLR 2021
A.6 COREFERENCE RESOLUTION
Datasets. We use the standard OntoNotes benchmark defined in the CoNLI-20012 shared task (Pradhan et al. 2012). It consists of 2,802 documents for training, 343 for validation, and 348 for testing, for total of about million words. Since documents can be large (up 4,000 words), we split each document into partially overlapping chunks up to 1,024 words long (and 128 or 196 words for the multi-task experiment, respectively for training and for testing). At test time, merge groups from different chunks if they have at least one mention in common in order to obtain document-level predictions. As in prior work, evaluation is done by computing the average Fl score of the three standard metrics for coreference resolution: MUC, B², CEAF dig- We train for 100 epochs, with maximum sequence length equal 1,536 tokens for input and 2,048 for output, and a batch size of per GPU.
Baselines. The e2e-coref model (Lee et al. 2017) is among the first end-to-end approaches to coreference resolution. It considers all spans as potential mentions and learns a distribution over possible antecedents for each span. Higher-order e2f-coref (Lee et al.] (018)) iteratively refines span representations taking into account higher-order relations between mentions. BERT c2f-coref (Joshi et al., 2019) combines the previous approach with BERT. SpanBERT (Joshi et al., (020)) introduces
pretraining method which is designed to better represent and predict spans of text. CorefQA (Wu et al.] (2020) generate queries for each mention from a mention proposal network and uses question answering framework to extract text spans of coreferences.
A.7 DIALOGUE STATE TRACKING
Datasets. We use the MultiWOZ 2.1 Eric et al. 2020 task oriented dialogue dataset in our experiments. It consists of 8,420 conversations for traning. 1,000 for validation, and 999 for testing. We follow the pre-processing procedure put forward in (Wu et al. 2019) for dialogue state tracking. In addition, we remove the "police" and "hospital" domains from the training set since they are not present in the test Removing these domains reduces the training set size from 8,420 to 7,904. We fine-tune for 100 epochs, with maximum sequence length set to 512 tokens. We train single generative model that predicts the dialogue state for the entire dialogue history up to the current turn. Following prior work, we report the joint accuracy.
Baselines. We compare our performance on MultiWOZ 2.1 against SimpleTOD (Hosseini-Asl et al. 2020), the current state of the art for MultWWOZ dialogue state tracking. SimpleTOD uses sequence sequence approach based on the GPT-2 (Radford et al., 2019) language model. Unlike our approach, SimpleTTO is trained to jointly generate actions and responses as well as dialogue states.
B ABLATION STUDIES
As outlined in Section 5.3 we conduct ablation studies on the CoNLL04 dataset (joint entity and relation extraction) demonstrate the importance of label semantics, natural output format, and optimal alignment. We compare TANL with the following three variations.
Numeric labels: we use numbers (1, 2, ...) to indicate entity and relation types in the output sentences, as in the following example.
Output: Boston University 2 J's Michael D. Papagiannis 3 Boston University said he believes crater was created 100 million years 14 ago when a 50-mile-wide meteorite slammed into the | Earth 1
Abridged output: here, the output consists of a list of entities, enclosed between [] tokens, without text between them.
Output: Boston University organization Michael Papagiannis person works for Boston University [[ 100 million years other ][ Earth location
No alignment: process output sentences without the alignment module. For each pre- dicted entity or relation, we look for the first exact match the input sentence (the entity relation is discarded if exact match is found).
22

---

Published as a conference paper at ICLR 2021
Table 4: Ablation studies on the CoNLLO4 dataset (using the full training set, and using only 50% of the training sentences). We report mean and standard deviation over 10 runs.
CoNLL04
CoNLL004 (50%)
Model
Entity F1
Relation F1
Entity F1
Relation F1
TANI.
89.44 + 0.30
71.44 + 1.15
87.15 + 1.08
68.30 + 1,47
TANL (numeric labels)
89,13 + 0.45
71.57 + 0.89
86.59 + 0.94
66.12 + 1.31
TANL (abridged output)
88,42 = 0.67
70.98 1.12
86.11 + 0.55
67.18 + 1.18
TANL (no alignment)
87.88 0.31
69.72
1.31
85.56 1.01
66.64 + 1.54
Relation
TANI (numeric labas) TANL outbut] agnment
TANI (numeric labels)
dgen DP alignment)
36%
1006
100%
Porcentage of training data
Porcantage of trainng data
Figure 4: Ablation studies on CoNLL04, using different portions of the training dataset.
The outcomes of these experiments are shown in Figures 2 and 4 and Table We run all exper- iments using a variable amount of training data, from 100% (1,153 sentences) down to 0.8% (9 sentences), and always evaluate the entire test set (288 sentences). To account for the variable size of the training dataset, we adjust the number of training epochs as follows: 200 epochs when using all training data; 400 epochs for 50% of the training data; 800 epochs for 25%; 1,600 epochs for 12.5%; 2,000 epochs for all remaining cases (6.3%, 3. %, 1.6%, 0.8%)
Results show that all three components (label semanties, natural output format, and alignment) posi- tively contribute to the effectiveness of TANL. The impact of label semantics is not noticeable when using the full CoNLL04 training dataset (natural and numeric labels give similar FI scores), but becomes statistically relevant when using 50% of the training data, or less. On the other hand, the impact of alignment is higher when the training dataset is larger. Interestingly, for entity extrac- tion (left plot of Figure D. repeating the input sentence is more important than using natural labels, whereas the opposite is true for relation extraction (right plot).
From these experiments, we deduce that: (1) the model indeed uses latent knowledge about label semantics, especially when the amount of training data is low; (2) using a "natural" output format (which replicates the input sentence as much as possible) allows the model to make more accurate predictions, likely by encouraging the use of the entire input as context; (3) alignment helps in locating the correct entity spans in the input sentence, and in correcting mistakes made by the model when replicating the input.
C ANALYSIS OF GENERATION ERRORS
The performance of TANL crucially depends on the quality the generated output sentences. Fig- ure 5 shows how often the following kinds of generation errors occur the CoNLLO4 dataset.
Reconstruction errors: the output sentence does not exactly replicate the input sentence.
Format errors: the augmented natural language format is invalid.
Entity errors: there is at least one relation whose predicted tail entity does not match any predicted entity.
Label errors: there is at least one predicted entity or relation type that does not exactly match any of the dataset's possible types.
Reconstruction emrors by far the most common, but they mitigated by alignment step. When using the full CoNLL04 training dataset, other errors appear very infrequently; therefore, not necessary add further post-processing steps mitigate them. We perform this generation
23

---

Published as a conference paper at ICLR 2021
Percentage of test sentences
00%
Replication errors
Format errors
Entity errors
60%
Label errors
30%
20%
10%
0%
10% Percentage training data
1007
Figure Percentage of output sentences presenting different kinds of errors, when train- ing with a variable portion of the CoNLL04 training dataset.
enror analysis the CoNLLO4 because it is the smallest of the benchmarks consider, and as result, the generation errors on CoNLL04 are likely to the most significant. Yet when training on only limited portion of the training data, format, and entity errors do occur. In this low-resource setting, TANL would benefit from additional post-pross post-processing. We leave the investigation of such post-processing strategies aimed at low-resource scenarios for future work.
24

---

Published as a conference paper at ICLR 2021
Table 5: Input-ouiput examples for all structured prediction datasets.
Dataset
Input
Output
CoNLL04
Boston University's Michael Papagiamnis said he believes the crater was created 100 million years ago when a 50-mile-wide meteorite slammed into the Farth.
Boston University I lorgunizuriinn I's MMichael Papuyriannis person
works for Boston University said he believes the crater was created [ 100 million years other ] ago when 50-milo-wide metcorite slammed into the [ Earth Jocution J.
ADE Progressive hypoxemia mandated endotrucheal rituximab administration and led to death 4 weeks
I Progressive hypusemia I disease effect rituxinab mandated endotracheal intubation week rituximab drug administration and led to death 4 woeks after admission.
intubation
week
after admission.
NYT
At the Tribero Coach depot East Elmhurst. Queens, this morning. about 20 workers were held
At the Triboro Coach depot East Elmburst location neighborkood of = Queens I. Queens location contatios = East Elmhurst I. this moming, about 20 workers wore or carried red union bandannas and held placards with messages like, "The Mayor Lied, There Goes Your Ride" and "On Strke.""
carried union handannas placards with messages like, "The Mayor Lied, There Goes Your Ride" and "On Strike."
ACE2005 (entity-rel
that the very joyous town of palestine, west virginia, on the news that jessica lynch is eventually going to come home.
I that geographical entity the very joyous town geographical entity I of palestine I geographical entity I part of west virginia J. [ west virginia geographical entity on news jessica lynch person located in home ] is eventually going to come [ home geographical entity 1.
extraction)
CoNLLO3
Chariton, 61. and his wife. Poggy, became citizens of Ireland when they formally received Irish passports from deputy Prime Minister Dick Spring who said the honour had been made recognition achievements the national soccer manager.
Charlton person 61, and his wife. Pepgy person became cltizens of I Ireland location when they formally received | Irish miscellaneoux passports from deputy Prime Minister Dick Spring person ] who said the honour had been made in recognition of [Charfton I person I'ss achievements as the national soccer manugr..
Charlton's
OnioNotes
The eventual court decision could bevome a landmark in Dutch corporate because lawsuit ASKO plans to file would be the first challenge the entire principle and practice of companies issuing varing prefemed shares management- - controlled trusts to dilute voting power of common stockholders.
The eventual court decision could becomee a landmark in Dutch nationality religious political group corporate law hecause lawsuit [ ASKO I organization plans to file would be the [ first ordinal I to challenge the entire principle and practice of companies issuing vating prefemeed shares management controlled trusts to dilute voting power of common stockholders.
GENIA
Activation of CD4 positive T cells a primary requirement for huraa immunndeficiency virus (HTV) entry, efficient HIV replication, and progression to AIDS. Utilizing CD4 positive cell lines and purified I normal individuals, we have demonstrated that native envelope glycopratcins can induce activation of transcription factor, nctivated protein- 1 (AP... l).
Activation of [ CD4 positive [ cells cell rype ] I cell rype is a primary requirement for human immunadeficiency virus (HIV) entry, efficient HIV replication, and progression to AIDS, Utilizing CD4 positive T cell lines I cell line I and I purified [ ( cells I cell rype have demonstrated
cell type I from normal individuals, native envelope glycuproteins protein J of IIVV, [ sp 160) protein I induce activation of transcription factor, activated protein- protein
can
HIV, gn 160,
(( AP-1 protein
ACE200S
While Starbucks does partner (airlines, airports, Barnes & Noble), most of its stores are company owned.
While | Starbucks orgumization does partner il airfines orgunization 1.1 airports faacllity J. Bames & Noble lorganization J). most of its organization ] stores [ucility ] focility ] are company organization ] owned.
(NER)
TACRED
The leader of the group. I Laura Silsby, I. businesswoman
relationship between Laura Silsby I and [ Idaho I state provinces residence
describes hersell' a missionary as well, has also come under serutiny home in Idaho J. where employees complain of unpaid wages and the state has placed liens on her company bank account. The relationship betwsen Laura Silsby and [ Idaho is
FewRel
June 2017 President Catalonia Carles Puigdemont announced that a referendom on [ Catalan indeqendence I would be held on October 2017. The relationship between referendum I and I Catalan independence is
relationship between [ main subject
referendum ] and Catalan independence ]
25

---

Published as a conference paper at ICLR 2021
Dataset
Input
Output
CoNLL.2005
signaled it's willing to entertain the notion, and the lawyers have renewed their argumens Texas and eight other states where the defense is permilieed under state law.
Suil, one appeals
appeals court signaled willing entertain the notion, and the lawyers have renewed their arguments in Texas eight other states AM-LOC where R-AM-LOC' the defense IA/ permitted I under state law AM AM-LOCC)
(SRL.)
CONT1.2012 (SRL)
I'm just delighted now, nine years. Inter, to able to welcome you and to learn about the great changes which have occurred in your country since was there.
I'm just delighted now, nine years later, to be able to welcome you here and to Iearn about the preat changes ARG!
which RARRGI have occurred your country location I since was there IARGM-TMM
ACE2005 (event trigger id.)
Hoon said Saddam's regime was crumbling under the pressure of a huge air assault.
Hoon said Saddam 's regime was crumhling end organization undor pressure huge assault lttack)) ].
ACI2000 (event argument cl.)
Chuirman Jack Welch is seeking work-related documents estranged wife high- stakes [ divnrce I divorce ] case.
Chairman Jack Welch is seeking work-related documents of estranged individunt person individual I person divorce ] tiigh-ttakes, divorce case.
C (coreference res.)
What's your TV series comning Oh it's new show for FX called Beef. Beef? You shonting? Um going Shriport tomorrow. Shriport Louisiana.
What's [[ your new TV series coming up Oh TV series coming I's new show for FX called Beel, I Beef! it [[ Beef, Beef?
your shooting? Um I'm going to [Shriport tomorrow. [ Shriport Louisiana. Shriporr ]
MuliWOZ
[sser : looking place to has cheap price range sbould be in a type of hotel agent okay, do you have specific area you want stay in? [ user] ] no, just need to make sure it cheap. ob, and need parking
that
helief hotel not given, hotel book day given, hotel book people not given, hotel book stay not given, hotel internet given,
given, hotel parking yes, hotel price range gheap, hotel stars not given, hotel rype hotel [ [ellief))
26


Reviews:
[{"review_id": "nuEfqjWBMn", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nIn this paper, the authors proposed a unified seq2seq model for structured prediction tasks in NLP. They let the seq2seq model produce mixed outputs of special tokens and the original sentence. Different NLP tasks, including relation classification, entity relation extraction, NER, etc. can be converted into this seq2seq problem by adding special tokens.  The experiments show that the proposed model does better than the previous state-of-the-art, albeit with the help of multi-task and multi-dataset learning, on some of the tasks/datasets. \n\n## Pros:\n1. A unified framework that allows for multi-task and multi-dataset learning. Their experiments also show that their model could benefit from multi-task, multi-dataset learning. The experiments on few-shot relation extraction show that their model could transfer knowledge from high-resource tasks to low-resource tasks.\n2. The formulation is neat and extensible. More difficult structured predictions tasks (in terms of structure), e.g. dependency parsing, are in principle convertible to this format, although the authors didn't try it on parsing. \n\n## Questions:\nFor structured prediction tasks, searching for the best output is a crucial part. However, this paper doesn't explore search strategies too much. Only in the appendix, beam search is mentioned. More concretely, I would suggest the authors try to answer the following questions:\n1. How much could we improve the current model purely by using better the searching strategy (the headroom)? Different from CRF, the structured prediction model is not markovian, which means we have to brute-force the best output. Is it possible to calculate such an upper-bound performance of the current model? If so, what would be the upper-bound for each task?\n2. In this paper, the DP alignment method is a post hoc method. What if we add such a monotonic alignment to the decoding process? \n\nTo summarize, I think this is a good paper in terms of extensive experiments and convincing results, but the search strategy still needs to be explored and justified for structured prediction tasks. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "hRjddVT9Ca3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed TANL, a novel approach by using generative models to solve structured prediction tasks in NLP. The key idea is that we can formulate this as a translation from natural language input to the augmented natural language with the structure of the input, and we can leverage the label semantics of the label in the augmented natural language output. In this way, it enables transfer learning from large pretrained generative language models such as T5. This augmented natural language unifies the input/output format for many structured prediction tasks in NLP and thus can facilitate multi-task learning. The experiments cover a dozen datasets on seven different structured prediction tasks (CoNLL04, ADE, NYT, ACE2005 for Entity Relation Extraction, CoNLL03, OntoNotes, GENIA, ACE2005 for NER, TACRED, FewRel 1.0 for Relation Classification, CoNLL05 WSJ, CoNLL05 Brown, CoNLL2012 for SRL, ACE2005 for Event Extraction, CoNLL2012 for Coreference Resolution, MultiWOZ for DST). The result demonstrates that the proposed approach can achieve SOTA on several of them for Entity Relation Extraction, Relation Classification, SRL.\n \nStrengths:\n- The idea of using the generative approach to solve structured prediction tasks in NLP is very novel to me, and the author shows there is a possible output format unification for a wide range of tasks and datasets, and thus we can study them all together using a single model even with the same set of hyperparameters.\n \n- The format of output augmented natural language is simple while being able to encode different structures even with nested entities.\n \n- The experimental results show that this generative approach even has superior performance while being much simpler than other task-specific classification models which require careful model architecture design for different tasks. \n \n- The author also shows that the proposed approach is data-efficient and has an advantage in low-resource settings.\n\nWeakness:\n- The experiments show that the proposed approach does not perform very well on two tasks: CoNLL-2012 for coreference resolution and MultiWOZ 2.1 for Dialog State Tracking. For CoNLL-2012, TANL has Avg.F1 72.8 while the SOTA CorefQA has 79.9. The author mentioned that CorefQA uses additional question answering pretraining, but TANL is based on T5 which also uses large-scale pretraining on question answering, translation, and summarization. On MultiWOZ 2.1, TANL has 50.5 while the SOTA has 55.7\n \n- The advantage of multi-dataset or multi-task learning is not obvious. The only major improvement from this is on CoNLL04, ADE, CoNLL05 Brown, possibly due to the fact that they are small datasets.\n \n- There is no significance test for the results in Table 1.\n \nQuestions:\n- Have you tried with other T5 models such as T5-large? or other generative models such as BART/GPT-2?\n- I didn't understand your explanation for why the multi-task model has lower scores on coreference resolution. Do you mean you only use smaller training data?\n- Why there is no multi-dataset for Relation Classification? \n- Why there is no multi-task or multi-dataset for DST?\n- How do you compare your methods with Athiwaratkun et al., 2020\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2yVvAEz2Lck", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a text-to-text translation approach to a variety of structured prediction problems. The authors explore several ways to represent each structured prediction problem as a text-to-text translation task and finetune T5 (Raffel et al., 2020) to perform each task. The resulting model gives better results than existing models in the tasks of joint entity relation extraction, relation classification, and semantics role labeling.\n\nAlthough this work is a rather straight-forward application of T5 to structured prediction problems, the reported experimental results and lessons learned regarding good text-to-text representations on the extensive set of structured prediction tasks should be useful to the community. The experimental results in the multi-dataset and multi-task settings are also interesting (although not much analysis is given in the paper). \n\nI was wondering why the authors did not apply their approach to the task of syntactic parsing, which is probably the most well-studied structured prediction task in NLP. Is there any difficulty in applying the same technique to dependency or phrase-structure parsing?\n\nI think the authors should also discuss the limitations of their approach. Are there any structured prediction tasks in NLP that are difficult to tackle with their approach?\n\nIn section 5.2, it is a bit surprising that the model was able to learn the correct output format with only 9 sentences. Were there no invalid outputs?\n\nMinor comments:\np.3: the current state-of-the-art -> the current state of the art?\np.4: dynamic-programming -> dynamic programming (DP)?\np.5: don’t -> do not?\np.5: previous state-of-the-art -> previous state of the art?", "extra_scores": {}, "extra_text": {}}, {"review_id": "-ieQZWWChE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Recently, multiple research papers focus on task transformations by bridging the gap between different tasks[1,2,3,4]. The original idea may go back to [5]. This paper follows this line of research ideas by reducing a structured prediction problem to a translation problem. The general idea is novel and very interesting. By defining several manually-designed rules, multiple structured outputs are transformed into the output of the translation model. The writing is clear and well-structured. \n\nPros:\n - A novel and interesting idea for formulating structured prediction tasks to translation problems. This idea is well-motivated in low-resource scenarios and multi-task learning settings. \n - The general framework is easy to implement (only requiring some scripts).\n\n\n\nCons:\n - My main concern with the proposed approach is the decoding process. If the translated sequence has a nice structure, the transformation process is well performed. However, the translated results might be invalid for a specific task. For example, in CoNLL NER, a nested or overlapping structure might be generated. It may need specially designed rules to filter them out. However, this paper does not have many discussions on this point. I would like to know more about this part.\n - I also would like to know the effectiveness of different pre-trained language models. In this paper, a T5-base model is utilized. It might be beneficial to know the empirical effectiveness of different kinds of language models.\n - Some words are not precise. For example, the phrase \"generative models\" are frequently used to illustrate the translation model. However, in the ML field, generative models may indicate the models that have a generative process of data and model the joint distribution of observed samples.\n\n\nI am willing to increase my score if some of the questions are well clarified by the authors.\n\n[1] Strzyz et al. Viable Dependency Parsing as Sequence Labeling, NAACL 2019\n\n[2] Yu et al. Named entity recognition as dependency parsing, ACL 2020\n\n[3] Gómez-Rodríguez et al. Constituency parsing as sequence labeling, EMNLP 2018\n\n[4] Li et al. A Unified MRC Framework for Named Entity Recognition. ACL 2020\n\n[5] Vinyals et al. Grammar as a Foreign Language. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a general framework to use MT to solve structural prediction problems.
The method is well developed and be verified in an arrange of tasks including entity recognition, relation classification, event extraction, semantic role labelling, coreference resolution and dialog state tracking and achieves new state-of-the-art in some of these tasks.  
Further experiments also suggest the method is especially effect for low resource scenario, if the label semantics can be used appropriately.Further experiments also suggest the method is especially effect for low resource scenario, if the label semantics can be used appropriately.
All reviewers agreed to accept the paper and gave very positive comments.  Some reviewers pointed out that the methods do not improve the performance significantly in some of the tasks.  And more analysis is wanted (by reviewer1).
Decision: Accept (Spotlight)