Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Recently, multiple research papers focus on task transformations by bridging the gap between different tasks[1,2,3,4]. The original idea may go back to [5]. This paper follows this line of research ideas by reducing a structured prediction problem to a translation problem. The general idea is novel and very interesting. By defining several manually-designed rules, multiple structured outputs are transformed into the output of the translation model. The writing is clear and well-structured. 

Pros:
 - A novel and interesting idea for formulating structured prediction tasks to translation problems. This idea is well-motivated in low-resource scenarios and multi-task learning settings. 
 - The general framework is easy to implement (only requiring some scripts).



Cons:
 - My main concern with the proposed approach is the decoding process. If the translated sequence has a nice structure, the transformation process is well performed. However, the translated results might be invalid for a specific task. For example, in CoNLL NER, a nested or overlapping structure might be generated. It may need specially designed rules to filter them out. However, this paper does not have many discussions on this point. I would like to know more about this part.
 - I also would like to know the effectiveness of different pre-trained language models. In this paper, a T5-base model is utilized. It might be beneficial to know the empirical effectiveness of different kinds of language models.
 - Some words are not precise. For example, the phrase "generative models" are frequently used to illustrate the translation model. However, in the ML field, generative models may indicate the models that have a generative process of data and model the joint distribution of observed samples.


I am willing to increase my score if some of the questions are well clarified by the authors.

[1] Strzyz et al. Viable Dependency Parsing as Sequence Labeling, NAACL 2019

[2] Yu et al. Named entity recognition as dependency parsing, ACL 2020

[3] Gómez-Rodríguez et al. Constituency parsing as sequence labeling, EMNLP 2018

[4] Li et al. A Unified MRC Framework for Named Entity Recognition. ACL 2020

[5] Vinyals et al. Grammar as a Foreign Language. 


review_title: A good and novel idea on structured prediction. 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: # Summary
In this paper, the authors proposed a unified seq2seq model for structured prediction tasks in NLP. They let the seq2seq model produce mixed outputs of special tokens and the original sentence. Different NLP tasks, including relation classification, entity relation extraction, NER, etc. can be converted into this seq2seq problem by adding special tokens.  The experiments show that the proposed model does better than the previous state-of-the-art, albeit with the help of multi-task and multi-dataset learning, on some of the tasks/datasets. 

## Pros:
1. A unified framework that allows for multi-task and multi-dataset learning. Their experiments also show that their model could benefit from multi-task, multi-dataset learning. The experiments on few-shot relation extraction show that their model could transfer knowledge from high-resource tasks to low-resource tasks.
2. The formulation is neat and extensible. More difficult structured predictions tasks (in terms of structure), e.g. dependency parsing, are in principle convertible to this format, although the authors didn't try it on parsing. 

## Questions:
For structured prediction tasks, searching for the best output is a crucial part. However, this paper doesn't explore search strategies too much. Only in the appendix, beam search is mentioned. More concretely, I would suggest the authors try to answer the following questions:
1. How much could we improve the current model purely by using better the searching strategy (the headroom)? Different from CRF, the structured prediction model is not markovian, which means we have to brute-force the best output. Is it possible to calculate such an upper-bound performance of the current model? If so, what would be the upper-bound for each task?
2. In this paper, the DP alignment method is a post hoc method. What if we add such a monotonic alignment to the decoding process? 

To summarize, I think this is a good paper in terms of extensive experiments and convincing results, but the search strategy still needs to be explored and justified for structured prediction tasks. 
review_title: Good paper, a few suggestions

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a text-to-text translation approach to a variety of structured prediction problems. The authors explore several ways to represent each structured prediction problem as a text-to-text translation task and finetune T5 (Raffel et al., 2020) to perform each task. The resulting model gives better results than existing models in the tasks of joint entity relation extraction, relation classification, and semantics role labeling.

Although this work is a rather straight-forward application of T5 to structured prediction problems, the reported experimental results and lessons learned regarding good text-to-text representations on the extensive set of structured prediction tasks should be useful to the community. The experimental results in the multi-dataset and multi-task settings are also interesting (although not much analysis is given in the paper). 

I was wondering why the authors did not apply their approach to the task of syntactic parsing, which is probably the most well-studied structured prediction task in NLP. Is there any difficulty in applying the same technique to dependency or phrase-structure parsing?

I think the authors should also discuss the limitations of their approach. Are there any structured prediction tasks in NLP that are difficult to tackle with their approach?

In section 5.2, it is a bit surprising that the model was able to learn the correct output format with only 9 sentences. Were there no invalid outputs?

Minor comments:
p.3: the current state-of-the-art -> the current state of the art?
p.4: dynamic-programming -> dynamic programming (DP)?
p.5: don’t -> do not?
p.5: previous state-of-the-art -> previous state of the art?
review_title: Straightforward application of T5 with some strong results

final_score: rating: 7: Good paper, accept
review_content: This paper proposed TANL, a novel approach by using generative models to solve structured prediction tasks in NLP. The key idea is that we can formulate this as a translation from natural language input to the augmented natural language with the structure of the input, and we can leverage the label semantics of the label in the augmented natural language output. In this way, it enables transfer learning from large pretrained generative language models such as T5. This augmented natural language unifies the input/output format for many structured prediction tasks in NLP and thus can facilitate multi-task learning. The experiments cover a dozen datasets on seven different structured prediction tasks (CoNLL04, ADE, NYT, ACE2005 for Entity Relation Extraction, CoNLL03, OntoNotes, GENIA, ACE2005 for NER, TACRED, FewRel 1.0 for Relation Classification, CoNLL05 WSJ, CoNLL05 Brown, CoNLL2012 for SRL, ACE2005 for Event Extraction, CoNLL2012 for Coreference Resolution, MultiWOZ for DST). The result demonstrates that the proposed approach can achieve SOTA on several of them for Entity Relation Extraction, Relation Classification, SRL.
 
Strengths:
- The idea of using the generative approach to solve structured prediction tasks in NLP is very novel to me, and the author shows there is a possible output format unification for a wide range of tasks and datasets, and thus we can study them all together using a single model even with the same set of hyperparameters.
 
- The format of output augmented natural language is simple while being able to encode different structures even with nested entities.
 
- The experimental results show that this generative approach even has superior performance while being much simpler than other task-specific classification models which require careful model architecture design for different tasks. 
 
- The author also shows that the proposed approach is data-efficient and has an advantage in low-resource settings.

Weakness:
- The experiments show that the proposed approach does not perform very well on two tasks: CoNLL-2012 for coreference resolution and MultiWOZ 2.1 for Dialog State Tracking. For CoNLL-2012, TANL has Avg.F1 72.8 while the SOTA CorefQA has 79.9. The author mentioned that CorefQA uses additional question answering pretraining, but TANL is based on T5 which also uses large-scale pretraining on question answering, translation, and summarization. On MultiWOZ 2.1, TANL has 50.5 while the SOTA has 55.7
 
- The advantage of multi-dataset or multi-task learning is not obvious. The only major improvement from this is on CoNLL04, ADE, CoNLL05 Brown, possibly due to the fact that they are small datasets.
 
- There is no significance test for the results in Table 1.
 
Questions:
- Have you tried with other T5 models such as T5-large? or other generative models such as BART/GPT-2?
- I didn't understand your explanation for why the multi-task model has lower scores on coreference resolution. Do you mean you only use smaller training data?
- Why there is no multi-dataset for Relation Classification? 
- Why there is no multi-task or multi-dataset for DST?
- How do you compare your methods with Athiwaratkun et al., 2020

review_title: AnonReviewer4

Meta-review: This paper proposes a general framework to use MT to solve structural prediction problems.
The method is well developed and be verified in an arrange of tasks including entity recognition, relation classification, event extraction, semantic role labelling, coreference resolution and dialog state tracking and achieves new state-of-the-art in some of these tasks.  
Further experiments also suggest the method is especially effect for low resource scenario, if the label semantics can be used appropriately.Further experiments also suggest the method is especially effect for low resource scenario, if the label semantics can be used appropriately.
All reviewers agreed to accept the paper and gave very positive comments.  Some reviewers pointed out that the methods do not improve the performance significantly in some of the tasks.  And more analysis is wanted (by reviewer1).
Decision: Accept (Spotlight)