Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: One existing problem in information extraction is that it consists of a wide range of tasks. Previous attempts to unify all IE tasks with one architecture are either complex or time-consuming to decode. To this end, this paper proposes a simple yet effective paradigm for unified IE by interpreting all IE tasks as span extraction and span relation extraction. Specifically, the paper introduces Plusformer, which models the axis-aware interaction and location interaction on top of the token-pair feature matrix. Experiments on 12 IE tasks show that the proposed approach achieves SOTA performance. The proposed model is also much faster than prior unified IE models
strength_and_weaknesses: Strengths: 
* The paper proposes an interesting idea to unify all IE tasks into token-pair classification tasks.
* The paper has performed extensive experiments on 12 IE tasks and showed the superiority of the proposed method on both performance and decoding speed.
* The paper is overall well written.

Weaknesses:
* The paper proposed a new attention mechanism for token-pair feature matrix. It would be better if the authors could provide some visualization examples on axis-aware attention and location interaction attention. The authors could also provide qualitative examples of how this new attention mechanism helps with performance.
* In equation (4), it is unclear the concatenation of Z_h and Z_v is along which axis.
* In Section 4.3, the author mentioned that “the performance of UTC-IE_single is from trigger/argument F1 of EE”; however, the trig. (and arg. F1) score for UTC-IE_single on ERE-EN in Table 2 is 57.01 (and 48.29), while the trig. (and arg. F1) score for UTC-IE on ERE-EN in Table 1 is 60.20 (and 52.51). According to the appendix, it seems like the difference is due to the usage of different pre-trained models? If so, the authors might want to consider pointing out this in the main content. 
clarity,_quality,_novelty_and_reproducibility: This work is of good quality and is overall clear and easy to follow. Moreover, the paper proposes an interesting idea to unify all IE tasks into token-pair classification tasks and proposes a relatively new attention mechanism for token-pair feature matrix; therefore, the originality of the work is good.
summary_of_the_review: The paper proposed an interesting idea to unify all IE tasks into token-pair classification tasks and proposes a relatively new attention mechanism for token-pair feature matrix. Extensive experiments and ablation studies have been performed to demonstrate the superiority of the proposed method on performance and decoding speed. Overall, I believe this paper should be accepted.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: To address the information extraction (IE) setting for multiple tasks, the authors propose to formulate the IE problem as token-pair classification. The proposed formulation represents the spans and span-pair relations through token-pair interactions which is similar to table-filling approaches. The experiment results show benefits compared to prior methods on 10 datasets.
strength_and_weaknesses: Strengths
 - The speed comparison in section 4.4 empirically supports the computational efficiency of the proposed model.

Weaknesses
 - The motivation for the token-pair reformulation is not convincing as there is no fundamental challenge in unified IE that is being addressed.
 - There is limited novelty as the model combines several existing components (position embeddings, CNN) that are not specific to the token-pair formulation.
 - The setting of unified IE is limited as previous works (e.g. GLAD benchmark) included more tasks such as coreference, parsing and aspect-based sentiment analysis.
 - The SpanRel model (Jiang et al., 2020) is similar to this work in motivation and formulation, but the authors’ comparison is superficial and SpanRel is missing from the experiment results.

clarity,_quality,_novelty_and_reproducibility:  - In the abstract, it is a stretch to claim to unify all IE tasks as there are only 4 tasks in the experiment setting.

 - Using the start and end token of the span as a representation is common practice for information extraction and relation extraction.

 - It is not clear why the CNN is limited to kernel size 3.

 - In Table 4, it is not clear what is the difference between UTC-IE without “axis-aware” and CNN-IE which does not have PlusAttention.

summary_of_the_review: Although the proposed approach is relatively simple and can be applied to multiple IE tasks, there are issues with the motivation and novelty of the work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a unified information extraction (UIE) framework based on the token pair classification idea. The UIE solution allows modeling all types of IE tasks such as NER, RE and EE with just one unified model. Also the authors propose a Plusformer on top of the token-pair feature matrix for better feature learning. They show their method advances better efficacy and efficiency. The proposal of UIE simplified with a token pair scheme is technically sound, and also the experiments are solid. However, I'm afraid this paper may suffer from some key flaws, with respect to the topic of UIE, which I list bellow.


1. Overall, this paper made limited contributions.

a. Token pair classification, aka., table filling/grid tagging, is one of the most typical and long-standing task modeling schemes for the information extraction jobs in NLP community.
Also, the paper claims to advance with unifying all IE tasks, yet the idea of token pair classification has been well explored in NER [2], RE [1] and EE [3].
Thus this paper leverages such a method for UIE, which in the perspective of technique, contributes less.

b. The proposed Plusformer is simply the combination of CNN layer with axis-aware addition. The module is designed for modeling the local interaction. However as far as I know, this will fall short on modeling some of the long-range dependences between elements for especially the RE/EE tasks [5].



2. Problematic/unconvincing motivations

a. The authors indicate the issues of the existing works: "However, these models are either complex to be designed or time-consuming to decode", which could be problematic. For example, generative seq2seq methods are very easy and straightforward, while graph-based methods enjoy high running efficiency due to high parallel architecture.

b. The motivation and designing detail of the use of token pair in this paper is essentially the same as the recent work [2], while the authors seem to skip [2] without acknowledging it, which is unacceptable.

c. Suspicious unified IE. I don't feel that the task modeling should be called as unified information extraction, since it seems different task types (NER/RE/EE) come with one specific task formalization. [4] models all different IE tasks with just one real single unified formulation under a well-designed structured extraction language/expression.

And also the paper writing or technique description is not clear, e.g.,
1) Y [s, e, t] = Y [e, s, t] , what does the [*] mean?
2) how to solve the issue of overlapping elements in NER/RE/EE with your formulization?
3) the task formulization lacks necessary examples for certain IE tasks.


3. Another potential problem in the technique method part: one of the key reasons to propose UIE in IE community lies in the capability of knowledge transfer, i.e., pre-training once and transferring everywhere, to naturally support few-shot learning for downstream IE task. But the UIE in this paper seems to fail to well support this advantage; they more likely perform unified IE task modeling, and nothing more. This would limit the real merit of UIE.


4. As an extension of token pair based IE method, this work didn't survey the existing works for this line, e.g., differences.
Also some critical existing works are overlooked [1,2,3].



[1] Modeling Joint Entity and Relation Extraction with Table Representation. EMNLP 2014.

[2] Unified Named Entity Recognition as Word-Word Relation Classification. AAAI 2022.

[3] OneEE: A One-Stage Framework for Fast Overlapping and Nested Event Extraction. COLING 2022.

[4] Unified Structure Generation for Universal Information Extraction. ACL 2022.

[5] Graph Convolution over Pruned Dependency Trees Improves Relation Extraction. EMNLP 2018.

strength_and_weaknesses: See above.
clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: See above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a transformer-based architecture for jointly performing three information extraction (IE) tasks: named entity recognition (NER), relation extraction (RE), and event extraction (EE). The  main contribution of this work is the Plusformer, an attention mechanism that is well-suited for capturing pairwise relationships over the Biaffine model.

Strengths:
- A new architecture for IE tasks.
- An extensive evaluation over a large set of benchmarks showing improvements over baselines.

Weaknesses:
- The various design decisions for Plusformer are not well ablated or motivated. For example, is the 3x3 CNN a good choice? What if a traditional transformer output was used instead? Why 3x3 and not 4x4 for example?
- Similarly, is the Biaffine model necessary after a PLM and before a Plusformer? Transformers are good at capturing (all) pairwise relationships among inputs, could the Plusformer be combined just with the standard PLM?
- What is the effect of different pretrained LMs (PLMs) to the performance of the model? What is the effect of the size of the PLM to the target tasks?

Overall, I recommend that this paper is not accepted at its current state as some additional evaluations and ablations would be needed to illustrate the advantages of the proposed model.

Minor notes:
- The Plusformer attention has a resemblance to many of the sparse transformer architectures (e.g. BigBird, Axial Transformer see Tay, Yi, et al. "Efficient transformers: A survey." ACM Computing Surveys (CSUR) (2020).). This should be acknowledged in Sec 3.2.
justification_for_why_not_higher_score: While accepting this paper would showcase a previously non-existing neural architecture, this would have relatively small novelty and it won't be clear how different elements of the architecture contribute to the performance over the IE task.
Decision: Reject