Set Learning for Generative Information Extraction
Authors: Jiangnan Li
Yice Zhang
Bin Liang
Kam-Fai Wong
Ruifeng Xu
Venue: EMNLP 2023 Main (2023)
Keywords: Information Extraction
TL;DR: We propose a task-agnostic approach called "Set Learning" to enhance generative information extraction by taking into account the unordered nature of structured tuples.

Abstract: Recent efforts have endeavored to employ the sequence-to-sequence (Seq2Seq) model in Information Extraction~(IE) due to its potential to tackle multiple IE tasks in a unified manner.
Under this formalization, multiple structured objects are concatenated as the target sequence in a predefined order.
However, structured objects, by their nature, constitute an unordered set. Consequently, this formalization introduces a potential order bias, which can impair model learning.
Targeting this issue, this paper proposes a set learning approach that considers multiple permutations of structured objects to optimize set probability approximately. Notably, our approach does not require any modifications to model structures, making it easily integrated into existing generative IE frameworks. Experiments show that our method consistently improves existing frameworks on vast tasks and datasets.

Set Learning for Generative Information Extraction
Jiangnan Li Yice Zhang! Bin Liang" 1..4 Kam-Fai Wong', and Ruifeng Xu 12,3+
Harbin Insitute of Technology, Shenzhen, China
2 Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies
Peng Cheng Laboratory, Shenzhen, China
The Chinese University of Hong Kong, Hong Kong, China
lijiangnan@stu. hit. edu. cn, zhangyc_hi t@i 63. com, suruifengehit.een edu. cn
Abstract
positive
positive
The drinks aspect
always well made and wine selection
fairly priced.
Recent efforts have endeavored employ the sequence-to- prruence-to-equeene so-sequence (Seq2Seq) model in In- formation Extraction (IF) due its potential tackle multiple IE tasks in unified manner. Under this formalization, multiple structured objects are concatenated as the target sequence in a predefined order. However, structured ob- jects, by their nature, constitute an unordered set. Consequently, this formalization intro- duces potential order bias, which can impair model learning. Targeting this issue, this paper proposes set learning approach that consid- ers multiple permutations of structured objects to optimize set probability upproximately. No- tably, approach does not require any modifi- cations to model structures, making it easily in- tegrated into existing generative IE frameworks. Experiments show that our method consistently improves existing frameworks on vast tasks and datasets.
opinion
aspoct
opinion
(drinks, well made, positive) (wine selection, fairly priced, positive)
[drinks, well made, positive) (wine selection, fairly priced, positive)
Predicted triplets are sorted the predefined ordier
(drinks, well made, positive) (wine selection, fairly priced, positive)
X
IX
(wine selection, fairly priced, positive) (drinks, well made, positive) Pradicted triples are sorted by predefined order
We use triplet extraction in aspect-based senti- ment analysis as an example to illustrate the order bias. As depicted in (b), the model incurs a significant loss value despite correctly generating the triplets.
Figure I:
However, essential aspect has been over- looked in these works, namely, that multiple struc- tured objects constitute a set. As illustrated in Fig- ure 1, assigning predefined order introduces a harmful order bias, which violates the inherent unordered nature and makes the model lose its gen- eralizability between permutations. Previous works have attempted to address this issue by modifying the decoder to generate multiple objects in an derless and parallel manner (Sui et al., 2021; Tan et al., 2021: Ye et al., 2021; Mao al., 2022), but these methods reduce model universality and make difficult to combine with existing PLMs.
1 Introduction
Information Extraction (IE) aims to identify struc- tured objects from unstructured text (Paolini et al.. 2021). Recently, many research efforts have been devoted to using sequence-to-sequenee (Seq2Seq) models to solve IE tasks (Lu et al., 2021; Huguet Cabot and Navigli, 2021; Yan al., 2021a,b; Josifoski et al., 2022). This generative approach enables the development of universal IE architecture for different IE tasks (Lu et al., 2022: Fei et al., 2022; Lou et al., 2023). Moreover, com- bining the generative approach with Pre-trained Language Models (PLMs) has shown promising results in many IE tasks (Ma et al., 2022).
Our objective is to tackle order bias while pre- serving the advantages of generative IE. Inspired by Qin al. (2019), we propose a novel approach called set learning for generative IE. The key idea of set learning is taking into account multiple possible orders to approximate the probability of structured objects set, thereby reducing order bias caused by only considering the predefined order.
To formulate IE task Seq2Seq problem, two steps are involved: (1) transforming each struc- tured object into flat sub-sequence, and (2) sorting and concatenating multiple sub-sequences accord- ing to a predefined order.
Our approach is task-agnostic and does not necessitate any modification to model structures. These strengths enable the seamless integration of
Corresponding Author

---

set learning with existing off-the-shelf methods. We conduct extensive experiments vast tasks and datasets. The results prove our approach can signif- icantly and consistently improve current generative IE frameworks.
tive IE framework. It decomposes the conditional
probability p( (M) X) using the chain rule:
Lseq2Ssq = log logp(T (ep((T(()) (Y) X);
(1)
L
Raw Samples
pog (an (Ye,
T.
Reciew: The drinks are always well made and wine selection is fairly priced.
Triplets (drinks, well made; positivel, (wine sdection. fairly priced, positive)
where I, 1 (V) denotes the number of target tokens, and Y<t ly132- yt-1] denotes the se- quence of target tokens up the t-th position.
Sampling
Training Samples
Review: The drinks well made and wine selection fairty prioed.
Permutation I: (drinka, will mads, posilive) (vine wdection, lion, lairly prical, posilive Permutation 2: (wine solcction, faizty priced, positive), (drinks, well made, postive
2.2 Proposed Set Learning Approach
sarning Sampled
2
Permulations
The Seq2Seq learning paradigm optimizes gener- ative models by maximizing p(=* X). How- ever, both & and Y are inherently unordered sets. Consequently, solely optimizing for a single per- mutation introduces a bias, as indicated by the in- equality expressed in Eq. 2:
Seq2Seq Model with Set Learning
Figure Overview of the proposed set learning ap- proach. Our approach first samples permutations and then conducts set learning using sampled permutations.
p(S) X) p(Y X) & p(x X).
(2)
2 Methodology
To address this limitation, it is crucial to compute the set probability by considering all possible per- mutations, which can be formulated as follows:
2.1 Generative IE and Seq2Seq learning
This section describes the general form of both IE tasks and generative IE.
p(S X) p
X)
Σ (Y):(Y)) X), (3)
Formally, an IE task generally takes text X [2), 222, #2,,
Th (P)Em))
. as input and outputs a set structured objects = {s1. 82, -.. Each structured object contains one or more spans from the input text, along with their types and relations.
where II(Y) represents all permutations of Y.
Loss Function Building upon Eq. 3, we define the following loss function. By minimizing this loss function, we can provide unbiased optimization for the generative model.
Generative IE usually takes two steps to trans- form an IE task into generative paradigm: (1) flat- ten all elements of S into sub-sequences Y, where Y is a set with the same size as S; (2) concatenate sub-sequee sebbsequesces of Y according to the predefined order π², resulting in permutation x" (Y). Here. 7" (Y) is a flat sequence composed with tokens [y1, 1/2 ...]. With the above two steps, a raw sample is converted into a sequence pair (X, x (Y)) that can be used to train Seq2Seq models.
Σ ((T=(2) |x)+ (4) (T)
Lset log
where 1, serves as a normalization term aimed at mitigating the impact of sequence length. In Ap- pendix B, we analyze other available loss functions.
Permutation Sampling The factorial complexity of size of I[(Y) makes direct optimization of Eq. computationally expensive. To overcome this challenge, we employ a sampling strategy to select subset from II(Y). Specifically, we select the top-k permutations that are most similar to ' (Y) among all possible permutations. The similarity be- tween permutations computed using token-level edit distance. During model training, we substitute the full set of permutations in Eq. with the sam- pled subset, effectively reducing the computational burden.
Remarkably, most of the existing frameworks sort structured objects according to their positions within the input text since it generally achieves good performance (Yan al., 2021a; Lu et al., 2022). Therefore, throughout the subsequent sec- tions of this paper, we adopt the symbol n to de- note this sorting method and refer to as "the reference permutation".
Seq2Seq Learning (Sutskever et al., 2014) is the default approach employed for training a genera-

---

Baseline
Unitied ABSA
REBEL
Text2Event
Unified NER
Backbone
BART- Basc (Lewis al., 2020)
BART-Large
T5-Basc
BART-Large
Task
ABSA Triplet Extraction
RE Triplet Extraction
End-to-Ead EE
NER
AVG
Daw
Dan
ACE05
Datasct
CoNLL NYT DocRED
CADEC
14res
141ap
15res
16res
14res
14lap
15tes
16res
T-C
A-C
Seq2Seq
72.4
57.5
60.1
69.9
65.2
58.6
59.2
67.6
75.4
92.0
47.1
82.2
69.2
49.8
70.6
Uniform
73.0
57.8
62.6
72.5
71.1
58.0
61.9
69.1
74.2
91.7
45.9
81.9
68.0
47.9
70.0
+0.59
SeIRNN
71.1
56.6
59.2
68.9
63.2
57.8
59.2
67.1
76.6
92.2
47.8
68.7
48.7
69.7
-0,51
Set (Ours)
73.4
60.8
63.5
74,4
71.7
58.7
62.2
70.6
76.8
92.2
48.2
82.9
69.6
51.5
72.2
+2.10
Baseline
UEL-SEL (T5-Large Backbone (Raffel al., 2020))
Task
ABSA Triplet Extraction
RÉ Triplet Extraction
AVG A
Dataset
14res
14lap
15res
16res
CoNLL
NYT
SciERC
ACB05
Seg2Seq
73.8
63.2
66.1
73,9
73.1
93.5
33,4
64.7
Uniform
73.9
63.0
66.2
73.4
73.4
93.2
32.7
64.1
-0.23
SeIRNN
73.5
62.6
73.1
73.0
92.8
33.1
63.7
-0.58
(Ours)
74.7
73.7
93.5
35,9
+0.99
Table 1; Experimental results. Our approach achieves improvements under different tasks and baselines.
3 Experiments
sampling permutations of the training data, and (2) adapting the existing Seq2Seq learning loss func- tion of the frameworks to the proposed set learning loss function. Additionally, we utilize the micro Fi score as the evaluation metric for assessing the performance of the models.
3.1 Experimental Setup
Datasets We conduct experiments on 9 widely used datasets across four 4 well-representatiie tasks of IE: Aspect-Based Sentiment Analysis (ABSA), Event Extraction (EE), Relation Extrac- tion (RE), and Named Entity Recognition (NER). The used datasets include Semeval (Pontiki al.,
Other Available Loss Functions In addition the proposed set learning loss function, we explore two alternative loss functions, which are originally designed for multi-label classification tasks.
2014) D20a version (Peng et al., 2020) and Se- meval D206 version (Xu et al., 2020) for ABSA; CoNLLO4 (Roth and Yih, 2004), NYT (Riedel et al., 2010), ADE (Gurulingappa et al., 2012), DocRED (Yao al., 2019), and SciERC (Luan al., 2018) for RE; ACE2005 (Christopher et al., 2006) for EE and RE; CADEC (Karimi al., 2015) for NER. We provide detailed statistics of these datasets in Appendix A.
(1) SeiRNN (Qin et al., 2019) directly optimizes Eq. 3:
LSARNN = log
Σ ( x)
(5)
(2) Uniform (Vinyals et al., 2016) calculates the total negative log-likelihood of permutations:
Baselines We apply the proposed learn- ing approach to five representative frameworks. These frameworks include Unified ABSA (Yan et al., 2021b) for Aspect-Based Sentiment Anal- ysis (ABSA), REBEL (Huguet Cabot and Navigli, 2021) for Relation Extraction (RE), Text2Event (Lu et al., 2021) for Event Extraction (EE), Unified NER (Yan et al., 2021b) for Named Entity Recog- nition (NER), and UIE-SEL (Lu al., 2022) for ABSA and EE. For REBEL, reproduce its pre- training using set learning. Since there no avail- able pre-training corpus for UIE-SEL, we compare its performance without pre-training.
1
LUniform
Σ
logg((=(Y)) | X). (6)
I.
mz mx(Y))l1())
3.2 Main Results
According to the result in Table 1, the proposed learning approach consistently achieves im- provements over five baselines across nine datasets. demonstrating its effectiveness and broad applica- bility. Moreover, the experimental findings reveal that the Uniform and SeIRNN loss functions exhibit significant instability and, in many cases, lead to detrimental effects. In contrast, proposed loss function consistently and significantly improves performance, highlighting its superiority over these alternative loss functions. A more detailed analysis and discussion of different loss functions can be
Implementation Details To ensure fair com- parison, we introduce minimal modifications to the original frameworks when applying the pro-
posed set learning approach to the basclines. Our modifications primarily focus on two aspects: (1)
found Appendix B.

---

Case
71.0
58.5
14lop
Revlew: This place classy, chic, service warm and hospitable, and food outstanding.
70.5
58.0-
Seg2Seq: (food, outstanding, POS), (place, classy, POS). (service, warm, POS), (sevice, hospilable, POS).
01E ned Srand red Goind
murs
nerd
Frand rand! Emind
15res
16res
Set Learning: (food, outstanding, POS), (place, classy, POS), (servicc, warm, POS), (servico, hospitable, POS). (place, chic, POS)
00.5
68.5
Case 2:
Srand F-mndd
Review: waiter was helpful and charming, the food was perfect, and the wine was good, too.
Figure 3: Comparisons of different sampling strategies.
Seq2Seq:
(food, perfect, POS), (wine, good, POS),
Vertical axes represent scores.
(waiter, helpful, POS).
Set L Learning: (walter, charming, POS), (food, perfect, POS), (walter, helpful, POSJ, [wine, good, POS).
70
Case study. The triplets in grey represent the correct triplets that Seq2Seq learning failed to generate. During the training phase, reference order sort objects according to their position in review (aspect term first).
Figure 5:
800r0
Iice
10
Method
14res
14lap
15res
16res
iipermatatiosss
Seq2Seq
65.2
58.6
59.2
67.6
Seq2Set
64.5
55.2
57.8
64.3
Figure 4: Effect of increasing permutations.
Scq2Sct (RS)
68.8
57.4
59.6
68.1
Set (Ours)
71.7
58.7
62.2
70.6
3.3 Further Analysis
Table 2: Comparison with Seq2Set on Semeval D206 version. RS indicates reward shaping.
Sampling Strategy We analyze the effective- ness of the proposed sampling strategy by com- paring with four other strategies: random (rand), fixed-random (f-rand), random (rand*), and fixed- random* (f-rand*). The random strategy randomly samples permutations for each epoch. The fixed- random strategy randomly samples a permutation for each training sample in the first epoch and fixes it for subsequent iterations. The random* and fixed- random strategies similar to random and fixed- random, respectively, but they always include the reference permutation.
6, the incremental improvement in performance becomes slight, indicating that 6 is good trade-off between computational cost and performance.
3.4 Case Study
We present illustrative examples in Figure 5 to fa- cilitate better understanding of the proposed ap- proach. When confronted with examples that tain multiple structured objects, Seq2Seq learning may deviate from the reference permutation and omit some objects. This deviation occurs because decoding is based on likelihood rather than posi- tion. Thus solely learning single permutation can easily result in sub-optimal predictions. In contrast, set learning has a stronger generalization across permutations, enabling the generation of objects in more flexible order and recalling more objects.
As shown in Figure 3, the random strategy per- forms the worst, while the fixed-random strategy shows some improvement. Introducing the refer- ence permutation enhances their performance, but they still lag significantly behind our sampling strat- egy. These results indicate the effectiveness of our sampling strategy.
3.5 Comparison with Order-Invariant Method
Number of Permutations We conduct experi- ments to investigate the impact of the number of permutations on the model performance. The re- sults in Figure 4 indicate that the model perfor- mance improves the number of permutations in- creases. This correlation implies that an increased number of permutations lead more accurate ap- proximation of the set probability. Besides, we also find that once the number of permutations exceeds
Another view for solving the order bias issue uses order-invariant loss, which is deeply explored in label generation tasks. A representative method is Seq2Set (Yang al., 2019). To address the order bias the label generation model, Seq2Set em- ploys reinforcement learning approach that uti- lizes the Fl score of the generated label sequence

---

reward for reinforcement fine-tuning. Since the Fl score an order-invariant metric, Seq2Set reduces the impact of order bias the label gener- ation model.
pose the set learning approach for generative IE to address this issue. The proposed set learning approach defines and optimizes set probability Seq2Seq models and reduces the computational cost by permutation sampling. Notably, the pro- posed approach can be easily integrated with ex- isting generative IE frameworks as plugin to en- hance performance. Experiments demonstrate that the proposed approach significantly improves exist- ing frameworks in various tasks and datasets. We believe our work can inspire further research on the potential of set learning in other natural language processing tasks.
We apply Seq2Set for generative information extraction to compare with our method. As illus- trated in Table 2. the performance of Seq2Set is notably inferior. We speculate that the reason is that rewards in generative information extraction are more sparse and exhibit more significant vari- ance. Furthermore, even after applying reward- shaping (Ng et al., 1999) techniques to enhance Seq2Set, the improvements in performance are still marginal compared to our approach.
Overall, our experiments show that order- invariant loss does not fit for generative information extraction, where task form is more complex than label generation tasks.
Limitations
One important limitation of our approach is that users need to perform a trade-off between per- formance and computational time consumption. When the size of training data is relatively small, the time consumption will concentrate on the infer- ence phase; an increase in the time consumption of the training phase is negligible compared to a notable increase in performance. However, when the training data is relatively large, sampling a large number of permutations for each sample may only result in marginal improvement but will signifi- cantly lengthen training time.
4 Related Work
Many studies have been conducted to explore how neural networks should be designed when deal- ing with sets as inputs or outputs. Vinyals et al. (2016) and Zaheer et al. (2017) proposed that neu- ral networks should be permutations invariant to in- puts when the inputs are sets. Vinyals et al. (2016) demonstrated that the permutation of outputs signif- icantly influences the performance of RNN models when generating sets.
Additionally, an important research challenge lies in accurately estimating set probability during the inference phase of Seq2Seq models. Seq2Seq models use greedy decoding or beam search for inference, which is based on sequence probabilities rather than set probabilities. However, the sequence with maximum probability does not necessarily cor- respond to the set with maximum probability (Qin et al., 2019). Therefore, investigating novel ap- proaches that provide more accurate estimations of set probability is a valuable direction for future rescarch.
Tasks in NLP, such as multi-label classification and keyword generation, can be formulated as set generation tasks. Madaan et al. (2022) proposed a permutation sampling strategy for multi-label clas- sification tasks, which uses the dependencies be- tween labels to generate informative permutations. Mao et al. (2022) uses beam search generate all triplets corresponding an input simultane- ously for sentiment triplet extraction tasks. Ye et al. (2021) proposed the One2Set model for keyword generation, which simultaneously generates all key- words corresponding to input through a parallel decoder.
Acknowledgements
We thank the anonymous reviewers for their valuable suggestions to improve the quality of this work. This work was partially supported by the National Natural Science Foundation of China (62006062, 62176076), Natural Science Foundation of GuangDong 2023A1515012922, Shenzhen Foundational Re- search Funding JCYJ202208 181028810103 Guang- dong Provincial Key Laboratory of Novel Security Intelligence Technologies 2022B 121 1212010005.
In contrast, propose more sample and uni- versal approach to optimize various existing gener- ative IE frameworks. Our approach can easily com- bine with the off-the-shelf methods and achieve promising improvements.
5 Conclusion
this paper, we reveal the order bias issue in generative Information Extraction (IE) and pro-

---

References
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Uni- fied structure generation for universal information extraction. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 5755-5772, Dublin, Ireland. Association for Computational Linguisties.
Walker Christopher, Strassel Stephanic, Medero Julie, and Maeda. Kazuaki. 2006. Ace 2005 multilingual training corpus.
Hao Fei, Shenggiong Wu, Jingye Li, Bobo Li, Fei Li, Libo Qin, Mcishan Zhang, Min Zhang, and Tat-Seng Chua. 2022. Lasuie: Unitying information extrac- tion with latent adaptive structure-aware generative language model. Advances in Neural Information Processing Systems, 35: 11140-154777
Yi T Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi task identification of entities. relations, and coreference for scientific knowledge graph construction. Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 3219-3232 Brussels, Belgium. Association for Computational Linguistics.
Harsha Gurulingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmann-Apitius, and Luca Toldo. 2012. Development of a benchmark corpus to support the automatic extraction of drug- related adverse effects from medical case reports. Journal of biomedical informatics, 45(5):885-892.
Yubo Ma. Zehao Wang. Yixin Cao, Mukai Li, Meiqi Chen, Kun Wang, and Jing Shao. 2022. Prompt for extraction? PAIE: Prompting argument interaction for event argument extraction. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 6759-67744 6774, Dublin, Ireland. Association for Compu- tational Linguistics.
Pere-Lluís Huguet Cabot and Roberto Navigli. 2021. REBEL: Relation extraction by end-to-end language generation. In Findings of the Association for Com- putational Linguistics: EMNLP 2021, pages 2370- 2381, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Aman Madaan, Dheeraj Rajagopal, Niket Tandon, Yim- ing Yang, and Antoine Bosselut. 2022. Conditional set generation using seq2seq models. ArXiv preprint, abs/2205. 12485.
Martin Josifoski, Nicola De Cao, Maxime Peyrard,
Fabio Petroni, and Robert West. 2022. GenlE: Gen- erative information extraction. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 4626-4643, Seattle, United States. Association for Computational Linguistics.
Yue Mao, Yi Shen. Jingchao Yang. Xiaoying Zhu, and Longjun Cai. 2022. Seq2Path: Generating sentiment tuples paths of tree. In Findings of the Asso- ciation for Computational Linguistics: ACL 2022, pages 2215-2225, Dublin, Ireland. Association for Computational Linguistics.
Sarvnaz Karimi, Alejandro Metke-Jimenez, Madonna Kemp, and Chen Wang. 2015. Cadec: A corpus of ad- verse drug event annotations. Journal of biomedical informatics, 55:73-81.
Andrew Y Ng. Daishi Harada, and Stuart Russell. 1999. Policy invariance under reward transformations: The- ory and application to reward shaping. In Icml, vol- ume 99, pages 278-287. Citeseer.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequemee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguistics.
Giovanni Paolini, Ben Athiwaratkun. Jason Krone, Jie Ma. Alessandro Achille, Rishita Anubhai, Cícero Nogucira dos Santos, Bing Xiang, and Ste- fano Soatto. 2021. Structured prediction as transla- tion between augmented natural languages. In 9th International Conference Learning Representa- tions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Jie Lou, Yaojie Lu, Dai Dai, Wei Jia, Hongyu Lin, Xi- anpei Han, Le Sun, and Hua Wu. 2023. Universal information extraction as unified semantic matching. ArXiv preprint, abs/2301.03288
Haiyun Peng, Lu Xu, Lidong Bing. Fei Huang, Wei Lu, and Luo Si. 2020. Knowing what, how and why: A near complete solution for aspect- aspect-based sen- timent analysis. In The Thirty-Fourlh AAAI Con- ference Artificial Intelligence, AAAI 2020, The Thirty-Secona Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial In- telligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8600-8607. AAAI Press.
Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. 2021. Text2Event: Controllable sequence-to- structure generation for end-to-end event extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1shh International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 2796-2806, Online. Association for Computational Linguistics.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Har- ris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect

---

based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (Se- mEval 2014), pages 27-35, Dublin, Ireland. Associa- tion for Computational Linguisties.
Hang Yan, Junqi Dai, Tuo Ji, Xipeng Qiu, and Zheng Zhang. 2021a. A unified generative framework for aspect-based sentiment analysis. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ih International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 2416-2429, Online. Association for Computational Linguistics.
Kechen Qin, Cheng Li, Virgil Pavlu, and Javed Aslam. 2019. Adapting RNN sequence prediction model to multi-label set prediction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 3181-3190, Minneapolis, Minnesota. Association for Computational Linguistics.
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang. and Xipeng Qiu. 2021b. A unified generative framework for various NER subtasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5808-5822, Online. Association for Computational Linguisties.
Colin Raffel. Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-67
Pengcheng Yang. Fuli Luo, Shuming Ma. Junyang Lin, and Xu Sun. 2019. A deep reinforced sequence-to-se model for multi-label classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5252-5258.
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions with- out laheled text. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 163. Springer.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang. Jie Zhou, and Maosong Sun. 2019. DocRED: large-scale document-level relation extraction dataset. In Pro- ceedings of the 57th Annual Meeting of the Associa- rion for Computational Linguistics, pages 764-777, Florence, Italy, Association for Computational Lin- guistics.
Dan Roth and Wen-tau Yih. 2004. A linear program- ming formulation for global inference in natural lan- guage tasks. Proceedings of the Fighth Confer- ence on Computational Natural Language Learn- ing (CoNLL-2004) at HLT-NAACL 2004, pages 1-8, Boston, Massachusetts, USA. Association for Com- putational Linguistics.
Jiacheng Ye, Tao Gui, Yichao Luo, Yige Xu, and Qi Zhang. 2021. One2Set: Generating diverse keyphrases set. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume I: Long Papers). pages 4598-4608, Online. Association for Computational Linguistics.
Dianbo Sui, Chenhao Wang, Yubo Chen, Kang Liu, Jun Zhao, and Wei Bi, 2021. Set generation networks for end-to-end knowledge base population. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9650-9660) Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnahás Póezos, Ruslan Salakhutdinov, and Alexan- J. Smola. 2017. Deep sets. In Advances in Neural Information Processing Systems 30: Annual Confer- ence on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 3391 -3411.
Ilya Sutskever, Oriol Vinyals, and Quoe V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Sys- tems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Mon- Ireal. Quebec, Canada, pages 3102-3112.
Zeqi Tan, Yongliang Shen, Shuai Zhang, Weiming Lu, and Yueting Zhuang. 2021. A soquecce-toseet net-
A Dataset Statistics and Reproducibility
work for nested named entity recognition. ArXiv preprint, abs/2 105001
The hyperparameters used in all experiments are provided in Table 3. For the experiments conducted on the UIE-SEL framework (Lu et al., 2022), we were constrained by computational resources and did not perform hyperparameter tuning. Instead, used the original hyperparameters and set the number of permutations 2.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. 2016. Order matters: Sequence sequence for sets. 4th International Conference on Learning Repre- sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Lu Xu, Hao Li. Wei Lu, and Lidong Bing. 2020. Position- aware tagging for aspect sentiment triplet extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2339-2349, Online. Association for Computational Linguistics.
Detailed statistics of the datasets used in our experiments are presented in Table 4. We have ensured that our use of the datasets falls within their intended scope and aligns with existing works.

---

learning,
opsillo
stjp max mux_eo
Psampled permucations
Dais
le-4
le8
100
3.1.
lineur
Past
le-4
le-8
100
0, 1,23
linear
CADEC
lo-8
0,3
lincar
ACE 2005 EC
le4
le-8
16
30
41, 42, 43,
linear
1o-8
43,
NYT
5o-5
1o-8
40
41, 42, 43,44
lincar
DecRED
2e.5
le.8
41,
43,44
linear
Table 3: Hyperparameters used for all the experiments.
B Discussion of Loss Functions
TT(Y) E II(Y). Solving Equation 9 involves find- ing the minimum point, which requires that each Te (Y) E IIY)) has the same probability.
In this section, our primary focus is analyze the probability distributions generated by different loss functions among permutations. This analysis is essential for comprehending the performance variations observed with these loss functions.
However, in Seq2Seq models, the probability of permutations can be influenced by various fac- tors (Vinyals et al., 2016; Madaan et al., 2022). leading to an inherent imbalance in the distribution of permutations. This indicates that fitting a uni- form distribution may be challenging. Therefore, imposing strict penalty non-uni non-uniformity, such the rigorous penalty of Uniform, may result in subpar performance (Qin et al., 2019).
Uniform Loss A possible solution for opti- mising Eq. is optimize all permutations equally (Vinyals et al., 2016). This can be achieved by Uniform loss function:
Luniform
E
log (og IX) (7)
L
SetRNN Loss The SetRNN loss, proposed by Qin et al. (2019), offers alternative approach by directly optimizing the probability (Eq. 3). Instead of computing the negative log- fog-likelihood of each permutation, SezRNN loss calculates the total probability of all permutations and utilizes it to compute the loss:
(x())EII(2)
Since the Uniform loss calculates the negative log-likelihood each permutation, it imposes a sig- nificant penalty on permutations with low probabil- ity. Consequently, Uniform results in a uniform distribution over permutations. It can also be demonstrated that the minimum value of Uniform is attained when the probabilities of all permuta- tions are equal.
LSARNN -log
Σ (Y) | Ts (+)CUI((
When optimizing Uniform, a crucial insight is that the total probability of Y across all permuta- tions should be less than or equal to 1. Thus, our optimization problem aims to minimize Eq. 7 while satisfying the following constraint:
(10)
Unlike the Uniform loss, the SeIRNN loss does not assume uniformity among permutations since the distribution is not explicitly considered the total probability.
Σ (V)
(Y==(Y) 1
(8)
However, a drawback the SerRNN loss is that it does not penalize non-uniformity. which can lead the model to take shortcuts and assign almost all of the probability density to single permutation. Our experiments have shown that such shortcuts prevalent when using the SetRNN loss, which may explain its poor performance compared other loss functions..
Under the constraint mentioned above, we can formulate the optimization of Uniform constrained problem using the Lagrange multiplier method in the following manner:
L
Σ
log ((xp(=()) X)
[Y)]
To (F)n)e))
Set Loss We propose the Set loss, which is derived from the Uniform loss but relaxes the penalty for low-probability permutations to support set learn- ing better.
(9)
(() x) (x(Y)EI(()
Σ
where À is a Lagrange multiplier, and a func- tion that depends on both À and each 2(==(" (Y)) for
We start by transforming the Uniform loss into the following form:

---

use the Lniiform =
E
log log (( x)a. (11)
(=(Y)e
By moving the log outside the summation, we can relax the penalty and obtain the Set loss:
Σ p()=(Y) x) 1 (12) (+(())
Cse log
The Ser loss reduces the penalty while maintain- ing a looser uniformity restriction. Therefore, does not suffer from the "rigorous penalty" or "tak- ing shortcuts" issues observed in other loss func- tions. The probability distribution produced by the Ser loss falls in middle state between the proba- bility distributions produced by the Uniform and SeIRNN losses.
Validation Experiment We design and imple- ment an experiment to corroborate the arguments in the above discussions. Specifically,
D206 version of the 14lap dataset and set the num- ber of permutations 2, optimize the model to convergence using different losses, and then record and analyze the probability distributions generated by the different losses. Our main findings are as follows: (1) SeIRNN loss assigns more than 0.7 probability mass to single permutation with 97% probability, while the other two losses have probability of 0%. (2) Uniform loss assigns prob- ability mass of 0.48 0.52 both permutations with a probability of 26%. (3) Ser loss assigns
probability mass of 0.48 to 0.52 to both permuta- tions with a probability of 21 %.
The results are consistent with our claims: SetRNN loss assigns much more probability density to single permutation, and Uniform loss produces more uniform distribution. In contrast, Set loss relaxes the uniformity of Uniform loss.

---

Dataset
#Samples
#Structured Objects
#Object Types
D20a 14res train
1,300
2,145
3
D 20a
14res dev
323
524
3
20
14res test
496
862
3
D 20a
14lap train
920
1,265
3
D 20a
14lap dev
228
337
3
D 20a
14lap test
339
490
3
D
20a
15res train
593
923
3
D 20a
15res dev
148
238
3
D
20a
15res test
318
455
3
D 20a
16res train
842
1,289
3
D 20a
16res dev
210
316
3
D 20a
16res test
320
465
3
D 200 14res train
1,266
2,338
3
D206 14res dev
310
577
3
200 14res lest
492
994
3
200 14lap train
906
1,460
3
200 14lap dev
219
346
3
220 14lap test
328
543
3
200 15res train
605
1,013
3
200 15res dev
148
249
3
200 15res test
322
485
3
D20 16res train
857
1,394
3
200 16res dev
210
339
3
D20 16res test
326
514
3
ACE 2005 train (EE)
17,172
4,202
33
ACE 2005 dev (EE)
923
450
33
ACE 2005 test (EE)
832
403
33
ACE 2005 train (RE)
10,051
4,788
6
ACE 2005 dev (RE)
2,420
1,131
6
ACE 2005 test (RE)
2,050
1,151
6
CADEC
7,597
6,318
1
CoNLL04 train
922
1,290
5
CoNLL004 dev
231
343
5
CoNLL04 test
288
422
5
NYT train
56,196
94,222
24
NYT dev
5,000
8,489
24
NYT test
5,000
8,616
24
DocRED train
3,008
37,486
96
DocRED dev
300
3,678
96
DocRED test
700
8,787
96
SciERC train
1,861
3,219
7
SciERC dev
275
455
7
SciERC test
551
974
7
ADE (10 fold)
4.272
6,821
1
Table 4: Datasets statistics.


Reviews:
[{"review_id": "EtVihMdm6U", "reviewer": "Reviewer_sCUS", "summary": "", "questions": "Why do Uniform and SetRNN losses perform worse than the base seq2seq model?\nDid you do significant tests or average multiple runs?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude for your observations and comments, which have been influential in enhancing the quality of our paper.\n \n**Weakness-1: The innovativeness is somewhat lacking.**\n\nResponse-1: We sincerely appreciate your constructive comments concerning our work's perceived lack of innovation. We are grateful for this opportunity to underscore its merits.\n\nWe would like to highlight that our core contribution is developing a simple but effective approach to mitigating order bias in existing generative information extraction methods. While this approach may seem simplistic at first glance, its true value lies in its ability to integrate with current models easily. This feature alone enhances its innovative appeal and has been acknowledged by two other reviewers.\n\nMoreover, our approach is not limited to the specific task at hand; it also shows promise for adaptation in a wide range of applications where order bias presents challenges or where task objectives involve unordered sets. Such a broad applicability amplifies the impact and relevance of our contributions.\n\nWe hope this clarifies the innovative aspects of our work.\n\n**Weakness-2: The paper lacks an analysis and comparison of reinforcement learning methods.**\n\nResponse-2: We are grateful for your insightful observation and constructive suggestions. We agree that incorporating an analysis and comparing reinforcement learning methods would significantly enhance our work. Consequently, we are committed to enriching our manuscript by adding this analysis and comparison. Below, we outline some initial experimental data along with corresponding analytical insights.\n\n| Method                    | 14res  | 14lap | 15res | 16res |\n| --------------------------| ------ | ----- | ----- | ----- |\n| Seq2Seq                   | 65.2   | 58.6  | 59.2  | 67.6  |\n| Seq2Set [1]               | 64.5   | 55.2  | 57.8  | 64.3  |\n| Seq2Set (reward-shaping)  | 68.8   | 57.4  | 59.6  | 68.1  |\n| **Set (Ours)**         | 71.7   | 58.7  | 62.2  | 70.6  |\n\nSeq2Set [1] uses order-invariant reinforcement learning to mitigate the order bias issue in multi-label generation tasks. However, when applied to generative information extraction, our results indicate that Seq2Set underperforms. We speculate that rewards in generative information extraction are more sparse and have a larger variance. Moreover, even after leveraging reward-shaping techniques to improve Seq2Set, the gains in performance remain marginal compared to our approach.\n\n\n\n**Q1: What leads to the Uniform and SetRNN losses underperforming?**\n\nA1: We are glad you highlighted this. A detailed analysis can be found in Appendix B, which we will supplement with additional experimental data. In essence, both Uniform loss and SetRNN loss tend to generate pathological probability distributions over the chosen permutations, which contradicts the intrinsic nature of generative information extraction tasks and set learning. \n\nSpecifically, due to the lack of regularization on probability distributions over permutations, SetRNN loss causes the model to take shortcuts to assign most of the probabilities to single permutations, which is contrary to the essence of set learning: learning multiple permutations to make the model generalize between permutations. Conversely, Uniform loss over-penalizes any non-uniform distribution, which is over-strict for the Seq2Seq model, which may have inherent order bias. These contradictions might primarily account for their poor results. \n\n**Q2: Are there significant tests and multiple run averages?**\n\nA2: All of our experiments were repeated four times, and average results were reported. In addition, we test the significance of our method against the base Seq2Seq model and the variants of our method, and the results show that our method significantly outperformed the others (p-value < 0.0001). We will integrate this information into our revised manuscript for better comprehensibility.\n\nReferences:\n\n[1] Yang, Pengcheng, et al. \"A deep reinforced sequence-to-set model for multi-label classification.\" Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method description is clear， and the experiments are sufficient.\n2. Their analysis of the proposed method is detailed, and they experiment with various loss functions and permutation sampling.", "reasons_to_reject": "1. I think innovativeness is somewhat lacking in their approach. Their method only attempts a slight refinement of the previously proposed set-learning loss function.\n\n2. The order bias of the generative model has been explored in previous work using reinforcement learning in other tasks, I think these work should be analyzed and compared. \n\n3. I think it is interesting to explore the problem of order bias when generative targets are “sets”. So it will be more convincing if the paper can find more advanced strategies and be extended to a long paper.", "questions_to_authors": "Why do Uniform and SetRNN losses perform worse than the base seq2seq model?\nDid you do significant tests or average multiple runs?", "paper_topic_and_main_contributions": "To tackle the potential order bias resulting from a predefined fixed target order in the previous sequence-to-sequence model of information extraction, this paper delves into the set learning approach. It introduces a novel loss function tailored for the generative information extraction task. Notably, their method is task-agnostic, and they conduct extensive experiments across various information extraction tasks and datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AOPTPsogLN", "reviewer": "Reviewer_rpsn", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your insightful feedback and recommendations, which have been invaluable in refining our paper.\n \n**Weakness-1: The computations may be expensive for long text.**\n\nResponse-1: Thank you for bringing this up. Indeed, our method does lead to an increase in computational cost, which is an inherent limitation of our approach. We hope further work can address this limitation.\n \nFor texts with a larger number of tuples, while using more permutations in principle can better model the probability of the set, sampling a limited number of permutations can still offer significant benefits. For instance, with the DocRED dataset, where an individual sample contains an average of 12 tuples, we observed a 1.1 F1 score improvement even when sampling only four permutations.\n \n**Weakness-2: The example in the introduction is not clear.**\n\nResponse-2: Thank you for highlighting this; it is instrumental in enhancing the presentation quality of our paper. We will replace it with an appropriate example to illustrate the order bias issue.\n\n**Typos Grammar Style And Presentation Improvements.**\n\nThank you for pointing out these issues. We will make corrections and conduct further checks to minimize such errors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Well-motivated and clear writing\n2. A novel method that considers order bias for the generative IE. I like the way that the authors highlight the issue\n3. Extensive experiments on 9 datasets across 4 tasks. ", "reasons_to_reject": "1. the computations of permutation sampling may be expensive for long text, which involves more entities and relations.\n2. the example in the introduction does not clearly demonstrate the order bias issue. I suggest the authors select another example for such an illustration.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies the order bias problem of the sequence-to-sequence (Seq2Seq) model in Information Extraction (IE). The authors propose a novel set learning method to address this potential bias. It considers multiple possible orders to approximate the probability of structured objects set. Specifically, the authors design a loss function and propose a customized sampling strategy for reducing computational costs.  Experiments show the effectiveness of the proposed method.\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "1. \"four 4\" --> \"4\" in line 140\n2. Fig.2 is not mentioned in the main body.\n", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}, {"review_id": "hs7a6T3rlo", "reviewer": "Reviewer_wMdM", "summary": "", "questions": "Please see my comment in the previous \"Reasons to Reject\" section. ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and suggestions. They are instrumental in enhancing our paper.\n\n**Weakness-1: Lack of some comparative analysis.**\n\nResponse-1: Thank you kindly for bringing this matter to our attention. We agree that a more thorough comparison could provide a stronger argument for the effectiveness of our work. We will refine our paper by adding comparisons and analysis.\n\nPresently, there are two prevalent strategies to tackle order bias in generative methods:\n\n1.\tThe first is modifying the model structure to generate multiple results in parallel, represented by Sui, Dianbo, et al. [1] and Ye, Jiacheng, et al. [2].\n\n2.\tThe other one is modifying the training objective of the generative model, represented by Seq2Set [3].\n\nThe first approach is difficult to integrate directly with existing generative information extraction methods. We add the second method for comparison.\n \nThe table below shows that Seq2Set is underperformed compared to the base Seq2Seq model. The sparse and high variance rewards in generative information extraction tasks may cause this. Even though we improve it with reward shaping, it performs much worse than our method.\n\n| Method                    | 14res  | 14lap | 15res | 16res |\n| --------------------------| ------ | ----- | ----- | ----- |\n| Seq2Seq                   | 65.2   | 58.6  | 59.2  | 67.6  |\n| Seq2Set [1]               | 64.5   | 55.2  | 57.8  | 64.3  |\n| Seq2Set (reward-shaping)  | 68.8   | 57.4  | 59.6  | 68.1  |\n| **Set (Ours)**         | 71.7   | 58.7  | 62.2  | 70.6  |\n\n**Weakness-2: Appendix B should be supplemented with experiments and shifted to the main paper.**\n\nResponse-2: We are very grateful for such insightful suggestions. We believe this will deepen the discussion and make our paper more engaging. We will move Appendix B to the main paper and add experiments to validate the theoretical analysis.\n \nIn addition, we design and implement experiments to corroborate the arguments in Appendix B. Specifically, we set the number of permutations to 2, optimize the model to convergence using different losses, and then record and analyze the probability distributions generated by the different losses. Our main findings are as follows:\n\n1.\tSetRNN loss assigns more than 0.7 probability mass to a single permutation with a 97% probability, while the other two losses have a probability of 0%.\n\n2.\tUniform loss assigns a probability mass of 0.48-0.52 to both permutations with a probability of 26%.\n\n3.\tSet loss assigns a probability mass of 0.48-0.52 to both permutations with a probability of 21%.\n \nThe results are consistent with our claims in Appendix B: SetRNN loss assigns much more probability density to a single permutation, and Uniform loss produces a more uniform distribution, while Set loss relaxes the uniformity of Uniform loss. We will incorporate the complete experiment into our refined paper.\n\nReferences:\n\n[1] Sui, Dianbo, et al. \"Set generation networks for end-to-end knowledge base population.\" Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.\n\n[2] Ye, Jiacheng, et al. \"One2Set: Generating Diverse Keyphrases as a Set.\" Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.\n\n[3] Yang, Pengcheng, et al. \"A deep reinforced sequence-to-set model for multi-label classification.\" Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel Solution: The paper proposes a unique 'set learning' solution to the problem. This solution, which incorporates multiple permutations of structured objects, innovatively addresses the problem without necessitating changes to existing model structures.\n2. Experimental Validation: The effectiveness of the proposed approach is empirically validated using various tasks and datasets, showcasing its broad applicability and robustness.\n3. Framework for Further Research: The proposed set learning approach could inspire new research not only in IE but also in other NLP tasks where order bias might be a concern. The paper could also serve as a basis for further explorations into other potential improvements in set learning and its applications in NLP.", "reasons_to_reject": "1. Lack of Comparative Analysis: In the experiments, the paper does not include a comparison with other existing methods for dealing with order bias in IE tasks, but only compares with the baseline seq2seq approach and some variations to the proposed method. A more thorough comparison could provide a stronger argument for the effectiveness of their approach. If the results are not comparable, the author should elaborate on the reasons.\n2. Shifting parts of Appendix B into the main paper would add depth to the discussion, particularly on why the proposed loss function outperforms the variations. Adding experimental results to validate the claims about the Uniform loss, SetRNN loss, and the proposed Set loss would also strengthen the paper. Specifically, for the Uniform loss, an experiment demonstrating that it indeed results in a uniform distribution over permutations would validate the authors' claim and offer a concrete understanding of the Uniform loss's behavior. Similarly, for SetRNN loss, showing experimentally that the model tends to assign most of the probability density to a single permutation would provide clear evidence supporting their discussion.", "questions_to_authors": "Please see my comment in the previous \"Reasons to Reject\" section. ", "paper_topic_and_main_contributions": "This paper tackles the problem of order bias in Information Extraction (IE) frameworks using sequence-to-sequence (Seq2Seq) models. Order bias arises due to the processing of structured objects in a predefined order. To address this, the paper introduces a set learning approach that considers multiple permutations of structured objects to approximate the set probability, reducing order bias. The main contributions are:\n1. Highlighting the issue of order bias in Seq2Seq-based generative IE.\n2. Proposing a novel set learning approach that can be integrated into existing frameworks without modifications.\n3. Validating the approach with experiments, showing consistent improvements across various IE tasks and datasets.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel 'set learning' approach to address order bias in Seq2Seq models for IE. Experiments across various IE tasks and datasets demonstrates the effectiveness and applicability of the proposed approach. The paper has the potential to inspire further research in IE and related NLP tasks. However, a more comprehensive comparative analysis with existing order bias mitigation methods is needed to strengthen the paper's claims. Additional experimental evidence is recommended to validate claims related to the proposed Set loss and various other losses. Overall, the paper shows promise but should address these areas in the next version.
Decision: Accept-Main