Adaptive Hinge Balance Loss for Document-Level Relation Extraction
Authors: Jize Wang
Xinyi Le
Xiaodi Peng
Cailian Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: document-level relation extraction
multi-label classification
balancing methods
loss function design

Abstract: Document-Level Relation Extraction aims at predicting relations between entities from multiple sentences. A common practice is to select multi-label classification thresholds to decide whether a relation exists between an entity pair. However, in the document-level task, most entity pairs do not express any relations, resulting in a highly imbalanced distribution between positive and negative classes. We argue that the imbalance problem affects threshold selection and may lead to incorrect "no-relation" predictions. In this paper, we propose to down-weight the easy negatives by utilizing a distance between the classification threshold and the predicted score of each relation. Our novel Adaptive Hinge Balance Loss measures the difficulty of each relation class with the distance, putting more focus on hard, misclassified relations, i.e. the minority positive relations. Experiment results on Re-DocRED demonstrate the superiority of our approach over other balancing methods. Source codes are available at https://github.com/Jize-W/HingeABL.

Adaptive Hinge Balance Loss for Document-Level Relation Extraction
Jize Wang', Xinyi Le'; Xiaodi Peng' and Cailian Chen'
'Deparmennt of Automation, Shanghai Jiao Tong University Inspur Genersoft Co., Ltd.
(jizewang2000, lexinyi, cail sailianchen)ssjtu.. edu. pengxdeinspur.com com
Abstract
Text: Ross Patterson Alger (August 20, 1920
January 16, 1992) was a politician in the Canadian province Alberta, ... After the war, he received an MBA from the University of Toronto. He settled in Calgary and started career in accounting...
Document-Level Relation Extraction aims at predicting relations between entities from mul- tiple sentences. A common practice is to select multi-label classification thresholds to decide whether relation exists between an entity pair. However, in the document-lewe task, most en- tity pairs do not express any relations, resulting in highly imbalanced distribution between positive and negative classes. We argue that the imbalance problem affects threshold selec- tion and may lead to incorrect "no-relation" predictions. this paper, we propose to down- weight the easy negatives by utilizing dis- tance between the classification threshold and the predicted score of each relation. Our novel Adaptive Hinge Balance Loss measures the dif- ficulty of each relation class with the distance, putting more focus on hard, misclassified rela- tions, i.e. the minority positive relations. Ex- periment results on Re-DocRED demonstrate the superiority of our approach over other bal- ancing methods. Source codes are available at https://g/ /gi thub. com/ Wwii-///ieeeeBBBBB
Subject: University Toronto
Object: Canadian
Relation: country, located in
(a) A sample document in Re Re-DoDDED dataset.
existent relation
non existent relation
0.39
-5.38
-4.89
-4.30
.3.44
(b) False negative prediction with correct label ranking.
THRESHOLD
country
marformar
dee beth
date death
Cast mommtor
located
(c) Correct prediction after utilizing adaptive hinge balance loss.
Figure 1: Illustration on multi-label classification in document-level relation extraction. (a) There are two relations existing between University of Toronto and Canadian. (b) The entity pair in (a) is incor- rectly predicted as "no-relation". Scores of existent relations (country, located in) are lower than the thresh- old (11.16), but significantly higher than all non- non-existent relations. (c) After utilizing adaptive hinge balance loss, the threshold is reduced to an appropriate value.
1 Introduction
Document-Level Relation Extraction (RE) plays an important role in NLP applications such as knowl- edge graph construction. It aims at predicting re- lations between entities from multiple sentences. As illustrated in Figure la, entity pair may have zero, one, multiple relations, document-level RE is a multi-label classification task. To solve this, a common practice is to adaptively select thresholds for multi-label classification (Zhou et al., 2021). For a correct prediction, the confidence scores of existent relations should be higher than the threshold, and conversely, those of non-existent relations should be lower.
increases quadratically with the number of entities. Thus, compared with the sentence-level counter- part, there are far more entity pairs to be classi- fied in document-level RE, and most entity pairs express no relation. For examplc, in the document- level RE dataset, Re-DocRED (Tan et al., 2022b), 94% of the entity pairs express no relation.
However, there is significant imbalance prob- lem between positive and negative classes in document-level RE. The number of entity pairs
The issue of class imbalance may lead to more incorrect "no-relation" predictions. In our paper, we mainly consider the "positive/negative imbal- ance", rather than the imbalance between different
*Corresponding author.

---

types of "positive relations". The positive/negative imbalance tends to drag the threshold towards the large class, that is, the "no-relation" class. We discover that 78.2% % of incorrect "no-relation" predictions have correct label ranking, but the confidence score is lower than the threshold, which is shown in Figure 1b. In other words, the model has enough confidence in the existent relations, but it makes "no-relation" prediction due to the un- necessarily high threshold.
subject and object entity through:
2g Zol = Rep( D, ess
(1)
where 2s and % are the representation of the sub- ject and object entity. Rep is representation mod- ule.
The score of relation is defined as Sy, which can be computed via the subject and object entity representation using a bilinear classifier:
Based on this intuitive finding, we aim to ad- dress this "incorrect predictions with correct la- bel ranking" phenomenon to improve the accuracy. We believe that overtraining on well-classified non- existent relations may lead to unnecessarily high thresholds. Therefore, we propose to adaptively select thresholds, and then down-weight the rela- tions that are far from the decision boundary using Hinge Weighting. Our contributions are three-fold:
ST z W -0o br,
(2)
where W, E Rddd x b, E are model parameters.
2.2 Adaptive Thresholding Loss
Adaptive Thresholding Loss (ATL) (Zhou et al.. 2021) is the most widely used loss function transformer-based document-level relation extrac- tion methods. It enables the model to choose multi- label classification thresholds, thereby achieving superior results when compared to the global threshold of BCE loss (Bengio al., 2013).
We design a general pipeline termed Separate Adaptive Thresholding, to adaptively select thresholds for multi-label classification.
In ATL,, the labels of entity pair = (es, Co) are divided into two subsets: positive classes Pr and negative classes No, where Py: S R denotes the relations that exist between T. and My S R denotes the relations that do not exist between the entities. ATL introduces an additional threshold class TH. If entity pair is correctly classified, the scores of Pr should be higher than TH while those of Ny should be lower. ATL comprises of two parts:
We propose a novel Adaptive Hinge Balance Loss, tackling the imbalance problem of pos- itive and negative classes in document-level RE
Among all the existing balancing methods, our method achieves the highest Fl score on the common dataset Re-DocRED.
2 Preliminary
exp(sr)
The task of document-level relation extraction is concerned with the prediction of relation types be- tween subject and object entities in given docu- ment. We will first introduce the formulation of this task, and then discuss the commonly used ATL method.
C - Σ log
ErPEPUTTH E exp(s,)
rEPr
(3)
exp(stn)
L2 -log
(4)
EEENN exp(sp)
C = L1 f2.
(5)
2.1 Problem Formulation
2.3 An Empirical Analysis of ATL
Given a document that contains a set of entities [e the task of document-level relation extrac- tion is to predict the relation types between the entity pairs (Ce, Co),.o (1f.. where e, and represent the subject entity and the object en- tity, respectively. The set of relations is defined as R U (NA), where R. is a of pre-defined rela- tions and NA stands for no relation between a pair of entities.
A preliminary analysis is conducted to investigate the cause of classification error in ATL, as shown in Table 1. All false predictions can be categorized into three patterns: FP, PN_CKK, and FN _RRK, which are illustrated in Figure 2. In particular, FN_ CN_C is the most dominant source of errors, which accounts for 78.2% of all false negative pre- dictions.
With the document D and an entity pair (Eg, eo) contained in it, we can get the representation of the
We notice that the number of relations in Af is significantly larger than that in Py, and therefore

---

# FP
# FN BRCK
# FN IRK
threshold
existent relation
existexttent relation
2859
3192
889
Table Number of three false patterns of ATL's predic- tions on Re-DocRED. Three patterns are illustrated in Figure 2.
R4
R4
(a) FP
(b) FN_ CLCKK
(c) FNJJRK
L2 has a much greater impact on the overall loss than L1 (see equations (3) and (4)). Due the dom- inance of C2, it can be rewritten as the following form:
Figure 2: Three false prediction patterns. (a) FP (False Positive): The entity pair is recognized lated, but not all relations are accurately recognized. (b) FN CRK (False Negative with Correct lahel RanKing): The entity pair is recognized as "no-relation", and all the existent relations have higher confidence scores than non-existent relations.
1
C2 -log 1+ I+ErEN exp(s, - 8TH)
(c) FN IRK (False Negative with Incorrect label RanKing): The entity pair is recog- nized as "no-relation", and not existent relations have higher confidence scores than non-existent relations.
C2 -> 0 when Sp' STH 00, which means 9TH > Sp'. This suggests ATL learns threshold 8Th well above the candidate score, which leads to an increase in the number of [N_C predictions.
A
3 Adaptive Hinge Balance Loss
Figure 3: Hinge weighting.
Based on the analysis above, we aim to maximize the distance between the decision boundary 8TI and the sample point Sy, 1' E R while simultane- ously down- weightingg the classes distant from the boundary. To this end, we propose our Adaptive Hinge Balance Loss.
score
decision boundary relation
well-classifed existent relation
m
Stu
m1
decision boundary of non-existent relation
well-classified non-existent relation
3.1 Separate Adaptive Thresholding
R3 R4
R6
relation
An ideal loss should maximize the distance from the decision boundary to the sample point. More- over, in the loss formulation, each relation class should be independent of the others to enable indi- vidual weighing of each class. Therefore, we pro- pose the Separate Adaptive Thresholding (SAT), which is formulated as:
Figure 4: The decision boundary and margin of hinge weighting. Positive relations whose scores are higher than 9TH + m and negative relations whose scores are lower than sTm m will not be punished.
3.2 Hinge Weighting
To down-weight the easy and well-classified lations, i.c. the majority negative relations. we propose Hinge Weighting inspired by hinge loss. (Hearst et al., 1998)
L
Σ log(a(-d)) dr)); FCR
Our Hinge Weighting is shown in Figure 3. Is formulated as:
8r - 9TH
r T TEPT PT 8r r Nr
dr
(8)
8TH
W/ max d,
(9)
where a is the sigmoid function, i.e. (xx) Tee We define 8T1 as the decision boundary. Then d, is the distance from the decision boundary to the score of relation с R. d, > 0 if a relation is correctly classified. L + 0 when d, + co.
where m is a constant. When the distance d, is larger than the relation is not penalized. Other- wise, it is penalized linearly with d,. Essentially, Hinge Weighting implics that we should avoid fo- cusing on the relationship with large dr. 2m is the maximum margin between positive and negative classes, which illustrated in Figure
The loss pushes d, to be large as possible. The score of each relation compared with the threshold separately. Thus we can assign different weights to different relations.
Note that our weighting mechanism down- weights the well classified samples to zero. Since

---

Model
F1
Fl with HingeABI.
Ign_F1
Ign_Fl with HingeABL.
ATLOP (Zhou al., 2021)
77.56
79.79 (+2.23)
76.82
78.82 (+2.00)
DocuNet (Zhang et al., 2021)
77.87
79.43 (+1.56)
77.26
78.39 (+1.1)
KD-DocRE (Tan et al., 2022a)
78.28
79.34 (+1.06)
77.60
78.26 (+0.66)
Table 2: Performance of adaptive hinge balance loss with different backbones.
the majority of well classified classes are negative, HingeABL achieves the effect of down-weighting the majority negative relations.
Loss Function
F1
Ign F1
ATL (Zhou al., 2021)
73.29
72.46
Balanced Softmax (Zhang et al., 2021)
73.68
72.85
AML (Wei and Li, 2022)
72.60
71.78
3.3 Loss Definition
AFL (Tan et al., 2022a)
74.15
73.20
Combining separate adaptive thresholding and hinge weighting, we obtain the adaptive hinge bal- ance loss (HingeABL):
SAT
73.46
72.61
MeanSAT HingeABL
74.68
72.90
75.15
73.84
L C=
Σ ENER wy
EyeR lo)()(-
Wr
(10)
Table 3: Comparison with other balancing methods.
Both AML and HingeABL. are margin-based loss functions.
where a is the sigmoid function and wy formu- lated as Equation (9). The hinge weights are nor- malized among all relations. Our adaptive hinge balance loss simultancously maximizes the dis- tance between the decision boundary and the sam- ple point and woighteights easy classes that are far from it. This helps prevent over-fitting well- classified relations.
To illustrate the effectiveness of hinge weight- ing, we implement an alternative weighted loss called MeanSAT by weighting positive and nega- tive classes of SAT by the inverse of their number. Its formulation is Appendix D.
The results are shown in Table 3. HingeABL achieves the highest F1 and Ign F1 of 75.15 and 73.84 among all balancing methods. We observe substantial increase in performance by implement- ing two weighting methods on the SAT. Our ex- periments indicate that hinge weighting surpasses constant weighting with MeanS which demon- strates the superiority of HingeABL.
4 Experiments
4.1 Setup
We conduct experiments on Re-DocRED (Tan et al., 2022b), the largest and well-labeled dataset for document-level RE. We use Fl and Ign_Fl as the metrics. lgn_Fl is measured by removing the rela- tions existing in the training set from the dev/test sets. More details about statistics and implementa- tion are provided in Appendix A and B. Note that we use micro Fl here in order maintain consis- tency with previous methods. However, macro F1 is more suitable to illustrate whether the proposed method can perform better on minority classes. Re- sults evaluated under macro Fl are provided in Appendix C.
Besides, we compare the two margin- based loss functions, AML and HingeABL, through mathe- matical analysis, provided in Appendix E. We find that AML penalizes the misclassified samples lin- early with the distance, while HingeABL penalizes the misclassified samples nonlinearly with the dis- tance. The nonlinear function is strictly convex. which benefits optimization.
Different document-level RE models. To test the generality of our approach, we select three commonly used transformer- based methods for document-level relation extraction and replace their loss functions with our adaptive hinge balance loss. Among the three original base methods, AT- LOP employs ATL loss, DocuNet employs Bal- anced Softmax loss, and KD-DocRE employs AFL loss. Both Balanced Softmax and AFL are the im- provements of ATL. All methods use RoBERTa (Zhuang et al., 2021) as their encoder. Table 2 shows the results, all of which demonstrate consis-
4.2 Results
Different balancing methods. To compare dif- ferent balancing methods, we use ATLOP (Zhou et al., 2021) the representation module and BERT hase (Devlin et al., 2019) the encoder of it. We also compare our method with three other ap- proaches: Balanced Softmax (Zhang et al., 2021),
AML (Adaptive Margin Loss) (Wei and Li, 2022), and AFL (Adaptive Focal Loss) (Tan et al., 2022a).

---

FN_CRK
Acknowledgements
Loss Function
F((T+F)
FN/F
IFN
This work was supported by the National Key Research and Development Program of China (2021YFB17) (716000)) and the National Natural Sci- ence Foundation of China (No.62176152).
ATL
3.59%
58.80%
78.22%
Balanced Softmax
4.84%
51.16%
79.95%
AML
3.54%
65.30%
57.14%
AFL
3.59%
52.95%
75.98%
HingeABL 3.49%
51.11%
43.84%
References
Table 4: Statistics of prediction patterns for different loss functions.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence. 35(8):1798-18))
tent performance gains with the use of HingeABL. This affirms the generalizability of our approach. Note that HingeABL's improvement seems to be less significant when the base method is more pow- erful. This is a natural result because better base methods employ better loss functions. Replacing a better loss function with HingeABL results in a smaller improvement.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisticx: Humun Lunguage Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Prediction statistics. To verify whether our model solves the problem of high thresholds, we count the number of prediction patterns from Figure and present the results in Table 4. Our analysis reveals that the proportion of FN and FN CR_C has decreased, indicating that the issue has been resolved. An example of prediction results before and after applying HingeABL shown in Figure lb and Ic. While one might assume that lower- ing the threshold would lead to more false positive predictions, we observe that the total proportion of false predictions actually decreases. This sug- gests that HingeABL achieves a good balance in its threshold selection.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter No- ordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Ac- curate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 77770066777
Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. 1998. Support vec- tor machines. IEEE Intelligent Systems and their applications, 13(4):18-28.
Quzhe Huang, Shibo Hao, Yuan Ye, Shengqi Zhu, Yansong Feng, and Dongyan Zhao. 2022. Does recommend-revise produce reliable annotations? an analysis missing instances in DocRED. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics ( (Volume I: Long Papers). pages 6241-6252, Dublin, Ireland. Associa- tion for Computational Linguistics.
5 Conclusion
We propose a novel Adaptive Hinge Balance Loss for document-level relation extraction to tackle the imbalance problem of positive and negative classes. Experimental results show our approach outper- forms existing methods. Since our loss is model- independent, it has potential applicability other multi-label classification scenarios.
Ilya Loshchilov and Frank Hutter, 2019. Decoupled weight decay regularization. ICLR.
Paulius Micikevicius. Sharan Narang, Jonah Alben, Gre- gory Diamos, Erich Elsen, David Garcia, Boris Gins- burg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2018. Mixed precision training. ICLR.
Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. 2022a. Document-leve relation extraction with adaptive focal loss and knowledge distillation. In Findings of the Association for Computational Lin- guisticx; ACL 2022, pages 1672-1681, Dublin, Ire- land. Association for Computational Linguistics.
Limitations
Compared with classifying entity pair known with relation, accurately determining whether re- lation exists between an entity pair is a more chal- lenging task. Despite attempts improve accuracy through better thresholding methods, the results are still far from ideal.
Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng, and Sharifah Mahani Aljunied. 2022b. Revisiting docred- addressing false negative problem relation ex- traction. In Proceedings of the 2022 Conference on

---

Empirical Methods Natural Language Processing, pages
# Relations
96
Avg. # Words
198.4
Avg. # Entities
19.4
Ying Wei and Qi Li. 2022. Sagdre: Sequence-aware graph-based document-level relation extraction with adaptive margin loss. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discov- ery and Data Mining, KDD '22, page 2000-2008, New York, NY, USA. Association for Computing Machinery.
Avg. # Entity Pairs
391.0
NA
94%
Table 5: Statistics on whole set of Re-DocRED.
Train
Dev
Test
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Seao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of- tatte-of-thh-arttt natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
# Documents
3053
500
500
Avg. # Entities
19.4
19.4
19.6
Avg. # Triples
28.1
34.6
34.9
Avg. # Sentences
7.9
8.2
7.9
Table 6: Statistics on different train/dev/test dataset of Re-DocRED.
B Implementation Details
All experiments are implemented based on Hug- ging Face's Transformers (Wolf et al., 2020). In the experiment comparing different balancing meth- ods, we use BERT (Devlin et al., 2019) as the encoder of ATLOP. In the experiment of comparing different RE models, we use RoBERTaaage (Devlin et al., 2019) as the encoder of these RE models, for the sake of comparison the benchmark.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-lewe level relation extraction dataset. In Pro- ceedings of the 57th Annual Meeting of the Associa- tion for Computational Linguistics, pages 764-777, Florence, Italy, Association for Computational Lin- guistics.
Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, Hua- jun Chen, and Hangzhou Innovation Center. 2021. Document-level relation extraction as semantic seg- mentation.
We select the margin of HingeABL as m 5 when conducting experiments. We use mixed- precision training (Micikevicius et al., 2018) based on the Py PyTorch amp library'. The models are opti- mized with AdamW W (Loshchilov and Hutter, 2019) with a linear warmup (Goyal et al., 2017) for the first 6% steps followed by linear decay to 0. The learning rate is 5e-5 for models with BERT the encoder and 3e-5 for models with RoBERTa the encoder. The train batch size is and the test batch size is 8. We train 30 epochs for each model. For each experiment, we run 5 different seeds (1, 5, 42, 66, 233) and report the average score. All models are trained with 1 Tesla A800 GPU.
Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. 2021, Document-level relation extraction with adaptive thresholding and localized context pool- ing. In Proceedings of the AAAI conference on artifi- cial intelligence, volume 35, pages 14612-14620.
Liu Zhuang, Wayne, Shi Ya, and Zhao Jun. 2021. robustly optimized BERT pre- preitrainin approach with post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218-1227, Huhhot, China. Chinese Informa- tion Processing Society of China.
C Comparison results under macro F1
A Re-DocRED Statistics
The comparison results among different balancing methods under macro Fl and macro Ign F1 are shown in Table 7. Our proposed HingeABL. still achieves the highest score under macro Fl.
Re-DocRED is a more reliable benchmark in document- relation extraction. It is a revised version of DocRED (Yao et al., 2019), whose anno- tations are pointed out to be incomplete by recent works ((Huagg al., 2022; Tan al., 2022a)). Re- DocRED contains 96 relations. Each document has an average of 391 entity pairs, among which 94% contains no relation. The detailed statistics are shown in Table 5 and 6.
D Formulation of MeanSAT
MeanSAT weights positive and negative classes of SAT by the inverse of their number. It is formulated
hatp://iyyacc..ngooooossallaam.m

---

Loss Function
F1
lgn_F1
the distance. (Note: A sample that is not well classified means G. -d, mn. A sample that misclassified means C. = -d, > 0.)
ATL (Zhou et al., 2021)
59.39
56.57
Balanced Softmax (Zhang al., 2021)
60.67
57.89
AML (Wei and Li, 2022)
58.65
55.81
Analysi 2: For a sample that is not well clas- sified, HingeABL. BL is a strictly convex function
AFL (Tan et al., 2022a)
61.48
58.66
SAT
60.23
57.41
with respect to the distance.
MeanSAT
63.34
60.91
HingeABL is defined as:
HingeABL.
64.13
61.34
P=
Ex'ER Wip' FER
E
log(o(-
(15)
Table 7: Comparison with other balancing methods under macro F1 and macro Fl.
max(O, m d-)
Σ nER
EVER max m drr) log 11 11+ (1
as:
(16)
The denominator EvER (0, m- d,¹) is nor- malization factor, which discard for ease of analysis.
C
1 Σ log(o(-d.)) N TEP
Σ
log(o(-d.))
Nn TENV
(11)
For class r, a sample is well classified, d, mi, C. =
where Np and N, are the number of positive and negative classes for the entity pair.
If a sample not well classified, d, < m,
Mathematical analysis of Adaptive Margin Loss and HingeABL
C, = -mm dr Mog
(17)
Ite-d.
In addition experiments, also compare the two margin-based losses, Adaptive Margin Loss (AML) and HingeABL, from a mathematical anal- ysis perspective.
= -(mm + ro)
1
(18)
ac.
1
m m+ c
log
(19)
Oc,
[+t
+1+
8c.,
eer
e (m+
Analysis 1: For a sample that is not well clas- sified, the Adaptive Margin Loss is a linear func- tion with respect to the distance.
0c
c
(e + 1)2
(20)
This means HingeABL penalizes the samples that are not well classified nonlinearly with the distance. The nonlinear function is strictly convex. Analysis 3: Comparision between the Adap- tive Margin Loss and HingeABI..
The Adaptive Margin Loss is defined as:
C = Σ max(0, m - dr).
(12)
FER
For class r, C, m0 m mn-dyy -d). For a well-classified sample, d, 2 m. L, = For sample that is not well classified, d, < m, C, = m d,-
1. Similaritics.
Both the Adaptive Margin Loss and Hinge ABL are margin-based loss functions. They do not pun- ish a prediction if it is correct and "good enough" (rather than "perfect"), which is form of regular- ization to prevent overfitting.
In the second condition, we denote c, = d
-m. It measures the distance between a sample that is not well classified to the decision boundary. Note that call c, "distance" here, but it is not necessarily greater than zero. The smaller G, is, the better the sample is classified. This means we should give a larger punishment to larger cy. Then we have:
2. Differences.
For the wrong prediction part, they both give a penalty according to the distance c,.. The Adaptive Margin Loss gives a linear penalty, while Hinge- Aivg a strictly convex penalty, Compared to a linear penalty, strictly convex penalty has mainly two advantages: 1. When e, is larger, HingeABL gives larger penalty than the Adaptive Margin Loss. 2. Compared to linear functions, the nature of strictly convex functions makes the optimiza- tion more stable and more likely to converge to a globally optimal solution.
C, = m + Cr.
(13)
OCr 1.
(14)
Ocr
This means Adaptive Margin Loss penalizes the samples that are not well classified linearly with


Reviews:
[{"review_id": "2312P3PJ7M", "reviewer": "Reviewer_yRY8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: With this method, as the false negative predictions decrease, the false positive predictions increase.**\n\nThanks. This phenomenon does not only occur in our method, but in all the methods compared in the paper, as shown in the table below. For example, compared with ATL, AFL reduces #FN by 409, but increases #FP by 404. **All the methods have this limitation. HingeABL has the largest reduction (196) of total false predictions, i.e. #(FP+FN).** \n\n|Loss Function|F1|#FP|#FN|#(FP+FN)|\n|---|---|---|---|---|\n|ATL|73.29|2859|4081|6940|\n|Balanced Softmax|73.68|3299|4222|7521|\n|AFL|74.15|3263|3672|6935|\n|SAT|73.46|2910|4008|6918|\n|MeanSAT|74.68|5196|2488|7684|\n|**HingeABL**|**75.15**|3297|3447|**6744**|\n\n**Q2: The improvement of the performance is limited, in terms of F1 scores.**\n\nAs shown in the table in Q1, HingeABL achieves a performance gain of 1.86 points in F1 score, which is **the most significant compared with other balancing methods**. Besides, the gain of 1.86 points in F1 score is already **a very large performance improvement in the field of document-level relation extraction**. \n\nIn order to get a clear picture of how much performance improvement can be achieved by approaches in the field of document-level RE, here we list several widely implemented methods and their performance on the Re-DocRED dataset. \n\n|Method|F1|\n|---|---|\n|ATLOP (baseline)|77.56|\n|DocuNet|77.87 (+0.31)|\n|KD-DocRE|78.28 (+0.72)|\n|SSR-PU|78.86 (+1.30)|\n|ATLOP+HingeABL (Ours)|79.79 (**+2.23**)|\n\nFor example, KD-DocRE is an effective and popular method, which is the improvement of ATLOP. **It only achieves a performance gain of 0.72 F1 compared with ATLOP**. \n\nIn our paper, we observe that the \"positive/negative class imbalance\" problem accounts for the majority of prediction errors, which is ignored by previous methods. Based on the observation, we thoughtfully implement HingeABL through a combination of hinge loss and cross-entropy loss and **achieve a performance gain of 2.23 F1, which is significant and substantial**. HingeABL addresses the \"positive/negative class imbalance\" that has not been carefully considered by previous methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of using Separate Adaptive Thresholding combined with Hinge Weighting to modulate loss function is reasonable.\n2. The results suggest this method is effective to some extent.\n", "reasons_to_reject": "1. As the authors point out, with this method \"as the false negative predictions decrease, the false positive predictions increase\". So the improvement of the performance is limited, in terms of F1 scores. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes the Adaptive Hinge Balance Loss to address the imbalance problem in document-level Relation Extraction (RE) tasks. In document-level RE, most entity pairs do not hold any relations, leading to a highly imbalanced class distribution. To model the difficulty of classification, the authors use the Separate Adaptive Thresholding (SAT), measuring the distance between a classification threshold score and the predicted score of each relation. \nThe proposed Adaptive Hinge Balance Loss puts the \"Hinge weighting\" on the loss function, drawing more focus on hard, misclassified relations. The weight is a nonlinear function based on SAT.\nThe experiment results on the Re-DocRED dataset demonstrate the superiority of this approach over other balancing methods. Analysis also shows this method indeed reduces false negatives and false negatives with correct ranking of classes.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4KjwEAgHmg", "reviewer": "Reviewer_zSKD", "summary": "", "questions": "According to Table 2, HingeABL's improvement appears to be less significant when the base model is more powerful. This raises the question of whether a strong base method could potentially alleviate the threshold problem. Do you have further experiments to provide additional statistics on the FN-CRK in these three methods?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: HingeABL's improvement appears to be less significant when the base model is more powerful.**\n\nThanks. The seemingly less significant improvement is not relevant to more powerful models, but it is a nature result of a higher baseline with improved loss functions. Among the three base models, ATLOP employs ATL loss, DocuNet employs Balanced Softmax loss, and KD-DocRE employs AFL loss. Both Balanced Softmax and AFL are the improvements of ATL. Therefore, replacing the better loss functions with HingeABL results in a smaller improvement.\n\n**Q2: Whether a strong base method could potentially alleviate the threshold problem?**\n\nWe find that stronger base models do not necessarily mitigate the thresholding problem better. \nAccording to our experiment, KD-DocRE is stronger than DocuNet, but it has a higher number of FN_CRKs, and the good results may be due to better relation representation, better resolution of the relation type imbalance, etc. The optimization method they use does not primarily address the positive/negative imbalance.\nThe statistics are shown in the table below.\n\n||#FN_CRK|#FN_CRK with HingeABL|\n|---|---|---|\n|ATLOP|3273|1438|\n|DocuNet|1494|1418|\n|KD-DocRE|2688|1275|\n\n**Q3: Do you have further experiments to provide additional statistics on the FN-CRK in these three methods?**\n\nAccording to our experiment, no matter how strong or weak the underlying model is, replacing the loss function with HingeABL, the number of FN_CRKs all gets decreased for three base models (ATLOP, DocuNet, and KD-DocRE). The statistics are shown in the table in Q2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is strongly motivated\n2. The paper exhibits a well-organized structure, addressing the problem at hand, identifying its root causes, presenting an effective solution, and supporting it with experimental evidence to demonstrate its efficacy.\n3. HingeABL consistently demonstrates improvement over three methods.", "reasons_to_reject": "None. ", "questions_to_authors": "According to Table 2, HingeABL's improvement appears to be less significant when the base model is more powerful. This raises the question of whether a strong base method could potentially alleviate the threshold problem. Do you have further experiments to provide additional statistics on the FN-CRK in these three methods?", "paper_topic_and_main_contributions": "This paper proposes a novel approach called the Adaptive Hinge Balance Loss (HingeABL) for document-level relation extraction. They build upon the empirical observation that the previous method (ATL) suffers from errors primarily due to an inappropriate threshold classifying positive/negative classes. This issue arises from the class imbalance problem, where the negative class is significantly larger than the positive class, which has a larger impact on the objective function. To tackle this problem, the authors introduce hinge weighting to mitigate the impact of the dominant negative relations. By incorporating this hinge weighting mechanism, the impact of the negative class would be close to zero, addressing the threshold problem. Experimental results demonstrate that HingeABL significantly improves the F1 score for document-level relation extraction on three fundamental methods, and it successfully alleviates the threshold-related challenges.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xTwhbgY1y1", "reviewer": "Reviewer_cr9k", "summary": "", "questions": "The method did not seem to address the class imbalance issues. The imbalance issue means the number of samples for each class is not uniform, and some relations have very few samples. A common solution from the loss function perspective is to put higher weights on minority classes (a higher penalty for misclassifying a sample in a minority class). The proposed weighting mechanism doesn't seem to favor minority relations. \n\nThe evaluation metric is also not proper if the paper aims to address the imbalance issue. The F1 used by baseline methods is micro F1, giving equal importance to each sample regardless of the class it belongs to. So in the final score, the performance of the minority classes is hidden, and the performance of the majority classes is amplified. Macro F1 should be used instead to better illustrate if the proposed method can perform better on minority classes.\n\nIf the authors used the term \"imbalance\" to mean other issues, please clearly define it in the paper.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The class imbalance issues, and the \"imbalance\" definition.**\n\nIn our paper, we mainly consider the imbalance problems of the “positive/negative imbalance”, and ignore the imbalance between different types of “positive relations”. We find that, in our document-level task, the negative class is significantly larger than the positive class, which has a larger impact on the objective function. \nIn the revised paper, we will clarify the \"imbalance\" into “positive/negative imbalance”.\n\n**Q2: Weighting mechanism to favor minority relations.**\n\nRefer to Q1, because we mainly tackle the imbalance problems of the “positive/negative imbalance”, “minority relations” here mean “positive relations”. Our weighting mechanism uses HingeABL to down-weight the majority negative class, thus favoring the “minority positive relations”.\n\n(Specifically, our weighting mechanism down-weights the well classified samples to zero. Since the majority of well classified classes are negative, HingeABL achieves the effect of down-weighting the majority negative class.)\n\n**Q3: Comparison with SagDRE.**\n\nHere we compare our method with Adaptive Margin Loss proposed by SagDRE from both the experiment results and the mathematical analysis.\nIn our setting (ATLOP-BERT-base as the representation module, details in 4.2 “Different balancing methods” in our paper), our proposed HingeABL **still achieves the highest score** among all balancing methods. The revised comparison table is as follows. (Note: Evaluated under Micro F1. Scores under Macro F1 are shown in Q4.)\n\n|Loss Function|Micro F1|Micro Ign_F1|\n|---|---|---|\n|ATL|73.29|72.46|\n|Balanced Softmax|73.68|72.85|\n|AFL|74.15|73.20|\n|Adaptive Margin Loss (SagDRE) [1]|72.60|71.78|\n|SAT|73.46|72.61|\n|MeanSAT|74.68|72.90|\n|**HingeABL (Ours)**|**75.15**|**73.84**|\n\nBoth HingeABL and Adaptive Margin Loss are margin-based. Adaptive Margin Loss penalizes the misclassified samples linearly with the distance, while HingeABL in our paper penalizes the misclassified samples **nonlinearly** with the distance, using logsoftmax function. We claim that a suitable nonlinear penalty benefits the “imbalance reduction”. The mathematical analysis is as follows:\n\n**Analysis 1: For a misclassified sample, the Adaptive Margin Loss is a linear function with respect to the distance.**\n\nThe Adaptive Margin Loss is defined as:\n$$\n\\mathcal{L}=\\sum_{1\\leq i \\leq C}\\text{max}(0,\\alpha-t_i(z_i-z_s))\n$$\nwhere $t_i=1$ if $R_i\\in\\mathcal{P}$;  $t_i=-1$ if $R_i\\in\\mathcal{N}$.\n\nFor class $i$, \n$\\mathcal{L}_i=max(0,\\alpha-t_i(z_i-z_s))$.\n\nFor a well classified sample, $|z_i-z_s|\\geq \\alpha$, \n$\\mathcal{L}_i=0$.\n\nOtherwise, $|z_i-z_s|< \\alpha$, $\\mathcal{L}_i=\\alpha-t_i(z_i-z_s)$.\n\nIn the second condition, we denote \n$d_i=-t_i(z_i-z_s)>0$. It measures the distance between a misclassified sample to the decision boundary. Then we have:\n$$\\mathcal{L}_i=\\alpha+d_i$$\n$$\n\\frac{\\partial \\mathcal{L}_i}{\\partial d_i}=1\n$$\n\nThis means Adaptive Margin Loss penalizes the misclassified samples linearly with the distance.\n\n**Analysis 2: For a misclassified sample, HingeABL is a strictly convex function with respect to the distance.**\n\nFor ease of comparison, we use the same set of symbols as above for HingeABL (Equation(10) in our paper):\n$$\n\\mathcal{L}=-\\sum_{1\\leq i \\leq C}w_i\\text{log}(\\sigma(-t_i(z_i-z_s))$$\n$$=-\\sum_{1\\leq i \\leq C}\\frac{max(0,\\alpha-t_i(z_i-z_s))}{\\sum_{1\\leq i \\leq C}max(0,\\alpha-t_i(z_i-z_s))}\\text{log}\\left(\\frac{1}{1+e^{-t_i(z_i-z_s)}}\\right)\n$$\n\nThe denominator $\\sum_{1\\leq i \\leq C}max(0,\\alpha-t_i(z_i-z_s))$ is a normalization factor, which we discard for ease of analysis.\n\nFor class i, if a sample is well classified, $|z_i-z_s|\\geq \\alpha$, $\\mathcal{L}_i=0$.\n\nOtherwise, $|z_i-z_s|< \\alpha$, \n$$\\mathcal{L}_i=-(\\alpha-t_i(z_i-z_s))\\text{log}\\left(\\frac{1}{1+e^{-t_i(z_i-z_s)}}\\right)\n=-(\\alpha+d_i)\\text{log}\\left(\\frac{1}{1+e^{d_i}}\\right)$$\n\n$$\n\\frac{\\partial \\mathcal{L}_i}{\\partial d_i} = -\\text{log}\\left(\\frac{1}{1+e^{d_i}}\\right)+\\frac{\\alpha+d_i}{e^{-d_i}+1}\n$$\n\n$$\n\\frac{\\partial^2\\mathcal{L}_i}{\\partial d_i^2}=\\frac{e^{d_i}}{1+e^{d_i}}+\\frac{e^{-{d_i}}+1+e^{-{d_i}}(\\alpha+d_i)}{(e^{-d_i}+1)^2}>0\n$$\n\nThis means HingeABL penalizes the misclassified samples nonlinearly with the distance. The nonlinear function is strictly convex. \n\n**Analysis 3: Comparision between the Adaptive Margin Loss and HingeABL.**\n\n1.Similarities.\n\nBoth the Adaptive Margin Loss and HingeABL are margin-based loss functions. They do not punish a prediction if it is correct and \"good enough\" (rather than \"perfect\"), which is a form of regularization to prevent overfitting. \n\n2.Differences.\n\nFor the wrong prediction part, they both give penalty according to the distance $d_i$. The Adaptive Margin Loss gives a linear penalty, while HingeABL gives a strictly convex penalty. Compared to a linear penalty, a strictly convex penalty has mainly two advantages: \n\n(1) When $d_i$ is larger, HingeABL gives a larger penalty than the Adaptive Margin Loss. \n\n(2) Compared to linear functions, the nature of strictly convex functions makes the optimization more stable and more likely to converge to a globally optimal solution.\n\nWe will add the comparison of Adaptive Margin Loss in SagDRE to our paper. We believe that this kind of margin-based loss functions worths exploring in depth for more complex tasks.\n\n**Q4: The evaluation metric Macro F1.**\n\nOur proposed HingeABL **still achieves the highest score under Macro F1**. The results are shown in the table below. \n\n|Loss Function|Macro F1|Macro Ign_F1|\n|---|---|---|\n|ATL|59.39|56.57|\n|Balanced Softmax|60.67|57.89|\n|AFL|61.48|58.66|\n|Adaptive Margin Loss (SagDRE) [1]|58.65|55.81|\n|SAT|60.23|57.41|\n|MeanSAT|63.34|60.91|\n|**HingeABL (Ours)**|**64.13**|**61.34**|\n\n[1] Wei, Y., & Li, Q. (2022, August). SagDRE: Sequence-Aware Graph-Based Document-Level Relation Extraction with Adaptive Margin Loss. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 2000-2008).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The research tackles an interesting and challenging problem concerning class imbalance in document-level relation extraction, which is a critical issue. \n\nThe adaptive Hinge loss is well-motivated and thoughtfully implemented through a combination of hinge loss and cross-entropy loss. \n\nThe evaluation utilizes real-world large-scale data, enhancing the reliability and practicality of the results.", "reasons_to_reject": "The paper lacks a comparison with state-of-the-art models, particularly the one employing margin-based losses for document-level relation extraction [1]. Including such comparisons is crucial to understanding the performance of the proposed method relative to the existing techniques. \n\n[1] Wei, Y., & Li, Q. (2022, August). SagDRE: Sequence-Aware Graph-Based Document-Level Relation Extraction with Adaptive Margin Loss. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 2000-2008). \n\nThe method did not address the class imbalance issues", "questions_to_authors": "The method did not seem to address the class imbalance issues. The imbalance issue means the number of samples for each class is not uniform, and some relations have very few samples. A common solution from the loss function perspective is to put higher weights on minority classes (a higher penalty for misclassifying a sample in a minority class). The proposed weighting mechanism doesn't seem to favor minority relations. \n\nThe evaluation metric is also not proper if the paper aims to address the imbalance issue. The F1 used by baseline methods is micro F1, giving equal importance to each sample regardless of the class it belongs to. So in the final score, the performance of the minority classes is hidden, and the performance of the majority classes is amplified. Macro F1 should be used instead to better illustrate if the proposed method can perform better on minority classes.\n\nIf the authors used the term \"imbalance\" to mean other issues, please clearly define it in the paper.", "paper_topic_and_main_contributions": "This work presents an adaptive Hinge balance loss for document-level relation extractions, addressing the critical issue of class imbalance in imbalanced relation types. The proposed method combines hinge loss and cross-entropy loss and is evaluated on real-world large-scale data using the Re-DocRED dataset, demonstrating its effectiveness.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 39, \"squarely\", do you mean \"quadratically\"?"}, "extra_text": {"Missing_References": "Wei, Y., & Li, Q. (2022, August). SagDRE: Sequence-Aware Graph-Based Document-Level Relation Extraction with Adaptive Margin Loss. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 2000-2008)."}}]

Author rebuttal:

Meta-review: This paper focuses on the imbalance problem of N/A. The reviewers and the authors respectively give the detailed comments and responses. I suggest the authors fully consider the reviews and make precise revisions (especially the missing baseline).
Decision: Accept-Findings