RAPL: A Relation-Aware Prototype Learning Approach for Few-Shot Document-Level Relation Extraction
Authors: Shiao Meng
Xuming Hu
Aiwei Liu
Shuang Li
Fukun Ma
Yawen Yang
Lijie Wen
Venue: EMNLP 2023 Main (2023)
Keywords: Document-Level Relation Extraction
Few-Shot Learning
Metric-Based Meta-Learning
Relation-Aware Prototype Learning
TL;DR: We propose a relation-aware prototype learning method to strengthen the relational semantics of prototype representations for few-shot document-level relation extraction, outperforming the state-of-the-art approaches by a large margin.

Abstract: How to identify semantic relations among entities in a document when only a few labeled documents are available? Few-shot document-level relation extraction (FSDLRE) is crucial for addressing the pervasive data scarcity problem in real-world scenarios. Metric-based meta-learning is an effective framework widely adopted for FSDLRE, which constructs class prototypes for classification. However, existing works often struggle to obtain class prototypes with accurate relational semantics: 1) To build prototype for a target relation type, they aggregate the representations of all entity pairs holding that relation, while these entity pairs may also hold other relations, thus disturbing the prototype. 2) They use a set of generic NOTA (none-of-the-above) prototypes across all tasks, neglecting that the NOTA semantics differs in tasks with different target relation types. In this paper, we propose a relation-aware prototype learning method for FSDLRE to strengthen the relational semantics of prototype representations. By judiciously leveraging the relation descriptions and realistic NOTA instances as guidance, our method effectively refines the relation prototypes and generates task-specific NOTA prototypes. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches by average 2.61\% $F_1$ across various settings of two FSDLRE benchmarks.

RAPL: A Relation-Aware Prototype Learning Approach for Few-Shot Document-Level Relation Extraction
Shiao Meng, Xuming Hu, Aiwei Liu, Shu'ang Li, Fukun Ma, Yawen Yang, Lijie Wen' School of Software, Tsinghua University
{msa21 hxm19, 1iwaaw2,,000, (aliaaao,,1ssa11e wal.aasa,,,.. mfk22, y y J@mails. weingsusiiigaeeee edu. wenljetsinghua. edu. cn
Abstract
Target Relation Types: R1: P of Birth
R2: Work Locasinn
Place Death
How to identify semantic relations among tities in a document when only few labeled documents are available? Few-shot document- level relation extraction (FSDLRE) is crucial for addressing the pervasive data scarcity prob- lem real-world scenarios. Metrie-based meta-learning is an effective framework widely adopted for FSDLRE, which constructs class prototypes for classification. However, existing works often struggle to obtain class prototypes with accurate relational semantics: 1) build prototype for target relation type, they aggre- gate the representations of all entity pairs hold- ing that relation, while these entity pairs may also hold other relations, thus disturbing the prototype. 2) They use set generic NOTA (none-of-thee thbe---ve-ee- prototypes across all tasks, neglecting that the NOTA semantics differs in tasks with different target relation types. In this paper, propose relation-awan aware prototype learning method for FSDLRE to strengthen the relational semantics of prototype representa- tions. By judiciously leveraging the relation de- scriptions and realistic NOTA instances as guid- ance, our method effectively refines the relation prototypes and generates task- task-specific NOTA prototypes. Extensive experiments demonstrate that method outperforms state-of-th the art ap- proaches by average 2.61% % Fi across various settings two FSDLRE benchmarks.'
Support Document:
Query Document:
was age became a lecturer st mot
Grace Taylor fasouu loyeers Taylor bom and rased in Birsbane,
Business School,
entered majoring law. and finily potained Dectorate degree. After gradusting,
natire Londoncr. Smith, togesher London years.
University
Sydinyy legal
away years
community y
Relation Instances:
Relation Instances Outputs:
IThomas Miller, 81, Cardiff)
?
IThomas
Cardiff)
(Thomes Miller, London]
Gold Outputs:
(Karen Smith, R1, London)
Taylor, Brisbanel
Smith, R2,
(Grace Taylor, R2, Sydneyl
Figure I: Illustration of 1-Doc FSDLRE task. Entity mentions involved in relation instances are colored and bold. Other mentions are also bold clarity.
and many specific domains often lack annotated documents, making data scarcity a common issue in real-world scenarios. This motivates us to ex- plore few-shot document-level relation extraction (F'DLREE (Popovic and Färber, 2022). We illus- trate an example of the FSDLRE task under the 1- Doc setting in Figure 1, where only one annotated support document is given along with three target relation types: Place of Birth, Work Locat tion and Place of Death. The task is to predict all in- stances of the target relation types for pre-given entities in the query document, such (Grace Taylor, Place of Birth, Brisbane).
1 Introduction
Current efforts on FSDLRE (Popovic and Fürber, 2022) mainly adopt the popular metric-based meta- learning framework (Vinyals et al., 2016; Snell et al., 2017), which aims to learn metric space in which classification can be performed by comput- ing distances to prototype representations of each class. By training collection of sampled FS- DLRE tasks, the model learns general knowledge for FSDLRE, enabling rapid generalization new tasks with novel relation types.
Document-level relation extraction (DocRE) aims to identify the relations between each pair of enti- ties within document, which is crucial for extract- ing complex cross-sentence relations and imple- menting large-scale information extraction (Zhou et al., 2021; Xie et al., 2022; Wei and Li, 2022; Sun et al., 2023). However, the annotation of DocRE data is both time-consuming and labor-intensive,
*Corresponding author:
Idcally, within the metric- based paradigm, pro- totype representations should accurately capture
'The data and code are available at ht ht http:::///ittub....o com.
THU- BPH/RAPL.

---

the relational semantics of each category. However, this can be challenging for existing FSDLRE meth- ods: (1) Considering that an entity pair may express multiple relations in a document, a relation proto- type is obtained by aggregating the representations of entity pairs in the support set holding that re- lation, the relational semantics of the prototype is inevitably disturbed by irrelevant relations, thus affecting the discriminability of the metric space, depicted Figure 2(a). (2) Since most query en- tity pairs do not express any target relation, NOTA chone-of-the-abon
(a) Previous Methods
(b) Our Method
Tašk
Task
is also considered as cate- gory. Given that the target relation types vary for different tasks, if we merely introduce a set of learn- able vectors NOTA prototypes and apply them to all tasks, this "one-size-fits-ll" strategy could result in the NOTA prototypes deviating from ideal NOTA semantics in certain tasks, thereby confus- ing the classification. As shown in Figure 2(a), the set of generic NOTA prototypes seems reasonable for task 1, while does not work well for task 2.
Task
Task
Support Entity Pair Embedding
Support Relation Instancee Embesding
Prototypc
elaaton Prosstyae
Generic NOTA Prototype
Task-spec NOTA rotooype
Figure 2:
Embedding space illustration of previous methods (left) and our method (right). Task 1&2 are two FSDLRE tasks with different target relation types.
trastive learning method to jointly refine the re- lation prototypes. We also design task-specific NOTA prototype generation strategy to better cap- ture the NOTA semantics in each task. (3) Exper- iments demonstrate that our method outperforms state-of-the art baselines by average 2.61% 1% in Fi across various settings of two FSDLRE bench- marks.
To address the two aforementioned issues in FSDLRE, we propose novel Relation-A ware Prototype Learning method (RAPL). First, for each entity pair that holds relations in the support docu- ment, we leverage the inherent relational semantics in relation descriptions guidance, deriving instance-level representation for each expressed re- lation, as illustrated in Figure 2(b). The relation prototype is constructed by aggregating the repre- sentations all its support relation instances, thus better focusing on relation-relevant information. Based on the instance-level support embeddings, we propose a relation relattion-weighted contrastive learning method to further refine the prototypes. By incor- porating inter-relation similarities into contrastive objective, we can better distinguish the prototypes of semantically-cloot relations. Moreover, we de- sign task-specific NOTA prototype generation strategy. For each task, we adaptively select sup- port NOTA instances and fuse them into a set of learnable base NOTA prototypes to generate task- specific NOTA prototypes, which more effectively capture the NOTA semantics in each task.
2 Problem Formulation
Few-shot document-level relation extraction is de- fined with an N-Doc setting (Popovic and Färber, 2022). In each individual FSDLRE task (also called an episode?), there are set of N support doc- uments [Ds,1, ..., )s.,, and query document DQ. and the entity mentions in each document are pre-annotated. For each support document Ds there also a triple set Ts,; containing all valid (ea, r, ex) triples in the document. Here en and e, are the head and tail entity of relation instance, and r C Repisadee a relation type, with Repisode being the relation type set for which instances are to be extracted. The annotations of support docu- ments are complete, which means any entity pair for which relation type has been assigned can be considered as NOTA. Given these as inputs, the FSDLRE task aims predict the triple set Tq for the query document DQ, which contains all valid triples in DQ of relation types in Repisodee- Our approach follows the typical meta-learning
In summary, our main contributions are as fol- lows: (1) We propose novel relation aelation-aware aware proto- type learning method (RAPL) for FSDLRE, which effectively enhances the relational semanties of pro- totype representations. (2) In RAPL,, we reframe the construction of relation prototypes into instance level and further propose a relation welition-weighted con-
FI this paper, we use the terms "task" and "cpisode" inter- changeably, which refer to the same concept.

---

Classification
query embeddings
Entity Pair Learning
quary entty pair embaddings
Isarnable base NOTA prototypes
query
Doe Encoder
NOTA Prototype Generation
taken attention
Support NOTA Instance Learning
support instance embeddings
LACK LRCL
Support toxen embeddings
Relation Prototype Generation
class protospe representatins
Support Relation Instance Learning
support instance embeddings
documents
support cross
Delation-Weighlee Contractive Learning skxptt RJ)
Rei Encoder
larget byges
embeddings
Fime
positive instance pair
negative instance pair
Figure 3: The overall architecture of our proposed RAPL method.
paradigm. In training phase, we construct a group of training episodes by sampling support and query documents from a training document corpus Ctrain The set Repisode of each training episode a subset of Rerain a relation type set for meta-training. The model aims to learn general knowledge from these training tasks to better generalize to novel tasks. In test phase, the model is evaluated on group of test episodes sampled from a test document cor- pus Creat, which is disjoint with Cirain- The set Repisode of each test episode is a subset Ressts a relation type set for meta-testing, which is also disjoint with Rerain-
where H [hi...., hNd R N₁ the number of tokens in D, d the output dimension of encoder, and A = [a...... aN, E RNxxN is the average of attention heads in the last encoder layer. We take the embedding of "*" before each entity mention the corresponding mention em- bedding. For entity e; mentioned Ne; times in the document via Me, = {m²} / we apply log- sumexp pooling (Jia et al., 2019) over mention em- beddings to obtain the entity embedding he; Rd.
he, = log دド=1 exp exp(hm;) where hmgg E Rd is the embedding es's j-th mention.
3.2 Relation-Aware Relation Prototype Learning
3 Methodology
The overall architecture of RAPL is illustrated in Figure 3. We first introduce the encoding proce- dure for documents and entities in Section 3.1. In Section 3.2 and Section 3.3, elaborate the learning of relation-awan relation prototypes and NOTA protolypes respectively. The training and inference processes are finally given Section 3.4.
For each target relation type in a given episode, we aim to obtain prototype representation that can better capture the corresponding relational seman- tics. To this end, first propose construct the relation prototypes based on instance-level support embeddings, enabling each prototype focus more on relation-relevan information support docu- ments. Then we propose an instance- relation- weighted contrastive learning method, which fur- ther refines the relation prototypes.
3.1 Document and Entity Encoding
We employ the pre-trained language model (Devlin et al., 2019) the document encoder encode each support or query document in given episode, For each document D), we first insert special to- ken "*" at the start and end of each entity mention to mark the position entity mentions. Then we feed the document into the encoder to obtain the contextualized token embeddings H and cross to- ken attention A:
3.2.1 Instance-Based Prototype Construction
Given relation instance (eh, T,))) in support document, we first compute pair-level importance distribution a (K.A) R. over all tokens in the document capture the context relevant to the entity pair (en,e) e,) (Zhou et al., 2021):
a (h.t)
aen O ae
(2)
H. DocEncooer(D))
(1)
a aei

---

where Oeg - is an entity-level attention ob- tained by averaging the mention-level attention am E RN at the token "*" before en's s each
where Wh. We Rdx2d, by be E Rd are learn- able parameters. The instance representation of (ch, r, et) then obtained by concatenating the head and tail entity representations, which we de- note s(h.r,t) [=/hard)) :2 =(h.r1)] R
mention : Bep. - N.. En Nen amp likewise for Ong and @ is the Hadamard product. Meanwhile, we compute a relation-level attention distribution a" R over all tokens to capture the context rel- evant to relation r. We employ another pre-trained language model as the relation encoder, and con- catenate the name and description of relation r into a sequence, then feed the sequence into the encoder. We take the output embedding of "[CLSJ" token as the relation embedding h, E R4:
Finally, denoting the set of all instances of re- lation r in support documents as Sy, we compute the relation prototype p' E I22 by averaging the representations of relation instances in S,:
D
1
Σ
g(h.r.))
(9)
S
(eh.Mee)ES.
3.2.2 Contrastive- Prototype Refining By reframing the construction of relation proto- types into instance level, each prototype can bet- ter focus on celtioonrrlevvnt support information. However, due to the complexity of document con- text, different instances of the same relation may exhibit varying patterns in expressing the relation, making it difficult for prototypes to capture the common relational semantics. Additionally, lim- ited support instances make it challenging for pro- totypes of semantically-close relations to capture their deeper semantic differences. Therefore, we propose a gllaion--eighhte contrastive learning method to further refine the relation prototypes.
h (elEncoder(r),
and compute the relation-level attention a" as:
HWh h Vd
a' = softmax(
(4)
where W E Raxd is learnable parameter.
Based on a (h,1) and a", we further compute an instance-leved attention distribution a (h.na) E R- over all tokens to capture the context relevant to (hrrat) i-th
the instance. Specifically, the value aj dimension of a (hrrt) is obtained by:
of (5) where top- returns the indices of the largest 1% elements of æ, k hyperparameter, and I is the indicator function. We also normalize (Hh,r1)) to regain the attention distribution. Here we do not use a (hit) c a" as the instance-level attention because, for an instance, the relation is expressed based on the entity pair. Multiplying them directly may erro- neously increase the weight of tokens unrelated to the entity pair. Instead, we leverage relation-leve attention amplify the pair-level weight of the most relevant context with the instance.
(h.() (hrrs)) =0; =a, + T(1 top-k%(a (h.t o a²))
Specifically, given an episode, we denote the set of all relation instances in support documents as S, i.e., S = UrERepteode S,. Also, for rela- tion instance (en, r, Ct), we define the set Ph.r, S, \ {EE, r, {t)} which contains all other instances in the support set that also express the relation r, and the set Ar,,, = S {(E,, r, et)} which simply contains all other instances in the support set. Then we incorporate inter-relation similarities into con- trastive objective and define the relation-weighted weighted contrastive loss LRCL as:
LRCL = [S]
Σ (en,nee)ES
Σ
Phrrrl
Then, we compute an instance context embed- ding clh,r.)) E Rd by:
(ep.niez)EPoos
exp(s((.r(() s(hrr:D// (7)
(hrrd)) - (b
(6)
Σ
Wh. - exp(s((r()) slh.f.)/ /T)
and fuse it into the embeddings of head entity and tail entity to obtain the instance-aware entity repre- (h.rd) (h.r..) sentations 2/ z E Rdd:
(10)
W/F = 1 I(r cossim (hr, hp) +
(11)
2 (7) where hyperparameter and cossim denotes
(b.r..) = tanh( W W [hee chhrrttl ] + bb), bb
the cosine similarity. We argue that the proposal (8) of this contrastive loss is non-trivial, considering
(h.r.)) tanh( (Wi chrrte) ( b,),
zt

---

two aspects. First, is difficult for previous meth- ods to integrate with contrastive objective as they only obtain the pair-level support embeddings. The multi-label nature of entity pairs makes it diffi- cult to define positive and negative pairs reason- ably. Moreover, by incorporating inter-relation similarities, the proposed contrastive loss focuses more pushing apart the instance embeddings of semantically-close relations, thus helping to better distinguish the corresponding relation prototypes.
Benchmark
Tusk
(micco)
K (macro)
In Domain I-Doc
2.18
2.36
2.21
In-Domaiin 3-Doe
4.30
ReFREDo
In. Domain L-Dec
3.50
3.50
3,11
In-Domain 3-Doc
5.73
Cross-Domain 1-Dac
4.26
2.73
2,40
FREDo ReFREDo
Cross Domain 3-Doc
6.08
5.55
5.27
Table I: Average values for N (way) and K (shot)
across test episodes in two benchmarks. K' (micro) denotes the average across all episodes, K (macro) de- notes the average mean K for each relation type.
3.3 Relation-Aware NOTA Prototype Learning
where is hyperparameter. In this way, we ob- tain a set of task-specific NOTA prototypes which not only contain the general knowledge from meta- learning but also implicitly capture the NOTA se- mantics in each specific task.
Since most query entity pairs do not hold any target relation, NOTA is also treated as class. Existing methods typically learn set of generic NOTA pro- totypes that are applied to all tasks, which may not be optimal in certain tasks since the NOTA semantics difffers in tasks with different larget rela- tion types. To this end, we propose a task-specific NOTA prototype generation strategy to better cap- ture the NOTA semantics in each individual task. Concretely, we first introduce set of learn- able vectors -Mase {p!uu R2d 2 Naota where Nuota is hyperparameter. Unlike previous works that directly treat this set of vectors as NOTA pro- totypes, we regard them as base NOTA prototypes that need further rectification in each task. Since the annotation of support documents are complete, we have access to a support NOTA distribution which implicitly expresses the NOTA semantics. Therefore we resort to support NOTA instances capture the NOTA semanties in each specific task. For a support NOTA instance (ex, nota, et), we use Equation 2 as the instance-level attention and obtain the instance representation a (h,nota.t)
3,4 Training Objective
Given an entity pair (en, et) in the query document, we use Equation 2 as the pair-level attention and adopt a similar approach as Equation 6-8 to obtain
(h.4), the pair representation q [z] 2 (1.4)) E R2d, For each target relation type in the episode, we compute the probability of r as:
Ji=1
exp(q (h.t) p²) exp(q pp) max exp( (1htt) ((ttt))pp ppota) IEI
p(h..))
(14)
where I = {1, Nnota). Then, denoting the set of all entity pairs in the query document Q, we compute the classification loss as:
LBC
Σ (En,e))E rERepisode
Σ
Q
(uthWllgg (plhit)) + (1 y (h.1) log(( p!h,1))
(15)
2d based Equa- tion 6-8. Denoting the set of all support NOTA instances as Snotas we adaptively select a NOTA instance for each base NOTA prototype plase:
[z/h (h.aa,,,), 2
(h.mola.()
where yr
(h.1)
if exists between (en, ex), other-
wise yr (h.c)
= 0. The overall loss is defined as:
C = LBCE + XLRCL,
(16)
(Ch, nota, et) = argmax Fxt
(s (shnoota,t) base
where À is hyperparameter. During inference, we extract the relation instance (ea, r, et) in the query document (h,1) pp" max (h.t) nota
(h,nota,t) p²),
max rERegisode
iCI
(12)
which locates the NOTA instance that is close the base NOTA prototype while being far away from relation prototypes. Then we fuse it into pl pase to
Experiments
4.1 Benchmarks and Evaluation Metric
obtain the final NOTA prototype phota E 2d,
We conduct experiments on the public FSDLRE benchmark FREDo (Popovic and Färber, 2022), and also construct ReFREDo, a revised version
pote = app
Juse + (1 - (h.
(13)

---

FREDo
ReFREDo
Model
In- Domain
Cros-Dommain
In- Domaini
Cross- Domain
1-Doc
3-Doc Pi
1-Doc Ft
3-Doc F
1-Doc
3-Doc
1-Doc fi
3-Doc
DL-Base
0.60
0.89
1,76
1.98
1.38
1.84
1.76
1.98
DL-MNAV
7.05 0.18
8.42
0.64
0.84 + 0.16
0.48 + 0.21
12.97
0.88
12.43
0.36
1.12 + 0.38
2.28
0.19
DL-MNAV
7.06 0.15
6.77
0.21
1.77 = 0.60
2.51 +: 0.66
13.37
:: 0.98
12,00
0.80
1.39
0.74
2.92
0.41
DL-MNAV STF+8BN
1.71 = 0.04
2.79
+: 0.24
2,85 = 0.12
3 0.14
4.59 = 0.30
5,43
0.24
2.84
2.24
3.86 +: 0.27
KDDocRE
2.59
0.71
4.66
0.83
1.03 + 0.31
2,00
+ 0.46
4.76
0.55
9.02
0.64
2,30
+ 0.59
3.61 + 0.43
RAPL (Ours)
8.75 0.80
10.67
0.77
3.33 = 0.50
5.35 +: 0.72
15.20 :: 0.82
16.35
0.60
3.51
0.79
5.48 +: 0.63
Table 2: Results on FREDo and ReFREDo benchmarks. Reported results are macro averages across relation types. The best and second best performance methods are denoted in bold and underlined respectively.
of FREDo which resolves the annotation errors, enabling more reliable evaluation.
benchmarks is listed in Appendix A. Following Popovic and Färber (2022), we use macro Fi to evaluate the overall performance.
FREDo consists of two main tasks, an in-domain and cross-domain task. For in-domain tasks, the training and test document corpus are from the same domain. For cross-domain tasks, the test doc- uments taken from different domain, leading wider disparities of text style, document topic and relation types between training and test. Each task has I-Doc and a 3-Doc subtask to measure the scalability of models. FREDo uses the training set of DocRED ( Yao et al., 2019) as the training and development document corpus, the development set of DocRED as the in-domain test document corpus, and the whole set of SciERC (Luan et al., 2018) the cross-domain test document corpus. The relation type set of DocRED is split into disjoint sets for training (62), development (16) and in-domain test (18) in FREDo. FREDo sam- ples 1 15k episodes for in in-domain evaluation and 3k episodes for cross-domain evaluation.
4.2 Baselines
We compare our method with four baselines of FREDo (Popovic and Färber, 2022): DL-Base is initial baseline which uses the pre-trained language model without fine-tuming. DL-MNAV is metric- based approach built upon the state-of-the-art su- pervised DocRE method (Zhou et al., 2021) and few-shot sentence-level relation extraction method (Sabo et al., 2021). DL-MNAV STE uses all indi- vidual support entity pairs during inference instead of averaging their embeddings into single proto- type improve DL-MNAV for cross-domain tasks. DL-MNAV SIE+SBN uses NOTA instances as ad- ditional NOTA prototypes during training and only uses NOTA instances during inference to further improve DL-MNAV STF for cross-domain tasks. Besides, we also evaluate the supervised DocRE model by learning on the whole training corpus and fine-tuning on the support set. Here we choose KDDocRE (Tan et al., 2022a) which is the state-of- the-art public-available supervised DocRE method. For a fair comparison, we follow Popovic and Fär- ber (2022) to use BERT-base (Devlin et al., 2019) as the encoder in our approach. We present the implementation details in Appendix B.
Considering that FREDo uses DocRED, which suffers from the problem of incomplete annotation (Huang et al., 2022; Tan et al., 2022b), as the under- lying document corpus, the episodes constructed in FREDo may also inherit these annotation errors. revised
Therefore, construct ReFREDo version of FREDo. ReFREDo, we replace the training, development and in-domain test document corpus as Re-DocRED (Tan et al., 2022b), a revised version of DocRED with more complete annota- tions. Then we follow the same split of relation types as FREDo and sample 15k episodes for in- domain evaluation. The cross-domain test episodes remain the same with FREDo. We also follow Popovic and Färber (2022) to calculate the average class number N and average support instance num- ber per class K across test episodes in ReFREDo, as shown in Table An overview of the relation types and total instance number per relation of two
4.3 Main Results
The main results on FREDo and ReFREDo are shown in Table 2. We have following observations from the experimental results: (1) RAPL achieves significantly better average results two bench- marks compared baseline approaches (2.50% Fi on FREDo and 2.72% Fi on ReFREDo), demon- strating the superiority of our method. (2) RAPL consistently outperforms the best baseline method (which varies in different task settings) in each task

---

In-Domain
Cross- Domss-Domain
Model/ /
F, Score
Fi Score
1-Doc
3-Doc
I-Doc
3-Doc
In-Domain Cress Domain
10
20
RAPL
15.20
16.35
3,51
5.48
RCL
14.13
15.32
2,51
4,63
IBPC RCL
13.36
13.96
1,68
3.10
10
20
25
0.2
0.3
0.4
0.5
0.6
IBPC RCCL- SCL
13.51
13.88
1.95
3.23
Hyperparameter k
Hyperparameter
TNPG
14.50
15.69
2.99
4.72
Fr Score
Fi Score
Table 3: Ablation study results ReFREDo.
10
20
setting, making more versatile than previous ap- proaches. (3) RAPL shows more improvements on in-domain tasks compared to cross moos--domain tasks. This further reflects the greater challenge posed by cross-domain settings. (4) The performance of RAPL on 3-Doc tasks is consistently higher than that on 1-Doc tasks, which is not always guaranteed for the best baseline method, demonstrating the bet- ter scalability of RAPL. (5) The in-domain perfor- mance of all methods ReFREDo significantly higher than that on FREDo, while this performance gap is not reflected between two benchmarks un- der the cross-domain setting. This suggests that a higher-quality training set may not effectively re- (6) The perfor- mance of KDDocRE is not satisfactory, indicating that the supervised DocRE method may not adapt well to few-shot scenarios.
0.50
0.85
0.90
0.95
1.00
10
15
20
Hyperparameter
Hyperparameter Nasta
Figure 4: Effect of hyperparameters k. т, a and Nnota on RAPL under the 3-Doc task setting in ReFREDo.
sides, integrating the contrastive objective at pair- level do not bring significant improvements, which indicates the importance of learning instance-level support embeddings.
4.5 Analyses and Discussions
Effect of Hyperparameters. We investigate the impact of different hyperparameters the perfor- mance of our approach. We conduct the experi- ments on 3-Doc tasks in ReFREDo. As shown Figure 4, we can observe that: (1) For the hyperpa- rameter is which controls the derivation of instance- level attention, the best value for in-domain tasks are larger than tross-domain tasks, which may be related to the longer document in in in-domain corpus. (2) An appropriate temperature hyperparameter t (around 0.4) in the contrastive objective is crucial for the synergy with classification objective and the overall model performance. (3) Blindly reduc- ing the hyperparameter a to increase the weight of support NOTA instances in NOTA prototypes may have negative impact on the lcarning of NOTA prototypes. (4) Compared to other hyperparame- ters, the model is not very sensitive to the number of NOTA prototypes Nnota within certain range.
solve the domain adaption problem.
4.4 Ablation Study
We conduct ablation study on ReFREDo to investigate the influence of each module in our method. Specifically, "o "CRL",,, we remove the welghton-weighted contrastive learning method; for "-IBPC-RCT we further remove the instance- based relation prototype construction method, and only obtain the pair-level embedding for each sup- port entity pair in the same way query entity add a super-
pairs; for -BBBC--CL+SSC vised contrastive learning objective (Khosla al., 2020; Gunel et al., 2021) into the "-IBPC-RCL" model, where we treat those entity pairs sharing common relations as positive pairs, else as negative pairs; for < TNPG", we remove the task-specific NOTA prototype generation strategy, and directly treat the base NOTA prototypes final NOTA pro- totypes. The average results shown in Table 3. We can observe that the performance of model "-RCL" and "NTG" drops to varying degrees compared to RAPL, and the model IBPC RCI." performs even worse than "-RCL", demonstrating the effectiveness each module in our method. Be-
Support Embeddings Visualization. To intu- itively illustrate the advantage of our proposed method, we select three semantically-close rela- tion types from the in-domain 3-Doc test corpus of ReFREDo and sample ten support instances for each relation type, then use t-SNE for visu- alization ( (Van der Maaten and Hinton, 2008), shown in Figure Apart from two model vari- ants ablation study, we also experiment with RAPL-RCL+SCL, which replaces the relation- weighted contrastive loss with the supervised con-

---

25
S.core
20
(a) RA PL. IRPC RCI.
(b) RAPL, RCI
15
5
210
Number of Support Instances
partef nember of sindee of
Figure 6: Performance of RAPL under different number of support relation instances on in-domain 3-Doc tasks of ReFREDo.
(c) RAPL - RCL SCL
(d) RAPL
Visualization of support entity pair embeddings for RAPL-IBPC-RCI and support relation instance embeddings for RAPL-RCL AAPL-RCL-SSE and RAPL.
Figure 5:
scenarios where the NOTA rate falls within [0%, 95%), [95%, 97%), [97%, 99%) and [99%, 100%]. respectively. Then evaluate models trained with or without task-specific NOTA prototype genera- tion strategy on each subset. The experiment re- sults are shown Table 4. It is observed that the task-specific NOTA prototype generation strategy has brought improvement on each subset. More importantly, the performance gain gets larger the NOTA rate increases. It demonstrates that the task-specific NOTA prototype generation strategy conduces the capture of NOTA semanties for derived NOTA representations, especially in those episodes with more NOTA query pairs involved.
NRE 95%)
NRs 97%%
NRE 99%%
NR= 199%, 100%]
/ Fj
RAPL TNPG
23.11
18.65
17.11
5.55
23.51
19.12
17.76
6,66
RAPL.
(10.4)
(10.47)
(1065)
(f1.11)
Table 4: Model performance episodes with different NOTA rates (abbreviated "NR") in-domain 3-Doc test set of ReFREDo.
trastive loss (Khosla al., 2020; Gunel et al., 2021) instance level. Since some entity pairs express both the "part of"" and "member of"" rela- tion, or both the "part of" and "subclass of" re- lation, we only visualize "part of" relation for RAPL-IBPC-RCL in Figure 5(a). We can ob- serve that the support instance embeddings learned by RAPL-RCL improve the support pair embed- dings learned by RAPL-IBBE HAPL--IPC-RCL demonstrat- ing the effectiveness of instance-level embeddings for relation prototype construction. Besides, al- though incorporating instance-level supervised con- trastive objective forms more compact clusters, the distinction among three relation types is still in- sufficient. As shown in Figure 5(d), our proposed relation-weighted contrastive learning method bet- ter distinguishes the three relation types.
Performance vs. Number of Support Relation Instances. We also analyze the effect of the num- ber of support relation instances on the overall performance. We conduct the experiments on in- domain 3-Doc tasks in ReFREDo benchmark. For each relation type in each test episode, we calcu- late the number of support instances of that relation type in the episode. Here we divide the number of support instances into 10 categories, where the first categories correspond to 1 to 9, and the last cate- gory corresponds to cases where the number of sup- port instances is greater than or equal to 10. Then we evaluate the performance of RAPL method on each of these categories, as shown in Figure 6. We can observe that the performance of RAPL gen- erally exhibits upward trend as the number of support relation instances increases, while fluctua- tions also appear certain points. This indicates that the proposed method demonstrates a certain level of scalability, but the performance may not be perfectly positively correlated with the number support relation instances.
Performance vs. NOTA Rate of Episodes. We further explore the impact of task-specific NOTA prototype generation strategy on the performance improvements. We divide the in-domain 3-Doc episodes of ReFREDo into disjoint subsets ac- cording the NOTA rate of each episode, i.e., the proportion of NOTA entity pairs the total number of entity pairs in the query document of an episode. We establish four subsets, corresponding to the

---

Preliminary Exploration of LLM for FSDLRE. Recently large language models (LLM) (Brown et al., 2020; Touvron et al., 2023) have achieved promising results many few-shot tasks through in-context learning (Wei et al., 2022; Rubin et al., 2022). Also some works focus on leveraging LLM to solve few-shot information extraction prob- lems (Ma et al., 2023c; Ye et al., 2023; Wadhwa al., 2023). However, most studies mainly tar- get sentence-level tasks. Therefore, we conduct a preliminary experiment using gpt-3.5-turbo" to explore the performance of LLM on FSDLRE tasks. Due to the input length limit, we only exper- iment on 1-Doc setting, We randomly select 1000 episodes from the in-domain test episodes of Re- FREDo and design an in-context learning prompt template that includes task description, demonstra- tion and query (detailed in Appendix C). The ex- perimental results show that gpt-3.5-turbo achieves only 12.98% % macro F1, even lower than some base- line methods. Although the test may not fully re- flect the capabilities of LLM, we argue that FS- DLRE remains a challenging problem even in the era of LLM.
the document by graph structures and perform infer- ence with graph neural networks. Sequence-basee based methods (Xu al., 2021a; Tan et al., 2022a: Yu et al., 2022; Xiao et al., 2022; Ma et al., 2023b) encode the long-distance contextual dependencies with transformer-only architectures. Both cate- gories of methods have achieved impressive results in DocRE. However, the reliance on large-scale an- notated documents makes these methods difficult to adapt low-resource scenarios (Li et al., 2023; Hu al., 2023a).
Few-Shot Document-Level Relation Extraction. To tackle the data scarcity problem prevalent real-world DocRE scenarios, Popovic and Färber (2022) formulate DocRE into few-shot learn- ing task. To accomplish the task, they propose multiple metric-based models built upon the state- of-the-art supervised DocRE method (Zhou et al., 2021) and few-shot sentence-level relation extrac- tion method (Sabo et al., 2021), aiming to address different task settings. We note that for an effec- tive metric-based FSDLRE method, the prototype of each class should accurately capture the corre- sponding relational semantics. However, this can be challenging for existing methods due to their coarse grained relation prototype learning strategy and "one-for-all" NOTA prototype lcarning strat- egy. In this work, we propose relation-awan aware prototype learning method to better capture the re- lational semantics for prototype representations.
5 Related Work
Sentence-Level Relation Extraction. Relation extraction is a pivot task of information extraction (Hu et al., 2021, 2023b; Yang et al., 2023). Early studies mainly focus on predicting the relation be- tween two entities within a single sentence. A variety of pattern based (Pantel and Pennacchiotti, 2006; Mintz et al., 2009; Qu et al., 2018) and neu- ral based (Zhang et al., 2018; Baldini Soares et al., 2019; Hu al., 2020; Liu al., 2022c) models have achieved satisfactory results. Nevertheless, sentence-level relation extraction has significant limitations in terms of extraction scope and scale. The demand for cross-sentence and large-scale rela- tion extraction has led to a surge of research interest in document-level relation extraction (Quirk and Poon, 2017; Yao et al., 2019).
Conclusion
In this paper, we propose RAPL, a novel relation- aware prototype learning method for FSDLRE. We reframe the construction of relation prototypes into instance level and further propose relation- weighted contrastive learning method jointly refine the relation prototypes. Moreover, we de- sign a task-specific NOTA prototype generation strategy to better capture the NOTA semantics in each task. Experiment results and further anal- yses demonstrate the superiority of our method and effectiveness of each component. For future work, would like to transfer method other few-shot document-level IE tasks such few-shot document-level event argument extraction, which shares similar task structure with FSDLRE.
Document-Level Relation Extraction. Most of existing DocRE studies are grounded on a data- driven supervised scenario, and can be generally categorized into graph-based and sequence- -ased approaches. Graph-based methods (Zeng et al., 2020; Xu et al., 2021b; Zhang et al., 2021; Xu et al., 2022; Duan et al., 2022) typically abstract
Limitations
Firstly, the incorporation of relation encoder and the search process for support NOTA instances add
'https://plaffrms openai. htt///doc///oellll gpt-3-5

---

both memory and time expenses. This motivates us to further refine the overall efficiency of our pro- posed method. Secondly, the assumption that the entity information should be specified may affect the robustness of the method (Liu et al., 2022b). We have noticed that in supervised scenarios, some recent DocRE studies explore the joint entity and relation extraction to circumvent this assumption (Eberts and Ulges, 2021; Xu and Choi, 2022; Zhang al., 2023). We believe it is beneficial inves- tigate end-to-end DocRE in few-shot scenarios, where the RAPL method may shed some lights on future work. Lastly, the performance gain of RAPL on cross-domain tasks is lower than that on in-domain tasks. An intriguing avenue for future research is to explore techniques for better perfor- mance cross-domaie tasks, e.g., data augmen- tation (Hu al., 2023c) and structured knowledge guidance (Liu al., 2022a; Ma ct al., 2023a).
deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186.
Zhichao Duan, Xiuxing Li, Zhenyu Li, Zhuo Wang. and Jianyong Wang. 2022. Not just plain text! fuel document-level relation extraction with explicit syn- tax refinement and subsentence modeling. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 1941-1951.
Markus Eberts and Adrian Ulges. 2021. An end-to- end model for entity-level relation extraction using multi-instance learning. In Proceedings of the 16th Conference of the European Chapter of the Associ- alion for Computational Linguistics: Main Volume, pages 3650-3660.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. 2021. Supervised contrastive learning for pre-trained language model fine-tuning, In Interna- tional Conference on Learning Representations.
Acknowledgements
Xuming Hu, Junzhe Chen, Shiao Meng, Lijie Wen, and Philip S. Yu. 2023a. SelfIre: Self-refining represen- tation learning for low-resource relation extraction. In Proceedings of the 46th International ACM SI- GIR Conference Research and Development in Information Retrieval, page 2364-2366
We sincerely thank the anonymous reviewers for their valuable comments. The work was supported by the National Key Research and Development Program of China (No. 2019YFB CONYFB1704000 7040033, the National Nature Science Foundation of China (No. 62021002). Tsinghua BNRist and Beijing Key Lab- oratory of Industrial Bigdata System and Applica- tion.
Xuming Hu, Zhaochen Hong, Chenwei Zhang, Aiwei Liu, Shiao Meng, Lijie Wen, Irwin King, and Philip S. Yu. 2023b. Reading broadly to open your mind im- proving open relation extraction with search docu- ments under self-supervisions. IEEE Transactions on Knowledge and Data Engineering, pages 1-14.
References
Xuming Hu, Aiwei Liu, Zeqi Tan, Xin Zhang, Chenwei Zhang, Irwin King. and Philip S. Yu. 2023c. GDA: Generative data augmentation techniques for relation extraction tasks. Findings of the Association for Computational Linguistics: ACL 2023, pages 10221 10234.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2895- 2905.
Xumiin Hu, Lijic Wen, Yusong Xu, Chenwei Zhang, and Philip Yu. 2020. SelfORE: Self-supervised re- lational feature learning for open relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 3673-30000 3682.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan. Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, pages 1807-1901.
Xuming Hu, Chenwei Zhang, Yawen Yang, Xiaohe Li, Li Lin, Lijie Wen, and Philip S. Yu. 2021. Gradi- ent imitation reinforcement learning for low resource relation extraction. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 2737-2746.
Quzhe Huang, Shibo Hao, Yuan Ye, Shengqi Zhu, Yansong Feng, and Dongyan Zhao. 2022. Does recommend-revise produce reliable annotations? an
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of

---

analysis on missing instances in DocRED. In Pro- ceedings of the 60th Annual Meeting of the Associa- rion for Computational Linguisties (Volume I: Long Papers). pages 2241-6252.
Youmi Ma. An Wang. and Naoaki Okazaki. 2023b. DREEAM: Guiding attention with evidence for im- proving document-level relation extraction. In Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 1971-1983.
Robin Jia, Cliff Wong, and Hoifung Poon. 2019. Document-level n-ary relation extraction with multi- scale representation learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long and Short Papers), pages 3693-3704.
Yubo Ma, Yixin Cao, YongChing Hong, and Aixin Sun. 2023c. Large language model is not good few-shot information extractor, but a good reranker for hard samples! arXiv preprint arXiv: 308505590..
Mike Mintz, Steven Bills, Rion Snow, and Daniel Ju- rafsky. 2009. Distant supervision for relation ex- traction without labeled data, In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference Natural Language Processing of the AFNLP, pages 1003-1011.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Super- vised contrastive learning. In Advances in Neural In- formation Processing Systems, pages 18661-18673.
Shu' ang Li, Xuming Hu. Li Lin, Aiwei Liu. Lijie Wen. and Philip S. Yu. 2023. A multi-level supervised contrastive learning framework for low-resource nat- ural language inference. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:1771- 1783.
Patrick Pantel and Marco Pennacchiotti. 2006. Espresso: Leveraging generic patterns for automatically har- vesting semantic relations. In Proceedings of the 21st International Conference on Computational Lin- guistics and 44th Annual Meeting of the Association for Compulational Linguistics, pages 113-120.
Aiwei Liu, Xuming Hu, Li Lin, and Lijie Wen. 2022a. Semantic enhanced text-to-sql parsing via iteratively learning schema linking graph. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 1021-1030.
Adam Paszke. Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Detito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high performance deep learning li- brary. In Advances in Neural Information Processing Systems.
Aiwei Liu, Honghai Yu, Xuming Hu, Shu' ang Li, Li Lin, Fukun Ma, Yawen Yang, and Lijie Wen. 2022b. Character-level white-box adversarial attacks against transformers via attachable subwords substitution. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 7664-7676.
Nicholas Popovic and Michael Färber. 2022. Few-shot document-level relation extraction. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies. pages 5733-5746.
Shuliang Liu, Xuming Hu, Chenwei Zhang, Shu' ang Li, Lijie Wen, and Philip Yu. 2022c. HiURE: Hier- archical exemplar contrastive learning for unsuper- vised relation extraction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 5970-5980.
Meng Qu, Xiang Ren, Yu Zhang, and Jiawei Han. 2018. Weakly-supervised relation extraction by pattern- enhanced embedding learning. In Proceedings of the 2018 World Wide Web Conference, page 1257-1266.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Chris Quirk and Hoifung Poon. 2017. Distant super- vision for relation extraction beyond the sentence boundary. In Proceedings of the 15th Conference of the European Chapter of the Association for Compu- tational Linguisties: Volume 1, Long Papers, pages 1171-1182.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Con- ference on Empiricul Methods in Natural Language Processing, pages 3219-3232
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655-2671.
Fukun Ma. Xuming Hu, Aiwei Liu, Yawen Yang. Shuang Li. Philip S. Yu, and Lijie Wen. 2023a. AMR- hased network for aspect-based sentiment analysis. In Proceedings of the st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 322-337.
Ofer Sabo, Yanai Elazar, Yoav Goldberg, and Ido Da- gan. 2021. Revisiting Few-shot Relation Classifica- tion: Evaluation Data and Classification Schemes.

---

Transactions of the Association for Computational Linguistics, 9:691-706.
Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush, 2020. Trans- formers: State-of teeee-of-thhtrrrrreeeeee art natural language processing. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing: System Demonstrations, pages 38 45.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In Ad- vances Neural Information Processing Systems.
Qi Sun, Kun Huang, Xiaocui Yang. Pengfei Hong, Kun Zhang, and Soujanya Poria. 2023, Uncertainty guided label denoising for document-level distant relation extraction. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 15960- 15973.
Yuxin Xiao, Zecheng Zhang, Yuning Mao, Carl Yang. and Jiawei Han. 2022. SAIS: Supervising and aug- menting intermediate steps for document-level rela- tion extraction. In Proceedings of the 2022 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 2409.
Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. 2022a. Document-level relation extraction with adaptive focal loss and knowledge distillation. In Findings of the Association for Computational Lin- guistics: ACL 2022, pages 1672-1681.
Yiqing Xie, Jiaming Shen, Sha Li, Yuning Mao, and Ji- awei Han. 2022. Fider: Empowering document-level relation extraction with efficient evidence extraction and inference- stteeence-stage fusion. In Findings of the As- sociation for Computational Linguisticx: ACL 2022, pages 257-268.
Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng. and Sharifah Mahani Aljunied. 2022b. Revisiting Do- cRED- addressing the false negative problem in re- lation extraction. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 9472-8487.
Benfeng Xu, Quan Wang. Yajuan Lyu. Yong Zhu, and Zhendong Mao. 2021a. Entity structure within and throughout: Modeling mention dependencies for document-level relation extraction. Proceedings of the AAAI Conference on Artificial Intelligence. 35( 14149-14157
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maanee-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.155 139711.
Liyan Xu and Jinho Choi. 2022. Modeling task inter- actions in decument-level joint entity and relation extraction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 5409-5416.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, ko- ray kavukcuoglu, and Daan Wierstra, 2016. Match- ing networks for one shot learning. In Advances in Neural Information Processing Systems.
Tianyu Xu, Wen Hua, Jianfeng Qu, Zhixu Li, Jiajie Xu, An Liu, and Lei Zhao. 2022. Evidence-aware document- level relation extraction. In Proceedings of the 31st ACM International Conference on Infor- mation & Knowledge Management, page 2311 1-2200.
Somin Wadhwa, Silvio Amir, and Byron Wallace. 2023. Revisiting relation extraction in the era of large lan- guage models. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 15566- 15589,
Wang Xu, Kehai Chen, and Tiejun Zhao. 2021b. Dis- criminative reasoning for document-leve relation ex- traction. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 1653-1663.
Yawen Yang, Xuming Hu, Fukun Ma, Shu' Ang Li, Ai- wei Liu, Lijie Wen, and Philip S. Yu. 2023. Gaussian prior reinforcement learning for nested named entity recognition. In ICASSP 2023 2023 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
Jason Wei, Xuezhi Wang, Dale Schuurmans. Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing clicits reasoning in large language models. In Advances in Neural Information Processing Systems. pages 24824-24837.
Yuan Yao. Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu. Lixin Huang. Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-level relation extraction dataset. In Pro- ceedings of the 57th Annual Meeting of the Associa- tion for Computational Linguistics, pages 764-777.
Ying Wei and Qi Li. 2022. Sagdre: Sequence-awar graph-based document-level relation extraction with adaptive margin loss. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 2000-2008.

---

Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. 2023. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303. 10420.
development set. We take the learning rate le-5, The episode number per batch during training is set to 4. We clip the gradients to a max norm of 1.0. The hyperparameters k., T, Nnoto- a and À are set to 15, 0.4, 13, 0.9 and 0. for in-domain tasks, 10, 0.4, 20, 0.95 and 0.1 for cross-domain tasks, All hyperparameters are tuned on the development set. We report the mean and standard deviation of macro Fi by five training trials with different random seeds. All experiments are conducted with one Tesla V100-32G GPU. For the baseline results on ReFREDo, we reimplement all baseline models with official public codes for comparison.
Jiaxin Yu, Deqing Yang, and Shuyu Tian. 2022. Relation-spoee sptlifii attentions over entity mentions for enhanced document-leve relation extraction. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computaational Linguistics: Human Language lechnologies, pages 1523-1529.
Shuang Zeng. Runxin Xu, Baobao Chang, and Lei Li. 2020. Double graph based reasoning for document- level relation extraction. In Proceedings of the 2020 Conference on Empirical Methods Natural Lan- guage Processing (EMNLP), pages 1630-1664 16400
C In-Context Learning Prompt Template for 1-Doc FSDLRE Tasks
Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng. Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021. Document-level relation extrac- tion as semantic segmentation. In Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI-21, pages 4009.4006.
In-context learning prompt template for 1-Doc FSDLRE tasks:
Given target relation type list, document, and all entity mentions of each entity in the document, please identify all valid given relation types be- tween any two given entities in the document.
Ruoyu Zhang. Yanzeng Li, and Lei Zou. 2023. A novel table-to-an to -ragrppph generation approach for document- level joint entity and relation extraction. In Proceed- ings of the 6] Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers),
Target relation type names and descriptions: <Relation Name 1>: <Relation Description 1>
<Relation Name 2>: <Relation Description 2>
pages 10853-10865.
Yuhao Zhang, Peng Qi. and Christopher D. Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2205-2215.
Document (each entity mention is enclosed by the ID of the entity):
<Document>
ID and mentions of each entity in the document:
[[]: <Mention of Entity 1>; <Mention 2 of
Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. 2021. Document-lewe relation extraction with adaptive thresholding and localized context pool- ing. In Proceedings of the AAAI conference on artifi- cial intelligence, pages 14622-1662)
Entity 1>; ......
[2]: <Mention of Entity 2>; <Mention 2 of Entity 2>;
All non-duplicate valid "subject entity""" type" "ppe"""bjec"" "object entity" triples in the document (out- put format: "entity Ir"""relation type name"- "entity ID", e.g. [1]-country-12]; one triple per line):
A Relation Types in Benchmarks
In Table 5-9, we list the relation types of training, development, in-domain test and cross-domain test document corpus in FREDo and ReFREDo. We present the name, description and total instance number each relation type.
|<Entity ID>I-<Relation Name>- ID>] [<Entity ID>]<<Relation Name>- (<Entiyy ID>>
B Implementation Details
Document (each entity mention is enclosed by the ID of the entity):
We implement our method with Pytorch (Paszke et al., 2019) and Huggingface's Transformers (Wolf al., 2020). We use AdamW (Loshchilov and Hut- ter, 2019) for optimization with a linear warmup for the first 4% steps followed by a linear decay to O. We train the model for 50k episodes and per- form early stopping based on the macro Fi on the
<Document>
ID and mentions of each entity in the document:
[1]: <Mention 1 of Entity 1>; <Mention 2 of
Entity 1>; .....
[2] <Mention of Entity 2>; <Mention 2 of
Entity 2>;

---

All non-duplicate valid "subject entity" "relaton typee-""obect entity" triples in the document (out- put format: "entity ID"-"relation type name". "entity ID", e.g., [/]-couttr--[2]; one triple per line):
D
Case Study
We select a representative in-domain 1-Doc episode from ReFREDo benchmark for a case study, as shown Table 10, which intuitively illustrates both the superiority and the bottleneck of the RAPL method. We can observe that: (1) RAPL corrected a false negative prediction of relation P361 for the baseline method. Note that the entity pair of the only instance for P361 in the support docu- ment also expresses the relation P140 and P279, and the relation P279 and P463 are semantically close to P361. This suggests the effectiveness of instance-level prototype construction and relation- weighted contrastive refinement in RAPL method. (2) RAPL corrected false positive prediction of relation P140 between entity Mayflower and Epis- copal Diocese of Connecticut. This pair of entities actually does convey any target relationship in the query document. Such case may benefit from the task-specific NOTA prototype generation strat- egy, which better characters the NOTA semantics. (3) When the patterns or reasoning processes of the relation instances in query document differ sig- nificantly from the support instances with same relation type, RAPL often struggles with extrac- tion. Also, RAPL tends to exhibit cases of over- prediction, resulting in relatively lower precision. Although the proposed RAPL method achieves cer- tain improvements, the overall performance of few- shot DocRE still lags far behind the supervised setting, and how to overcome the two aforemen- tioned challenges is worth further exploration in future research.

---

# Iostances in FREDo
# Instances in ReFREDo
Wikidata ID
Name
Description
P131
located in the admin- istrative territorial en- tity
the item is located on the territory of the following administra- tive entity
4193
20402
P577
publication date performer
date point in time work is first published released performer involved in the performance or the recording of work
1142
1621
P175
1052
1773
PS69
date of birth date death
date on which the subject was horn date on which the subject died
1044
1172
P570
P527
has part
part this subject. Inverse property of "prrt of" actor performing live for a camera or audience brand and trademark associated with the marketing of subject music recordings and music videos most specific known (e.g. city instead of country, hospital instead of city) birth location of person, animal fictional character
632
2313
P161
cast member
621
919
P264
record lahel
583
923
P19
place of birth
511
692
P54
member of sports team child
sports teams or clubs that the subject currently represents formerly represented subject has the object in their family as their offspring son or daughter (independently of their age) continent of which the subject is a part
379
P40
360
703
P30
continent educated at platform
356
761
P69
educational institution attended by the subject platform for which a work has been developed or released
316
503
P400
304
460
specific platform version of softwure developed
P26
spouse
etc.) subject has the object as their spouse (husband, wife, partner,
303
640
P607
conffict
battles, wars or other military engagements in which the person or item purticipated male parent of the subject
275
575
P22
father
273
466
P159
headquarters tion
loca-
specific location where an organization's 's headquarters is or has
264
263
been situated
P178
developer
organisation or person that developed this item maker of creative work other object (where no more specific property exists)
238
402
P170
creator
231
410
P1344
purticipant of
event person or an organizution was participant in, inverse of "participant"
1168
P6
head of government
head of the executive power of this town, city, municipality, state. country, or other governmental body owner of the subject
210
368
P127
owned by place of death
208
389
P20
most specific known (c.g. city instead of country, hospital insteud of city) death location of person, animal fictional character
203
281
P108
employer
person organization which the subject works or worked scu, lake or river
421
P206
located or next to body of water followed by
194
431
P156
immediately following item in some series of which the subject
192
506
is part
P710
purticipant
person, group of people organization (object) that actively takes/took part the event (subject)
191
1168
P155
follows
immediately prior item in series of which the subject is part
188
506
P166
award received
award or recognition received by a person, organisation cre- ative work
340
P276
location
location of the item, physical object or event is within
172
336
Table 5: Relation types of training document corpus in FREDo and ReFREDo (continued on next page).

---

# Instances in FREDo
# Instances in ReFREDo
Wikidata ID
Name
Description
P123
publisher
organizution person responsible for publishing books, peri- odicals, games software
172
298
P58
screenwriter
author(s) the screenplay script for this work language(s) 6) that a person speaks or writes, including the native language(s)
237
P1412
languages spoken, written or signed
155
366
P449
original network
network(s) the radio or television show was originally aired on, including
152
264
PS00
notable work
notable scientific, artistic literary work, other work of significance among subject's works
150
3055
P706
located terrain fea-
located the specified landform
293
P37
official language
language designated as official by this item producer(s) of this film music work (film: not executive producers, associate producers, etc.)
119
281
P162
producer
119
249
P580
start time
indicates the time an item begins to exist or statement starts being valid branch to which this military unit, award, office, person belongs
110
222
P241
military branch
108
191
P937
work location instance
location where persons active that class of which this subject a particular example member. (Subject typically an individual member with Proper Name label.)
204
P31
225
point in time P585
time and date something took place, existed or statement was true the body of water to which the watercourse drains
96
191
P403
mouth of the water- course
95
200
P749
purent organization capital
parent organization of organisation, opposite of subsidiaries primary city of country, state other type administrative tervitorial entity
92
230
P36
85
178
P205
basin country ethnic group
country that have drainage to/from or border body water subject's ethnicity (consensus is that a VERY high standard of proof is needed for this field to be used. In general this means 1) the subject claims it him/berself, or 2) it is widely agreed by scholars. or is fictional and portrayed as such).
174
P172
79
155
P576
dissolved, abolished
date point time which organisation solved/disupeeared or building demolished
dis-
79
181
demolished capital
P1376
country, state, department, canton or other administrative sion of which the municipality is the govermeental seat
76
178
P171
parent taxon
closest parent taxon the taxon in question
75
117
P740
locution of formation
locution where a group or organization was formed the narrative of the work is set in this location author of song lyrics
62
102
P840
narrative location
48
83
P676
lyries by
36
79
P551
residence
the place where the person has been, resident administrative divisions that claim control of given person item replaced
35
66
P1336
territory claimed by
33
P1365
replaces
18
P737
influenced by
this person, idea, etc. is informed by that other person, idea, etc. other
P190
sister city
twin towns, sister cities, twinned municipalities calities that have a partnership or cooperative agreement, either legally or informally ucknowledged by their govermments portion of a workforce population that is not employed subject was founded or sturted by separating from identified object
P1198 PS07
unemployment rate separuted from
2
2
2
8
Table 6: Relation types of training document corpus in FREDo and ReFREDo (continued).

---

# Instances
Instances in ReFREDo
Wikidata ID
Name
Description
in FREDo
P27
country of citizen- ship
the object country that recognizes the subject its citizen
2689
4665
P150
contains administra- tive territorial entity
(list of) direct subdivisions of an administrative territorial entity
2004
3369
P571
inception
date or point in time when the organization/subjett was founded/created
475
868
P50
author
main creator(s) of written work (use on works, not humans) work in which this fictional entity or historical person is present director's) of this motion picture. TV-series, stageplay. video game similar
320
489
P1441
present in work
299
669
P57
director
246
341
P179
series
subject part of series, whose sum constitutes the object creative work's genre or an field of work founder co-founder of this organization, religion or place person or organization that operates the equipment, facility, or service
144
245
P136
genre
239
P112
founded by
204
P137
operator
95
192
P355
subsidiary
subsidiary of a company organization, opposite of parent compuny
92
230
P176
manufacturer
manufacturer producer of this product
83
144
P86
composer
person(s) who wrote the music presiding member of organization, group or body material or product produced by government agency, business, industry, facility, or process person or item which replaces another
79
171
P488
chairperson
63
145
P1056
product material
produced replaced by
P1366
36
96
Table 7: Relation types of development document corpus in FREDo and ReFREDo.
# Instances
# Instances in ReFREDo
Wikidata ID
Name
Description
in FREDo
P17
country
sovereign state of this item; don't use on humans country of origin of the creutive work subject item object of which the subject is part. Inverse property of "has part"
2831
5505
P495
country of origin part of
212
455
P361
194
900
P3373
sibling member member of politicul party
subject has the object as their sibling (brother, sister, etc.) organization or club to which subject belongs the political party of which this politician or has been a member
274
578
P102
98
P1001
upplics to jurisdic- tion
the item (an institution, law, public office ...) belongs to or has power over or applies to the value (a territorial jurisdiction: a country, state, municipality... ...)
83
485
P140
religion
religion of person, organization religious building, ciated with this subject
82
184
P674
characters
characters which appear in this (like plays, operas, op- erettas, books, comics, films, series, video games) legislative body governing this entity; political instirution with elected representatives, such as parliament//egislatee or coun- cil
74
204
P194
legislative body
56
119
P118
league
league in which team or player plays or has played in official with the highest formal authority in country/state company that produced this film, audio performing arts work all instances these items instances of those items; this item is a class (subset) that item language which film or performance work was originally created
56
126
P35
head of state
51
131
P272
production company
36
79
P279
subclass
36
P364
original language work end time
P582
indicates the time an item ceases to exist or a statement stops being valid
23
53
P25
mother position held
female parent of the subject
15
59
P39
subject currently or formerly holds the object position public office
8
19
Table 8: Relation types of in-domain test document corpus in FREDo and ReFREDo.

---

Instances FREDo
Instances in ReFREDo
SciERC ID
Name
Description
used-for
used for
subject is used for the object: subject models the object; object is trained on the subject; subject exploits the object; object based the subject.
2415
2415
conjunction hyponym-of
conjunction hyponym
function similar role usefincorporate with. subject hyponym of object; subject is type the object.
$77
477
477
evaluate-for purt-of feature-of
evaluate for part of feature of
evaluate for subject is purt of the object.
447
447
268
268
subject helongs the object; subject is feature of the ohject; subject is under the object domain.
264
264
compare
compare
compare models/methods, listing two opposing entities.
232
Table Relation types cross-domain test document corpus in FREDo and ReFREDo.

---

Support Document:
Adolfo Nicolás Pachón (born 29 April 1936), is a Spanish priest of the Roman Catholic Church. He was the thirtieth Superior General of the Society of Jesus, the largest religious order in the Roman Catholic Church. Nicolás, after consulting with Pope Francis, determined to resign after his 80th birthday. and initiated the process of calling Jesuit General Congregation to elect his successor. Until the resignation of his predecessor, Peter Hans Kolvenbach, was not the norm for a Jesuit Superior General to resign; they, like the great majority of the Popes up until Benedict XVI, generally served until death. However, the Jesuit constitutions include provision for a resignation. In October 2016 the thirty-sixth General Congregation of the Society of Jesus appointed his successor, Arturo Sosa from Venezuela.
Support Relation Instances:
P1001 [applies to jurisdiction): <Arturo Sosa P1001 - Venezucla>
P463 [member of]: <Adolfo Nicolás Pachón P463 Society of Jesus>; <Arturo Sosa P463 Society of Jesus>
P35 [head of state]: < <Venezuela P35 Arturo Sosa>
P279 [subclass of]: <Society of Jesus P279 Roman Catholic Church>; <Jesuit P279 Roman
Catholic Church>
P140 [religion]: <Adolfo Nicolás Pachón P140 Roman Catholic Church>; <Peter Hans Kolvenbach-
P140 Roman Catholic Church>; <Benedict XVI P140 Roman Catholic Church>; <Arturo Sosa
P140- Roman Catholic Church>; <Francis P140 Roman Catholic Church>; <Socicty of Jesus P140-
Roman Catholic Church>
P361 [part of]: <Society of Jesus. P361 Roman Catholic Church>
Query Document:
Chauncey Bunce Brewster (September 5, 1848 - April 9, 1941) was the fifth Bishop of the Episcopal Diocese of Connecticut. Brewster was born in Windham, Connecticut, to the Reverend Joseph Brewster and Sarah Jane Bunce Brewster. His father was rector St. Paul's Church in Windham and later became rector of Christ Church in New Haven, Connecticut. His younger brother was the future bishop Benjamin Brewster. The family were descendants of Mayflower passenger William Brewster. Brewster attended Hopkins Grammar School, then went to Yale College, where he graduated in 1868. At Yale he was elected Phi Beta Kappa and was a member of Skull and Bones. He attended Yalc's Berkeley Divinity School the following year. He was consecrated as bishop on October 28, 1897. He was coadjutor bishop before being diocesan bishop from 1899 1928.
Gold Outputs for Query Document:
<Chauncey Bunce Brewster P463 Phi Beta Kappa>; <Episcopal Diocese of Connecticut P1001 Connecticut>; <Berkeley Divinity School P361 Yale College>; <Chauneey Bunce Brewster P463 Skull and Bones>; <Chauncey Bunce Brewster P361 Skull and Bones>
Examples of RAPL Method Correcting Errors in DL-MNAV Method:
(1) Add the triple <Berkeley Divinity School P361 Yale College>, which is false negative case for DL-MNAV method.
(2) Drop the triple <.aaffower- P140 Episcopal Diocese of Connecticuts, which is false positive for DL-MNAV method.
Examples of Errors in RAPL Method:
(1) False negative prediction of the triple <Chauncey Bunce Brewster P463 Phi Beta Kappa>,
(2) False positive prediction of the triple <Benjamin Brewster P1001 New Haven>.
Table 10: Case study an in-domain I-Doe episode in ReFREDo, Entity mentions are indicated italics.


Reviews:
[{"review_id": "N0L5RjFCOB", "reviewer": "Reviewer_cDEF", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewers' recognition of the novelty and thoroughness of our work. We are also thankful for the insightful comments provided. We have responded to each suggestion regarding the areas for improvement in the manuscript, and will incorporate all these adjustments into the next version of the manuscript.\n\n> (1) \"Considering adding one more dataset (maybe modified one existing DocRE benchmark) to demonstrate this proposed method can generalize.\"\n\nThank you for the valuable feedback. We conducted a preliminary experiment on the DWIE dataset [1], which is an entity-centric dataset for document-level multi-task information extraction. DWIE consists of 802 general news articles in English, selected randomly from a corpus collected from Deutsche Welle between 2002 and 2018 [1]. We focus on relation extraction task and follow the preprocessing steps as [2] to obtain a version that meets the required format. We use the whole dataset of DWIE as a novel cross-domain document corpus and also sample 3k episodes as sciERC in FREDo and ReFREDo. Here we randomly select 15 out of 65 relation types in DWIE, where we also remove the relation types annotated in the in-domain training corpus from (Re-)DocRED, like 'spouse of' and 'child of', in order to prevent data leakage between train and test sets. We conducted experiments on ReFREDo benchmark (without too much hyperparameter tuning) and the results are shown in the table below. It is observed that the RAPL method consistently and significantly outperforms three baseline methods, demonstrating better generalization ability. We also notice that the overall results of these methods are higher than that on sciERC, which could be attributed to a smaller domain shift between DWIE and DocRED (news vs. wikipedia) compared to sciERC (science). We will further extend DWIE to training corpus (like DocRED in FREDo) and utilize the entire relation type set, in order to enrich the relevant benchmark dataset.\n\n| Model                | Cross-Domain 1-Doc $F _1$ | Cross-Domain 3-Doc $F _1$ |\n| -------------------- | :-----------------------: | :-----------------------: |\n| DL-MNAV              |      $2.51 \\pm 0.40$      |      $3.83 \\pm 0.48$      |\n| DL-MNAV$ _{SIE}$     |      $2.86 \\pm 0.68$      |      $4.53 \\pm 0.30$      |\n| DL-MNAV$ _{SIE+SBN}$ |      $3.79 \\pm 0.56$      |      $4.65 \\pm 0.73$      |\n| RAPL (Ours)          |      $4.98 \\pm 0.60$      |      $6.62 \\pm 0.83$      |\n\n> (2) \"Adding a few more case studies in the experiment section to intuitively show the output and also to promote the FSDLRE problem setting.\"\n\nThank you for the insightful suggestion. Here we select a representative in-domain 1-Doc episode from ReFREDo benchmark for a case study, as shown below, which intuitively illustrates both the superiority and the bottleneck of the RAPL method (we will also present more cases in the subsequent version of our manuscript). We can observe that: (1) RAPL corrected a false negative prediction of relation P361 for the baseline method. Note that the entity pair of the only instance for P361 in the support document also expresses the relation P140 and P279, and the relation P279 and P463 are semantically close to P361. Thus, this suggests the effectiveness of instance-level prototype construction and relation-weighted contrastive refinement in RAPL method. (2) RAPL corrected a false positive prediction of relation P140 between entity *Mayflower* and *Episcopal Diocese of Connecticut*. This pair of entities actually does not convey any target relationship in the document. There are also several similar false positive corrections, which are not listed here. All of these cases may benefit from the task-specific NOTA prototype generation strategy, which better characters the NOTA semantics. (3) When the patterns or reasoning processes of the relation instances in query document differ significantly from the support instances with same relation type, RAPL often struggles with extraction. Also, RAPL tends to exhibit cases of over-prediction, resulting in relatively lower precision. Although the proposed RAPL method achieves certain progress in FSDLRE, the overall performance of few-shot DocRE still lags far behind the supervised setting, and how to overcome the two aforementioned challenges is worth further exploration in future research.\n\n**Support Document:**\n\n*Adolfo Nicolás Pachón (born 29 April 1936), is a Spanish priest of the Roman Catholic Church. He was the thirtieth Superior General of the Society of Jesus, the largest religious order in the Roman Catholic Church. Nicolás, after consulting with Pope Francis, determined to resign after his 80th birthday, and initiated the process of calling a Jesuit General Congregation to elect his successor. Until the resignation of his predecessor, Peter Hans Kolvenbach, it was not the norm for a Jesuit Superior General to resign; they, like the great majority of the Popes up until Benedict XVI, generally served until death. However, the Jesuit constitutions include provision for a resignation. In October 2016 the thirty-sixth General Congregation of the Society of Jesus appointed his successor, Arturo Sosa from Venezuela.*\n\n**Support Relation Instances:**\n\n*P1001 [applies to jurisdiction]: <Arturo Sosa - P1001 - Venezuela>*\n\n*P463 [member of]: <Adolfo Nicolás Pachón - P463 - Society of Jesus>; <Arturo Sosa - P463 - Society of Jesus>*\n\n*P35 [head of state]: <Venezuela - P35 - Arturo Sosa>*\n\n*P279 [subclass of]: <Society of Jesus - P279 - Roman Catholic Church>; <Jesuit - P279 - Roman Catholic Church>*\n\n*P140 [religion]: <Adolfo Nicolás Pachón - P140 - Roman Catholic Church>; <Peter Hans Kolvenbach - P140 - Roman Catholic Church>; <Benedict XVI - P140 - Roman Catholic Church>; <Arturo Sosa - P140 - Roman Catholic Church>; <Francis - P140 - Roman Catholic Church>; <Society of Jesus - P140 - Roman Catholic Church>*\n\n*P361 [part of]: <Society of Jesus - P361 - Roman Catholic Church>*\n\n**Query Document:**\n\n*Chauncey Bunce Brewster (September 5, 1848 – April 9, 1941) was the fifth Bishop of the Episcopal Diocese of Connecticut. Brewster was born in Windham, Connecticut, to the Reverend Joseph Brewster and Sarah Jane Bunce Brewster. His father was rector of St. Paul's Church in Windham and later became rector of Christ Church in New Haven, Connecticut. His younger brother was the future bishop Benjamin Brewster. The family were descendants of Mayflower passenger William Brewster. Brewster attended Hopkins Grammar School, then went to Yale College, where he graduated in 1868. At Yale he was elected Phi Beta Kappa and was a member of Skull and Bones. He attended Yale's Berkeley Divinity School the following year. He was consecrated as a bishop on October 28, 1897. He was a coadjutor bishop before being diocesan bishop from 1899 to 1928.*\n\n**Gold Outputs for Query Document:**\n\n*<Chauncey Bunce Brewster - P463 - Phi Beta Kappa>; <Episcopal Diocese of Connecticut - P1001 - Connecticut>; <Berkeley Divinity School - P361 - Yale College>; <Chauncey Bunce Brewster - P463 - Skull and Bones>; <Chauncey Bunce Brewster - P361 - Skull and Bones>*\n\n**Examples of RAPL method correcting errors in DL-MNAV method:**\n\n1. Add the triple <Berkeley Divinity School - P361 - Yale College>, which is a false negative case for DL-MNAV method.\n2. Drop the triple <Mayflower - P140 - Episcopal Diocese of Connecticut>, which is a false positive case for DL-MNAV method.\n\n**Examples of errors in our RAPL method:**\n\n1. False negative prediction of the triple <Chauncey Bunce Brewster - P463 - Phi Beta Kappa>.\n2. False positive prediction of the triple <Benjamin Brewster - P1001 - New Haven>.\n\n> (3) \"Adding more recent DocRE literature in the reference.\"\n\nThanks for the constructive suggestion. We will add references to more recent DocRE literatures, including but not limited to \\[3]\\[4]\\[5]\\[6]\\[7].\n\n**References**\n\n[1] Zaporojets, K., Deleu, J., Develder, C., & Demeester, T. (2021). DWIE: An entity-centric dataset for multi-task document-level information extraction. Information Processing & Management, 58(4), 102563.\n\n[2] Ru, D., Sun, C., Feng, J., Qiu, L., Zhou, H., Zhang, W., ... & Li, L. (2021, November). Learning Logic Rules for Document-Level Relation Extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 1239-1250).\n\n[3] Xu, T., Hua, W., Qu, J., Li, Z., Xu, J., Liu, A., & Zhao, L. (2022, October). Evidence-aware Document-level Relation Extraction. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (pp. 2311-2320).\n\n[4] Xie, Y., Shen, J., Li, S., Mao, Y., & Han, J. (2022, May). Eider: Empowering Document-level Relation Extraction with Efficient Evidence Extraction and Inference-stage Fusion. In Findings of the Association for Computational Linguistics: ACL 2022 (pp. 257-268).\n\n[5] Xiao, Y., Zhang, Z., Mao, Y., Yang, C., & Han, J. (2022, July). SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 2395-2409).\n\n[6] Zhang, R., Li, Y., & Zou, L. (2023, July). A Novel Table-to-Graph Generation Approach for Document-Level Joint Entity and Relation Extraction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 10853-10865).\n\n[7] Sun, Q., Huang, K., Yang, X., Hong, P., Zhang, K., & Poria, S. (2023). Uncertainty Guided Label Denoising for Document-level Distant Relation Extraction. arXiv preprint arXiv:2305.11029.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper studies an important task and proposes an interesting approach to address the FSDLRE problem.\n2. The experiments are overall clear and well structured.\n3. The paper is clearly written and easy to access.", "reasons_to_reject": "Overall I feel the paper is solid, presenting some interesting findings and demonstrate the effectiveness of its proposed approach via multiple experiments. There are a few places where the authors can further improve the current writing:\n1. Considering adding one more dataset (maybe modified one existing DocRE benchmark) to demonstrate this proposed method can generalize.\n2. Adding a few more case studies in the experiment section to intuitively show the output and also to promote the FSDLRE problem setting.\n3. Adding more recent DocRE literature in the reference.  ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper studies the Few-shot document-level relation extraction (FSDLRE) problem where we need to identify semantic relations among entities in a document when there are only a few labeled samples available. The authors show the limitations of existing metric-based meta-learning approach and to address the problems, this paper introduces a relation-aware prototype learning method for FSDLRE. This method refines the relation prototypes and creates task-specific NOTA prototypes by utilizing relation descriptions and real NOTA instances. Through extensive experiments, this proposed method has shown to surpass current top-performing methods by an average 2.61% F1 across multiple benchmark settings for FSDLRE.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Just list a few more recent DocRE paper for citation\n\n1. Evidence-aware Document-level Relation Extraction. CIKM '22\n2. EIDER: Evidence-enhanced Document-level Relation Extraction. ACL'22\n3. A Novel Table-to-Graph Generation Approach for Document-Level Joint Entity and Relation Extraction. ACL'23"}}, {"review_id": "8MNX5PkSHI", "reviewer": "Reviewer_rUUw", "summary": "", "questions": "Question A: In line 280, what is the intuition behind defining the contrastive loss based on P_{h,r,t} and A_{h,r,t}? More explanation on what these two sets comprise would be helpful.\n\nQuestion B: What are the different tasks for the NOTA vector (line 344)?\n\nQuestion C: Since the main performance gain for RAPL comes from instance-level meta-learning, it would be good to analyze the effect of the number of relation-specific instances on the overall performance. The number of instances vs. performance curve should also help to understand the scalability of RAPL when the document length increases, which directly increases the number of entity pairs.\n\nQuestion D: As mentioned in line 449, does the higher performance on 3-Doc come from more relation instances? It needs to be clarified how scalability could be the reason for better performance on 3-Doc vs. 1-Doc.\n\nQuestion E: In line 593, it's mentioned that 1000 episodes were directly selected to run the relation extraction task on the ReFREDo benchmark using GPT-3.5. Getting the RAPL performance on this sample would be suitable for better comparison.\n\nQuestion F: Although RAPL beats the chosen baselines, presenting an error analysis on the cases where RAPL fails would be insightful, motivating further research in this direction. Especially for the NOTA case, it would be helpful to understand whether task-specific learning is beneficial.\n\nQuestion G: It's understandable that for comparison on the FREDo benchmark, macro-F1 is used. However, listing the relation-specific precision and recall scores would help understand whether the relation-specific instance-level tuning really helps RAPL's performance or if the gain only comes from handling the easy relations in the benchmark dataset.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewers' recognition of the structure and innovation of our work. We are thankful for the detailed feedback provided to help us enhance the quality of our manuscript. We will address each point of feedback and incorporate all these revisions into the subsequent version of our manuscript.\n\n> (1) \"The architecture uses the weak BERT encoder, which might cause a performance bottleneck.\"\n\nThanks for the constructive comment. As mentioned in Lines 430-432, the reason we use BERT-base as the document encoder is to make a fair comparison with all baseline methods (which also employ BERT-base encoder). In fact, other stronger text encoders can also be considered, such as RoBERTa-large, DeBERTa-large, etc. We have replaced the encoder of three best baseline methods and our RAPL method with RoBERTa-large, and conducted experiments on ReFREDo benchmark (without too much hyperparameter tuning). The results are shown in the table below. It can be observed that the stronger RoBERTa-large encoder brings an average improvement of 1.53% $F _1$ across four methods. Meanwhile, our RAPL method still consistently and significantly outperforms the baselines. We also argue that adopting DeBERTa-large may lead to even greater improvements.\n\n| Model                | In-Domain 1-Doc $F _1$ | In-Domain 3-Doc $F _1$ | Cross-Domain 1-Doc $F _1$ | Cross-Domain 3-Doc $F _1$ |\n| -------------------- | :--------------------: | :--------------------: | :-----------------------: | :-----------------------: |\n| DL-MNAV              |    $14.31 \\pm 0.79$    |    $15.20 \\pm 0.89$    |      $2.21 \\pm 0.40$      |      $3.64 \\pm 0.34$      |\n| DL-MNAV$ _{SIE}$     |    $14.57 \\pm 0.83$    |    $14.07 \\pm 0.61$    |      $2.59 \\pm 0.38$      |      $4.56 \\pm 0.36$      |\n| DL-MNAV$ _{SIE+SBN}$ |    $5.89 \\pm 0.22$     |    $7.17 \\pm 0.43$     |      $3.82 \\pm 0.48$      |      $5.44 \\pm 0.18$      |\n| RAPL (Ours)          |    $16.76 \\pm 0.86$    |    $18.30 \\pm 0.93$    |      $4.55 \\pm 0.65$      |      $7.12 \\pm 0.62$      |\n\n> (2) \"The method assumes that entity mentions should be specified for a query document, making it less robust.\"\n\nThank you for pointing out this issue. On one hand, this assumption tightly follows the task definition of FSDLRE as outlined in [1], where it is mentioned that the entity information is provided as part of the input. On the other hand, this assumption could indeed impact the robustness of the method. We have noticed that in supervised scenarios, some recent DocRE studies have started to exploring the joint entity and relation extraction to circumvent this assumption, and have made some progress \\[2]\\[3]\\[4]. We would like to investigate end-to-end DocRE in few-shot scenarios in our future work, where the RAPL method may shed some lights on the design of end-to-end models.\n\n> (3) \"One of the paper's main contributions is to build a task-specific NOTA prototype; however, precise experiments to evaluate NOTA relations are missing.\"\n\nThank you for the valuable feedback. We have further explored the impact of task-specific NOTA prototype generation strategy on the performance improvements. We divide the episodes of in-domain 3-Doc setting in ReFREDo benchmark into disjoint subsets according to the NOTA rate of each episode, i.e., the proportion of NOTA entity pairs to the total number of entity pairs in the query document of an episode. We establish four subsets, corresponding to the scenarios where the NOTA rate falls within [0%, 95%), [95%, 97%), [97%, 99%) and [99%, 100%], respectively. Then we evaluate models trained with or without task-specific NOTA prototype generation strategy on each subset. The experiment results are shown in the table below (where 'NR' refers to NOTA rate and 'TNPG' refers to task-specific NOTA prototype generation strategy). It is observed that the task-specific NOTA prototype generation strategy has brought improvement on each subset. And more importantly, the performance gain gets larger as the NOTA rate increases. It demonstrates that by incorporating task-specific NOTA prototype generation strategy, the derived NOTA representations conduce to capture the NOTA semantics, especially in those episodes with more NOTA query pairs involved.\n\n| Model         | NR $\\in$ [0%, 95%) $F _1$ | NR $\\in$ [95%, 97%) $F _1$ | NR $\\in$ [97%, 99%) $F _1$ | NR $\\in$ [99%, 100%] $F _1$ |\n| ------------- | :-----------------------: | :------------------------: | :------------------------: | :-------------------------: |\n| RAPL w/o TNPG |          $23.11$          |          $18.65$           |          $17.11$           |           $5.55$            |\n| RAPL          | $23.51$ ($\\uparrow 0.40$) | $19.12$ ($\\uparrow 0.47$)  | $17.76$ ($\\uparrow 0.65$)  |  $6.66$ ($\\uparrow 1.11$)   |\n\n> (4) \"Question A: In line 280, what is the intuition behind defining the contrastive loss based on P_{h,r,t} and A_{h,r,t}? More explanation on what these two sets comprise would be helpful.\"\n\nThank you for the constructive suggestion. We apologize for not providing a detailed explanation on these two sets. Following the symbol definitions in the manuscript, for a relation instance $(e _{h}, r, e _{t})$ in the support set, the set $\\mathcal{P} _{h,r,t}=\\mathcal{S} _{r}\\setminus \\{(e _{h}, r, e _{t})\\} $ contains all other instances in the support set that also express the relation $r$, and the set $\\mathcal{A} _{h,r,t}=\\mathcal{S}\\setminus \\{(e _{h}, r, e _{t})\\} $ simply contains all other instances in the support set. Therefore, considering $(e _{h}, r, e _{t})$ as an anchor in the contrastive objective, each instance in $\\mathcal{P} _{h,r,t}$ constitutes a positive example of $(e _{h}, r, e _{t})$, and the contrastive objective aims to pull close the instances of same relation (i.e. positive examples) and push apart the instances of different relations (i.e. negative examples).\n\n> (5) \"Question B: What are the different tasks for the NOTA vector (line 344)?\"\n\nWe speculate that this comment pertains to the sentence in Lines 306-311, where we mentioned that \"the NOTA semantics differs in tasks with different target relation types\". Here what we mean is, for two tasks (i.e. episodes) with different target relation types, suppose being R1, R2, R3 and R4, R5, R6 respectively, the \"NOTA\" of these two tasks can be considered as \"none of R1, R2, R3\" and \"none of R4, R5, R6\", which are also different in terms of relational semantics.\n\n> (6) \"Question C: Since the main performance gain for RAPL comes from instance-level meta-learning, it would be good to analyze the effect of the number of relation-specific instances on the overall performance. The number of instances vs. performance curve should also help to understand the scalability of RAPL when the document length increases, which directly increases the number of entity pairs.\"\n\nThank you for the insightful suggestion. We have further analyzed the effect of the number of support relation instances on the overall performance. We conduct the experiments on in-domain 3-Doc tasks in ReFREDo benchmark. For each relation type in each test episode, we calculate the number of support instances of that relation type in the episode. Here we divide the number of support instances into 10 categories, where the first 9 categories correspond to 1 to 9, and the last category corresponds to cases where the number of support instances is greater than or equal to 10. Then we evaluate the performance of RAPL method on each of these categories. The results are shown in the table below (we denote the number of support relation instances as $K$ in the table; and we will present the relevant findings in a more visual manner using line charts in the subsequent version of our manuscript). We can observe that the performance of RAPL method generally exhibits an upward trend as the number of support relation instances increases, while fluctuations also appear at certain points. This indicates that the RAPL method demonstrates a certain level of scalability, but the performance may not be perfectly positively correlated with the number of support relation instances.\n\n| K      |    1    |    2    |    3    |    4    |    5    |    6    |    7    |    8    |    9    | $\\ge$ 10 |\n| ------ | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :------: |\n| $F _1$ | $11.52$ | $12.75$ | $17.47$ | $18.89$ | $16.67$ | $20.58$ | $20.59$ | $14.62$ | $24.57$ | $20.64$  |\n\n> (7) \"Question D: As mentioned in line 449, does the higher performance on 3-Doc come from more relation instances? It needs to be clarified how scalability could be the reason for better performance on 3-Doc vs. 1-Doc.\"\n\nThanks for the suggestion. As mentioned in [1], the logical relationship here is that, the performance variation on 3-Doc setting (which has a higher average number of support instances for each relation type than 1-Doc setting, as shown in Table 1 in the manuscript) vs. 1-Doc setting can be used to measure the scalability of the model. Based on the experimental results we can observe that the performance of RAPL on 3-Doc tasks is consistently higher than that on 1-Doc tasks, which is not completely achieved by some strong baseline methods, demonstrating the better scalability of our proposed method.\n\n> (8) \"Question E: In line 593, it's mentioned that 1000 episodes were directly selected to run the relation extraction task on the ReFREDo benchmark using GPT-3.5. Getting the RAPL performance on this sample would be suitable for better comparison.\"\n\nThank you for the constructive feedback. It is indeed fairer to compare the results of baseline&RAPL methods on these 1000 sampled episodes with the reported result of gpt-3.5-turbo in the manuscript. We have tested our RAPL method and two strong baseline methods (DL-MNAV, DL-MNAV$ _{SIE}$) on these 1000 sampled episodes, and the results are shown in the table below. Compared to the results obtained on the entire test set of 15k episodes, the performances on sampled episodes do not exhibit significant differences. As a result, similar to the discussion in the manuscript, utilizing a generic LLM for in-context learning on FSDLRE does not demonstrate superior performance compared to the baseline and our RAPL methods.\n\n| Model            | ReFREDo In-Domain 1-Doc $F _1$ |\n| ---------------- | :----------------------------: |\n| DL-MNAV          |        $12.89 \\pm 0.93$        |\n| DL-MNAV$ _{SIE}$ |        $13.11 \\pm 0.87$        |\n| RAPL (Ours)      |        $15.40 \\pm 0.80$        |\n| gpt-3.5-turbo    |            $12.98$             |\n\n> (9) \"Question F: Although RAPL beats the chosen baselines, presenting an error analysis on the cases where RAPL fails would be insightful, motivating further research in this direction. Especially for the NOTA case, it would be helpful to understand whether task-specific learning is beneficial.\"\n\nThank you for the helpful suggestion. Here we select a representative in-domain 1-Doc episode from ReFREDo benchmark for a case study, as shown below, which intuitively illustrates both the superiority and the bottleneck of the RAPL method (we will also present more cases in the subsequent version of our manuscript). We can observe that: (1) RAPL corrected a false negative prediction of relation P361 for the baseline method. Note that the entity pair of the only instance for P361 in the support document also expresses the relation P140 and P279, and the relation P279 and P463 are semantically close to P361. Thus, this suggests the effectiveness of instance-level prototype construction and relation-weighted contrastive refinement in RAPL method. (2) RAPL corrected a false positive prediction of relation P140 between entity *Mayflower* and *Episcopal Diocese of Connecticut*. This pair of entities actually does not convey any target relationship in the document. There are also several similar false positive corrections, which are not listed here. All of these cases may benefit from the task-specific NOTA prototype generation strategy, which better characters the NOTA semantics. (3) When the patterns or reasoning processes of the relation instances in query document differ significantly from the support instances with same relation type, RAPL often struggles with extraction. Also, RAPL tends to exhibit cases of over-prediction, resulting in relatively lower precision. Although the proposed RAPL method achieves certain progress in FSDLRE, the overall performance of few-shot DocRE still lags far behind the supervised setting, and how to overcome the two aforementioned challenges is worth further exploration in future research.\n\n**Support Document:**\n\n*Adolfo Nicolás Pachón (born 29 April 1936), is a Spanish priest of the Roman Catholic Church. He was the thirtieth Superior General of the Society of Jesus, the largest religious order in the Roman Catholic Church. Nicolás, after consulting with Pope Francis, determined to resign after his 80th birthday, and initiated the process of calling a Jesuit General Congregation to elect his successor. Until the resignation of his predecessor, Peter Hans Kolvenbach, it was not the norm for a Jesuit Superior General to resign; they, like the great majority of the Popes up until Benedict XVI, generally served until death. However, the Jesuit constitutions include provision for a resignation. In October 2016 the thirty-sixth General Congregation of the Society of Jesus appointed his successor, Arturo Sosa from Venezuela.*\n\n**Support Relation Instances:**\n\n*P1001 [applies to jurisdiction]: <Arturo Sosa - P1001 - Venezuela>*\n\n*P463 [member of]: <Adolfo Nicolás Pachón - P463 - Society of Jesus>; <Arturo Sosa - P463 - Society of Jesus>*\n\n*P35 [head of state]: <Venezuela - P35 - Arturo Sosa>*\n\n*P279 [subclass of]: <Society of Jesus - P279 - Roman Catholic Church>; <Jesuit - P279 - Roman Catholic Church>*\n\n*P140 [religion]: <Adolfo Nicolás Pachón - P140 - Roman Catholic Church>; <Peter Hans Kolvenbach - P140 - Roman Catholic Church>; <Benedict XVI - P140 - Roman Catholic Church>; <Arturo Sosa - P140 - Roman Catholic Church>; <Francis - P140 - Roman Catholic Church>; <Society of Jesus - P140 - Roman Catholic Church>*\n\n*P361 [part of]: <Society of Jesus - P361 - Roman Catholic Church>*\n\n**Query Document:**\n\n*Chauncey Bunce Brewster (September 5, 1848 – April 9, 1941) was the fifth Bishop of the Episcopal Diocese of Connecticut. Brewster was born in Windham, Connecticut, to the Reverend Joseph Brewster and Sarah Jane Bunce Brewster. His father was rector of St. Paul's Church in Windham and later became rector of Christ Church in New Haven, Connecticut. His younger brother was the future bishop Benjamin Brewster. The family were descendants of Mayflower passenger William Brewster. Brewster attended Hopkins Grammar School, then went to Yale College, where he graduated in 1868. At Yale he was elected Phi Beta Kappa and was a member of Skull and Bones. He attended Yale's Berkeley Divinity School the following year. He was consecrated as a bishop on October 28, 1897. He was a coadjutor bishop before being diocesan bishop from 1899 to 1928.*\n\n**Gold Outputs for Query Document:**\n\n*<Chauncey Bunce Brewster - P463 - Phi Beta Kappa>; <Episcopal Diocese of Connecticut - P1001 - Connecticut>; <Berkeley Divinity School - P361 - Yale College>; <Chauncey Bunce Brewster - P463 - Skull and Bones>; <Chauncey Bunce Brewster - P361 - Skull and Bones>*\n\n**Examples of RAPL method correcting errors in DL-MNAV method:**\n\n1. Add the triple <Berkeley Divinity School - P361 - Yale College>, which is a false negative case for DL-MNAV method.\n2. Drop the triple <Mayflower - P140 - Episcopal Diocese of Connecticut>, which is a false positive case for DL-MNAV method.\n\n**Examples of errors in our RAPL method:**\n\n1. False negative prediction of the triple <Chauncey Bunce Brewster - P463 - Phi Beta Kappa>.\n2. False positive prediction of the triple <Benjamin Brewster - P1001 - New Haven>.\n\n> (10) \"Question G: It's understandable that for comparison on the FREDo benchmark, macro-F1 is used. However, listing the relation-specific precision and recall scores would help understand whether the relation-specific instance-level tuning really helps RAPL's performance or if the gain only comes from handling the easy relations in the benchmark dataset.\"\n\nThanks for the insightful suggestion. To investigate the performance gain of RAPL at a finer granularity of per-relation level, we first compare the per-relation performance of RAPL with the best baseline method on each task setting of ReFREDo benchmark. The results show that, for in-domain setting (with 18 relation types in the test set), RAPL outperforms the baseline method on 17 (1-Doc) and 16 (3-Doc) relation types respectively. For cross-domain setting (with 7 relation types in the test set), RAPL outperforms the baseline method on 5 (1-Doc) an 7 (3-Doc) relation types respectively. This demonstrates that RAPL has improved the predictions for the majority of relation types. Furthermore, in the table below, we have listed the 5 relation types with the highest (the first 5 lines) and lowest (the last 5 lines) $F _1$ scores for the best baseline method (DL-MNAV) on in-domain 3-Doc setting. We present the precision, recall and $F _1$ scores of DL-MNAV and RAPL on these 10 relation types, and compare the variations in $F _1$ scores in the last column. We can observe that, the performance gain of RAPL is similar across the 5 easiest and 5 most difficult relation types of the baseline method. This further suggests that RAPL is capable of handling both easy and hard relation types simultaneously.\n\n| Relation                  |   DL-MNAV $P$    |     RAPL $P$     |   DL-MNAV $R$    |     RAPL $R$     |  DL-MNAV $F _1$  |   RAPL $F _1$    |   $\\Delta F _1$   |\n| ------------------------- | :--------------: | :--------------: | :--------------: | :--------------: | :--------------: | :--------------: | :---------------: |\n| sibling                   | $21.30 \\pm 1.52$ | $24.94 \\pm 2.07$ | $49.27 \\pm 8.78$ | $51.07 \\pm 5.71$ | $29.60 \\pm 2.65$ | $33.38 \\pm 2.96$ |  $\\uparrow 3.78$  |\n| country of origin         | $15.28 \\pm 2.13$ | $13.13 \\pm 3.05$ | $58.72 \\pm 5.18$ | $50.01 \\pm 4.68$ | $24.13 \\pm 2.47$ | $20.67 \\pm 3.39$ | $\\downarrow 3.46$ |\n| league                    | $14.36 \\pm 2.38$ | $24.94 \\pm 4.99$ | $75.74 \\pm 3.67$ | $71.49 \\pm 7.07$ | $24.05 \\pm 3.35$ | $33.46 \\pm 5.98$ |  $\\uparrow 9.41$  |\n| country                   | $59.18 \\pm 3.54$ | $52.93 \\pm 2.43$ | $12.83 \\pm 2.32$ | $17.67 \\pm 2.80$ | $20.97 \\pm 3.20$ | $26.66 \\pm 2.59$ |  $\\uparrow 5.69$  |\n| production company        | $8.67 \\pm 0.61$  | $10.77 \\pm 1.51$ | $54.73 \\pm 6.30$ | $42.52 \\pm 3.61$ | $14.95 \\pm 0.97$ | $17.24 \\pm 1.83$ |  $\\uparrow 2.29$  |\n| religion                  | $4.63 \\pm 0.92$  | $11.18 \\pm 1.91$ | $11.56 \\pm 1.31$ | $11.71 \\pm 1.25$ | $6.60 \\pm 1.15$  | $11.55 \\pm 1.35$ |  $\\uparrow 4.95$  |\n| part of                   | $6.36 \\pm 0.17$  | $7.99 \\pm 0.63$  | $5.12 \\pm 0.42$  | $11.69 \\pm 0.88$ | $5.66 \\pm 0.25$  | $9.45 \\pm 0.70$  |  $\\uparrow 3.79$  |\n| member of political party | $3.00 \\pm 0.38$  | $8.40 \\pm 1.50$  | $26.64 \\pm 3.30$ | $38.43 \\pm 4.09$ | $5.39 \\pm 0.67$  | $13.71 \\pm 1.84$ |  $\\uparrow 8.32$  |\n| subclass of               | $2.18 \\pm 0.62$  | $4.72 \\pm 0.71$  | $5.40 \\pm 2.04$  | $12.39 \\pm 3.05$ | $3.08 \\pm 0.94$  | $6.79 \\pm 1.04$  |  $\\uparrow 3.71$  |\n| position held             | $0.65 \\pm 0.11$  | $1.98 \\pm 1.01$  | $37.55 \\pm 2.74$ | $30.20 \\pm 1.92$ | $1.29 \\pm 0.21$  | $3.69 \\pm 1.12$  |  $\\uparrow 2.40$  |\n\n**References**\n\n[1] Popovic, N., & Färber, M. (2022, July). Few-Shot Document-Level Relation Extraction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 5733-5746).\n\n[2] Eberts, M., & Ulges, A. (2021, April). An End-to-end Model for Entity-level Relation Extraction using Multi-instance Learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (pp. 3650-3660).\n\n[3] Xu, L., & Choi, J. D. (2022, July). Modeling Task Interactions in Document-Level Joint Entity and Relation Extraction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 5409-5416).\n\n[4] Zhang, R., Li, Y., & Zou, L. (2023, July). A Novel Table-to-Graph Generation Approach for Document-Level Joint Entity and Relation Extraction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 10853-10865).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very well-written and easy to follow. The RAPL model proposed in the paper handled an important task of the few-shot document-level relation extraction task. Although the work can seem incremental on the FREDo benchmark, the claims and rationales proposed in the paper are intuitive and justified with experimental results. The methodology is explained in detail, and combining contrastive learning with instance-level aggregation for prototype tuning is interesting.", "reasons_to_reject": "The RAPL model is incremental work, and the architecture uses the weak BERT encoder, which might cause a performance bottleneck. The method assumes that entity mentions should be specified for a query document, making it less robust. One of the paper's main contributions is to build a task-specific NOTA prototype; however, precise experiments to evaluate NOTA relations are missing.", "questions_to_authors": "Question A: In line 280, what is the intuition behind defining the contrastive loss based on P_{h,r,t} and A_{h,r,t}? More explanation on what these two sets comprise would be helpful.\n\nQuestion B: What are the different tasks for the NOTA vector (line 344)?\n\nQuestion C: Since the main performance gain for RAPL comes from instance-level meta-learning, it would be good to analyze the effect of the number of relation-specific instances on the overall performance. The number of instances vs. performance curve should also help to understand the scalability of RAPL when the document length increases, which directly increases the number of entity pairs.\n\nQuestion D: As mentioned in line 449, does the higher performance on 3-Doc come from more relation instances? It needs to be clarified how scalability could be the reason for better performance on 3-Doc vs. 1-Doc.\n\nQuestion E: In line 593, it's mentioned that 1000 episodes were directly selected to run the relation extraction task on the ReFREDo benchmark using GPT-3.5. Getting the RAPL performance on this sample would be suitable for better comparison.\n\nQuestion F: Although RAPL beats the chosen baselines, presenting an error analysis on the cases where RAPL fails would be insightful, motivating further research in this direction. Especially for the NOTA case, it would be helpful to understand whether task-specific learning is beneficial.\n\nQuestion G: It's understandable that for comparison on the FREDo benchmark, macro-F1 is used. However, listing the relation-specific precision and recall scores would help understand whether the relation-specific instance-level tuning really helps RAPL's performance or if the gain only comes from handling the easy relations in the benchmark dataset.", "paper_topic_and_main_contributions": "The paper proposes the RAPL model for a few-shot document-level relation extraction task. RAPL model follows a relation-aware prototype learning method for generating a better representation of semantic relations. In contrast to previous approaches, which aggregate the representations of all relevant entity pairs for learning the prototype, RAPL computes instance-level representation for each relation and incorporates contrastive learning to distinguish semantically-close relations better. RAPL also handles the \"none-of-the-above\" (NOTA) case of relation extraction by constructing task-specific NOTA vectors in contrast to the generalized NOTA vector. RAPL is tested on the FREDo benchmark and performs better than the chosen baseline methods. The authors also propose a new benchmark, ReFREDo, which replaces the DocRED version in FREDo with an improved version previously released by Tan et al., 2022b.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8Fqm2Q5BSy", "reviewer": "Reviewer_e9xn", "summary": "", "questions": "1.The computation process to $\\bm{a}_{e_h}$, $\\bm{a}_{e_t}$ in Line210 should be specified.\n\n2.$a^{(h,r,t)}$ in Eq.5 is not in bold while $a^{(h,r,t)}$ in Line235 is in bold. Are these two symbols the same, or what are the differences between them?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewers' recognition of the significance and validity of our work. We are also grateful for the constructive comments provided. We have responded to each suggestion pertaining to areas for enhancement in the manuscript, and will integrate all these revisions into the subsequent version of the manuscript.\n\n> (1) \"The computation process to $\\boldsymbol{a} _{e _{h}}, \\boldsymbol{a} _{e _{t}}$ in Line210 should be specified.\"\n\nThanks for the constructive suggestion. We apologize that our explanation regarding the computation process of $\\boldsymbol{a} _{e _{h}}$ and $\\boldsymbol{a} _{e _{t}}$ in Lines 211-214 is not clear enough. Here we elaborate on it more formally. Following the symbol definitions in the manuscript, given the cross token attention matrix $\\boldsymbol{A} \\in \\mathbb{R}^{N _{t}\\times N _{t}}$ output by the document encoder, we first denote the attention from \"*\" symbol before $e _{h}$'s $i$-th mention $m _{i}^{h}$ as the mention-level attention $\\boldsymbol{a} _{m _{i}^{h}} \\in \\mathbb{R}^{N _{t}}$. Then we average the mention-level attention of each mention of entity $e _{h}$ to obtain the entity-level attention $\\boldsymbol{a} _{e _{h}} \\in \\mathbb{R}^{N _{t}}$: $\\boldsymbol{a} _{e _{h}}=\\frac{1}{N _{e _{h}}} \\sum _{i=1}^{N _{e _{h}}} \\boldsymbol{a} _{m _{i}^{h}}$, where $N _{e _{h}}$ is the number of mentions of entity $e _{h}$. The calculation process of $\\boldsymbol{a} _{e _{t}}$ is similar.\n\n> (2) \"$a _{i}^{(h, r, t)}$ in Eq.5 is not in bold while $\\boldsymbol{a}^{(h, r, t)}$ in Line235 is in bold. Are these two symbols the same, or what are the differences between them?\"\n\nThank you for the valuable feedback. These two symbols are not the same. As literally shown, $a _{i}^{(h, r, t)}$ in Eq.5 has an additional subscript \"$i$\" compared to $\\boldsymbol{a}^{(h, r, t)}$ in Line 235. In terms of meaning, the latter is a vector ($\\boldsymbol{a}^{(h, r, t)} \\in \\mathbb{R}^{N _{t}}$), representing the instance-level attention over all tokens in the document; while the former is a scalar ($a _{i}^{(h, r, t)} \\in \\mathbb{R}$), which is precisely the $i$-th dimension of $\\boldsymbol{a}^{(h, r, t)}$ and represents the instance-level attention value on the $i$-th token in the document.\n\n> (3) \"An analysis, which is about how the issue of NOTA is tackled, should be added to support the claim.\"\n\nThank you for the helpful suggestion. We have further explored the impact of task-specific NOTA prototype generation strategy on the performance improvements. We divide the episodes of in-domain 3-Doc setting in ReFREDo benchmark into disjoint subsets according to the NOTA rate of each episode, i.e., the proportion of NOTA entity pairs to the total number of entity pairs in the query document of an episode. We establish four subsets, corresponding to the scenarios where the NOTA rate falls within [0%, 95%), [95%, 97%), [97%, 99%) and [99%, 100%], respectively. Then we evaluate models trained with or without task-specific NOTA prototype generation strategy on each subset. The experiment results are shown in the table below (where 'NR' refers to NOTA rate and 'TNPG' refers to task-specific NOTA prototype generation strategy). It is observed that the task-specific NOTA prototype generation strategy has brought improvement on each subset. And more importantly, the performance gain gets larger as the NOTA rate increases. It demonstrates that by incorporating task-specific NOTA prototype generation strategy, the derived NOTA representations conduce to capture the NOTA semantics, especially in those episodes with more NOTA query pairs involved.\n\n| Model         | NR $\\in$ [0%, 95%) $F _1$ | NR $\\in$ [95%, 97%) $F _1$ | NR $\\in$ [97%, 99%) $F _1$ | NR $\\in$ [99%, 100%] $F _1$ |\n| ------------- | :-----------------------: | :------------------------: | :------------------------: | :-------------------------: |\n| RAPL w/o TNPG |          $23.11$          |          $18.65$           |          $17.11$           |           $5.55$            |\n| RAPL          | $23.51$ ($\\uparrow 0.40$) | $19.12$ ($\\uparrow 0.47$)  | $17.76$ ($\\uparrow 0.65$)  |  $6.66$ ($\\uparrow 1.11$)   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.The proposed issues worth investigation for Few-shot Doc RE and the proposed method seems reasonable.\n\n2.The proposed method achieves great performances on the public benchmarks.\n\n3.It is glad to see a Preliminary Exploration of LLM for FSDLRE.", "reasons_to_reject": "1.Some technical details are not clear enough. Please refer to Question to authors for details.\n\n2.An analysis, which is about how the issue of NOTA is tackled, should be added to support the claim.", "questions_to_authors": "1.The computation process to $\\bm{a}_{e_h}$, $\\bm{a}_{e_t}$ in Line210 should be specified.\n\n2.$a^{(h,r,t)}$ in Eq.5 is not in bold while $a^{(h,r,t)}$ in Line235 is in bold. Are these two symbols the same, or what are the differences between them?", "paper_topic_and_main_contributions": "This paper studies the task of Few-shot Document-level Relation Extraction, where two issues are explored:1) how to refine relation prototype representations which are distinct to the confusing ones. 2) how to customize the representations of NOTA types for specific tasks. To tackle the issues above, Relation-Aware Prototype Learning method (RAPL) is proposed. For the first issue, the semantics of relations are used as guidance and the prototype is aggregated from all the supporting instances. Besides, a relation-weighted contrastive learning module works to further refine the prototypes. For the second issue, a set of learnable base NOTA prototypes are derived to produce the task-specific prototypes. Experiments on the public benchmarks in two scenarios demonstrate the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None", "Justification_for_Ethical_Concerns": "None"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is well-written and proposes an important research question. The experiment results show that their method is able to solve this problem.
Decision: Accept-Main