HyperNetwork-based Decoupling  to Improve Model Generalization for Few-Shot Relation Extraction
Authors: Liang Zhang
Chulun Zhou
Fandong Meng
Jinsong Su
Yidong Chen
Jie Zhou
Venue: EMNLP 2023 Main (2023)
Keywords: Relation Extraction
Few-Shot Relation Extraction

Abstract: Few-shot relation extraction (FSRE) aims to train a model that can deal with new relations using only a few labeled examples. Most existing studies employ Prototypical Networks for FSRE, which usually overfits the relation classes in the training set and cannot generalize well to unseen relations. By investigating the class separation of an FSRE model, we find that model upper layers are prone to learn relation-specific knowledge. Therefore, in this paper, we propose a HyperNetwork-based Decoupling approach to improve the generalization of FSRE models. Specifically, our model consists of an encoder, a network generator (for producing relation classifiers) and the produced-then-finetuned classifiers for every N-way-K-shot episode. Meanwhile, we design a two-step training framework along with a class-agnostic aligner, in which the generated classifiers focus on acquiring relation-specific knowledge and the encoder is encouraged to learn more general relation knowledge. In this way, the roles of upper and lower layers in an FSRE model are explicitly decoupled, thus enhancing its generalizing capability during testing. Experiments on two public datasets demonstrate the effectiveness of our method.

-yperNetwork-based Decoupling to Improve Model Generalization for Few-Shot Relation Extraction
Liang Zhangl 12" Chulun Zhou² Fandong Meng
Jinsong Su
Yidong Chen 1.2 and Jie Zhou'
"School of Informatics, Xiamen University, China
Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China Patttern Recognition Center, WeChat AI, Tencent Inc, China 1zhang@stu. xmu. edu.edd.com cn, chulunzhouutencent.ccm com, ejssu, eemmu edu. cn
Abstract
Classifier
Classifier M
Few-shot relation extraction (FSRE) aims to train model that can deal with new relations using only a few labeled examples. Most ex- isting studies employ Prototypical Networks for FSRE, which usually overfits the relation classes in the training set and cannot gener- alize well to unseen relations. By investigat- ing the class separation of an FSRE model, we find that model upper layers are prone to learn relation-specific knowledge. Therefore, in this paper, propose HyperNetwork- based Decoupling approach to improve the gen- eralization of FSRE models. Specifically, our model consists of an encoder, a network gen- erator (for producing relation classifiers) and the smnerated-then-ffnttneee classifiers for ev- ery N-way- - Khshot episode. Meanwhile, we design two-step training strategy along with class- aaaso-an aligner, by which the generated classifiers focus on acquiring colation-specifiie knowledge and the encoder encouraged to learn more general relation knowledge. In this way. the roles of upper and lower lay- ers in FSRE model are explieitly decou- pled, thus enhancing its generalizing capabil- ity during testing. Experiments on two public datasets demonstrate the efffectiveness of our method. Our source code is available at https: //gi thub. . com/ http//eeepeannMM/SREEEEE.
Classifier
generate
Generator
Enceder
Encoder
Episode
Episods M
Episode 1
..
Episode M
Protutypical Network
Our Model
Figure Illustration of the difference between conven-
tional Prototypical Network model and our model.
of labeled instances and enhance the generalizing capability of FSRE model to new relations.
Current FSRE studies are usually conducted in an N-way-K-shee setting. In this setting, the model is trained on a series of episodes, each of which has N relation classes. In each episode, ev- cry relation class includes K support instances (as support set) and Q query instances (as query set). Under this setting, most existing studies (Gao et al.. 201 9a; Yang et al., 2020; Zhang and Lu, 2022) em- ploys Prototypical Network (Snell et al., 2017) for FSRE. The Prototypical Network is designed learn a suitable prototypical vector for each rela- tion using the support instances from the support set. Then, these vectors are used to predict the relation of the query instances. Moreover, to fur- ther improve the performance of the model, some recent studies (Han et al., 2021at la; Liu et al., 2022) introduce relation description and adopt contrastive learning in Prototypical Network. Although these methods have achieved improvements, their mod- els still tends to overfit relation classes appearing in the training set, and exhibit an unsatisfactory generalizing capability to unseen relations.
1 Introduction
Relation extraction is a fundamental task in infor- mation extraction that aims to identify the semantic relations between two entities in sentences (Zeng et al., 2020; Han et al., 2020). Typically, conven- tional approaches are highly dependent on large amount of labeled data and cannot deal with unseen relations well. Therefore, recent studies (Han et al., 2018: Gao et al., 2019c) turn to Few-Shot Rela- tion Extraction (FSRE) that only requires a handful
To further explore what restricts the generaliz- ing capabilities of Prototypical Network models, inspired by Kornblith et al. (2021), we conduct a preliminary study to investigate the internal repre- sentations within different model layers. Specifi- cally, we measure the layer-wise class separation
This work is done when Liang Zhang was interning at Pattern Recognition Center, WeChat AL, Tencent Inc, China. "Equal contribution
"Corresponding author

---

between representations of instances belong into different relation classes during training. We ob- serve that the representations present higher class separation in the upper layers of the model than those in the lower layers. This intuitively shows that the upper layers are likely to learn relation- specific knowledge and overfit the relation classes in the training set, while the lower layers basically acquire more general relation knowledge. We spec- ulate that this addatationn upper layers limits the generalizing capability of the model.
of these M episodes.
During testing, when there is an episode contain- ing unprecedented relation types not existing in the training set, we use the generator to produce a fresh classifier that is then fine-tuned by samples from the support set. Finally, only the encoder and the fine-tuned classifier are used to predict the relation of each query instance. By doing so, the model can quickly adapt to episode.
However, as each updating interval contains only handful of episodes, so that our model is still prone to bias towards relation classes in these episodes. To address this issue, we additionally design a class-agnostic aligner, which undergoes all episodes in the training set throughout the train- ing process. Thus, with the help of the aligner, our encoder is able to learn more global general rela- tion knowledge, further alleviating the overfitting to specific relations.
Based on the above finding, in this paper, we pro- pose a HyperNetwork-basee Decoupling approach for FSRE. As illustrated in Figure I, our model consists of three components: an encoder at the bottom, network generator in the middle (for gen- erating the initialized relation classifiers) and the generated relation classifiers the top. In each episode, the network generator takes in the encoder output and generates an initialized relation classi- fier for current episode. Subsequently, the gener- ated classifiers are fine-tuned, so that the model can quickly learn relation-specific knowledge and adapt to new relations. Meanwhile, the encoder and generator are encouraged to learn more general relation knowledge from the training set. In this way, we explicitly decouple the roles of the lower and upper components in FSRE model.
Experimental results on two public benchmarks show that our model consistently outperforms all competitive baselines. Extensive ablation and case studies demonstrate the effectiveness of various components in our model.
2 Preliminary Study
To better understand the limited generalizing capa- bilities of current FSRE models, inspired by (Ko- mblith et al., 2021), given a BERT-based Proto- typical Network model, we first investigate layer- wise class separation between representations instances belonging to different relations. Specifi- cally, the class separation indicates the dispersion of representations belonging to the same class rela- tive to the overall dispersion of all embeddings. In each N-way-K K-shot training episode, we measure the class separation in every layer of the model following the metrics in (Kornblith et al., 2021). Particularly, we denote the average within- class cosine distance and the overall average cosine distance in the 1-th layer Z'n, and doveratt re- spectively. They are calculated using the following equations:
To this end, our model update procedure consists of two steps: the fine-tuning of the classifier and the update of the encoder together with the generator. For the classifier fine-tuning, we first use the gen- crator to produce an initialized classifier for every episode to discriminate the relation classes within them. Then, we use corresponding support set to optimize its generated classifier, which endows the classifier with relation-specific knowledge. Con- sequently, model can effectively adapt to new relations through such produce-then-finetue pro- cess. Moreover, the update of the encoder and generator resorts to learning more general knowl- edge not specific to a certain episode, which is crucial for an FSRE model not to overfit. To better learn such knowledge, we train the model across different episodes so that it is not biased towards narrow set of relations. Particularly, we treat collection of M sampled episodes as an updating interval for the encoder and generator. In each updating interval, the encoder and generator are jointly optimized through maximizing the overall performance of all trained classifiers on the query
N
K
K
1
sim n.i: bl.s))
drn
Σ
(1)
NK²
R=1
N
N
K
K
sim h/ns))
dovcrali Σ
ΣΣΣ
N2K2
where h is the embedding of the i-th instance (from the n-th relation class) in the I-th layer, and

---

Training Validation /
Class Sqparstion (Rop))
A-way-K-shot Sampling
0.6
Eplsode Query sec:
Support set:
0.4
Rel Tassace
Instaare X
Rel
Bostance
124
Rel N:
Instance
Irotance X
Rel N:: Iestane
Iustance
126
Context Sentence x: Parias was son of Philomelus. Relation Lahel y: Father
1=10
Instance:
0.0
Entities e: (Parias, Philomelus)
2500
12500
15000
Step. Number
Figure 3: The illustration of sampling an N-way-K shot episode from training/validationeessss set.
Figure 2: Layer-wise class separation Sept in 12-layer BERT-based Prototypical Network FSRE model with respect the training steps.
and test sets, i.e., relation classes for test do not exist in the training set. As shown in Figure 3, a sampled episode contains N relation classes, each of which has K instances (as support set S) and Q instances (as query set Q). Meanwhile, each instance (z, v) in the episode is comprised of a context sentence æ, two entities e- (=(e, et), and relation label y. where e,, and e; refer to head entity and tail entity, respectively.
si is the cosine similarity. Then, d &overall represents the relative within-class variance, higher de- gree of class separation the 1-th layer. Thus, the degree of class separation can be defined as Sep' /
lower value of which corresponds
As explored Kornblith et al. (2021), higher class separation means the model tends overfit the classes in the training set and is less transferable to unseen classes. Hence, we depict the layer-wise change of class separation with respect to the num- ber of training steps in Figure 2. We can observe that Sep' in upper layers (e.g., Layer-12 12 and Layer- 10) quickly increase early steps and consistently stay much higher than those in lower layers (e.g.. Layer-1 and Layer-2). Therefore, Figure shows that the upper layers of the Prototypical Network model are likely to learn relation-specific knowl- edge and overfit the relation classes during training,
3.2 Model Architecture
As shown in Figure 4, except the aligner, our model consists of three components: encoder at the bottom, network generator in the middle and the generated classifiers the top.
Encoder. Following existing studies (Soares et al., 2019; Han et al., 2021a; la; Liu et al., 2022), BERT (Devlin et al., 2019) as the encoder E of model encode both instances and lation descriptions. Specifically, for each instance, we first use four special tokens "[PI][/P]] and "[P2][/P2]" to mark the start and end positions of the head and tail entities within an instance, respec- tively. Then, the encoder is employed to obtain a contextual representation for each token. Finally, concatenate the representations of these tokens (i.e., [P1] and [P2)) at the start positions of the head and tail entities form the representation of this instance:
3 Methodology
In this section, introduce HyperNetwork- based Decoupling approach for FSRE. Specifically, we first provide the problem formulation of FSRE and give a detailed description of our model archi- tecture. Then, we elaborate the two-step training strategy of our model. Morcover, we additionally design alass-agnostic aligner to further enhance the generalizing capability of our model. Finally, we describe how our model is used for test in N-way-K Kssot setting.
[hp]: hp2).
Moreover, as in (Liu et al., 2022; Li and Qian, 2022), we also encode relation descriptions and pro- duce corresponding relation representations with the same dimensions instance representations. Specifically, the representation of the relation de- scription is obtained by concatenating the repre- sentation of the [CLS] token and the average rep- resentations of the remaining other tokens: r [hcus; havg)
3.1 Problem Formulation
Current FSRE studies are usually conducted in an N-way- Khot setting. In this setting, the model trained and tested on a series of episodes, each of which is sampled from mutually exclusive training

---

Produce Initialiaed Clessifiers
Pine suning Classifters and Aligner
Optimizing Enender & Generator
Low
lie
M
m 20 0
Lusign
C
C
Gu
Firestning
Initialize
C
CNs
Aligner
Aligner
Generator
Generator
Encoder (E)
Encoder (B)
Enceder (E)
Encoder (E)
Episods
N-ag--Kssoot
Episode
N-way-K-shh
Support
xx
Query set
xQ
Query set
XQ
S
Support set
XX
Rel Desc
[P]]ea[P1]... [22],
[P]]e[[P]]... [P2]e.(12)
Rel Desc
[P[]e_[P]]] [role//e)]
[P]]eA[01]
Episode M
Query Nowgy-K-shor
Episode M
XQ Support N-mny-R-shot
Support
Query
xX
Rel Dose
[P]]a[[P1]... ([le/
[P]6 [P3]e[
Rel Desc
[Pile ... (esler
[Ptle (P2)
(b)
Figure 4: The architecture and two-step training framework of our model. In each episode, for the first step, the network generator of our model produces an initialized N-way relation classifier subsequently fine-tuned by the classification loss Teg (i.e., Cm)) Meanwhile, the class-agnostic aligner trained through the contrastive loss hatign. For the second step, the encoder and generator are jointly optimized by Ie and batign. Gray indicates frozen modules in each step.
Generator & Generated Classifier. Unlike pre- vious FSRE models, our classifier used to discrim- inate relation classes is not continually optimized across the episodes during training or testing, but is freshly produced for each episode. Concretely, we use Multi-Layer Perceptron (MLP) as the net- work generator, which serves as a hypernetwork that aims to produce an initialized N-way relation classifier for each episode. In this way, we can ex- plicitly decouple the upper layers (classifiers) and lower layers (encoder) of our model. Through the two-step training strategy that will be elaborated Section 3.3, the entire encoder of our model, not just its lower layers, is encouraged to focus on learning general relation knowledge not specific to certain relations in the training set. Specifically, in each episode with N relation classes, the generator takes as input the representation R [r..... rx] of all relation descriptions to generate the initial- izing parameter of a customized relation classi- fier. The generated classifier is single linear layer used to predict the relation class of instances in the episode. Notably, unlike previous hypernetwork- based methods (Ha et al., 2016) where the gener- ated module is directly used to make prediction, our produced classifiers are further fine-tuned before actually being used.
knowledge while the encoder and generator learn more general relation knowledge. To this end, we separate the training of our model into two steps: the fine-tuming tne-tuning of the generated classifiers and the updating of the encoder and generator. Suppose we first sample M episodes from the training set and use the generator to produce an initialized relation classifier for each episode, i.e., CI....,Ca..
Step 1: Classifier Fine-tuning. Let us denote the generated classifier in the m-th episode as C.. the classifier is fine-tuned using the corresponding support set (See @ in Figure 4(a)). The training objective of C,, is formalized as the following clas- sification loss:
Len 10 - Σ
Yýlog(ūj), (Z).
(2)
új = -m = oo ( cm hj + bem):
where Sm refers to the support set of the m-th episode, 0z is the parameter of C. produced by our generator (i.c., W and b,.)). In this way, the relation-specific knowledge in the m-th episode
can be effectively learned by the classifier, resulting in a set of M fine-tuned classifiers C,, S..... ,CMs-
3.3 Two-step Training
Step Encoder & Generator Updating. Af- ter obtaining all fine-tuned classifiers (i.c., C,,"..CM we optimize our encoder and gen-
Based on the above model architecture, we expect the generated classifier to acquire relation-specific

---

erator using the following objective:
where h stands for instance representations ob- tained from the E, hy refers to the representation of the positive query instance 2i belonging to the same class as support instance 24- Thus, corre- sponding to Section 3.3, the overall objective of the first-step training can be written as
M Cemm Come
C.(02; 04)
(3)
cum
yplog(Cm(1z)), (.9.8))E
where 2m denotes the query set of the m-th episode, Len. is the classification loss computed by Cm. 0g and gg represent the parameters of the encoder and generator, respectively. Since all fine- tuned classifiers have been updated at the first step, the parameter deviation disables the gradient di- rectly backpropagated the generator. To solve this, we adopt a trick that denotes the classifiers in an equivalent form, i.e., c... of the trained classifier Cm is denoted as cc... - the help of above loss, the encoder and generator are optimized to enhance the overall classification performance across multiple episodes. By doing so, they encouraged to learn more general relation knowledge specific certain relations.
C1
Σ frim (0cm) Calign (0.).
(5)
For the second-step training, the aligner better encourages the encoder to learn general relation knowledge using similar contrastive loss:
M
Σ C.m=
Lalign (0g) = M
(Ôc.. & With
m
(6)
A(hrr)
C Σ
F.EQm. Layes, sim A(h,)),Ah))
Thereby, the overall objective the second-step training is formulated as
3.4 Class-Agnostic Aligner
L2 Co 0g) + BLalign (0z),
(7)
Although the above two-step training strategy can already enchance the generalizing capability of our model, there are still only a limited number of episodes involved the update the encoder and generator, as in Equation Therefore, the encoder is still possible to overfit the currently sampled M episodes. To avoid this problem, we additionally design global class-agnostic aligner A that is an MLP layer. Unlike the classifiers, the parameters of A are just randomly initialized and then con- tinually optimized across all episodes during the whole training, not freshly produced by the gen- erator in every episode. In this way, the aligner can further encourage the encoder to learn more global knowledge, alleviating its overfitting to the relations within currently sampled M episodes. Specifically, we first use our encoder E and its exponential moving average (EMA) counterpart E obtain the representations of query and support instances in each of the sampled M episodes. Then, along with the fine-tuning of the generated classi- fiers, the aligner A is simultancously trained using a contrastive loss as follows:
where since both loss terms update the encoder, we balance them using a hyperparameter 3.
3.5 Inference
During testing, for new episode from test set, we first use the generator to produce a freshly initial- ized classifier. Then, the classifier is fine-tuned using the support set to quickly learn the relation- specific knowledge within current episode. Finally, only the encoder and the fine-tuned classifier are used to predict the relation of each query instance.
Experiment
4.1 Datasets
Our model is evaluated on two commonly-usee datasets:
FewRel 1.0 (Han et al., 2018). It is a large- scale human- annotated FSRE dataset structed from Wikipedia articles, containing 100 relations. There are 700 instances in each relation. The training, validation and test sets contain 64, 16 and 20 relations, respectively.
1 M
Caligu (0
Σ
Lam
M
FewRel 2.0 (Gao et al., 2019c). To evalu- ate the generalizing capability of our model, we also conduct experiments on Few 2.0, whose training set is the same as FewRel 1.0.
(4)
sim (((h:), A(h+))
Lam-
I.ES Σ En sim sim,)(h)))

---

Particularly, the test set of FewRel 2.0 is con- structed from the biomedical domain (no over- lap with relations in training set) with 25 re- lations, each which contains 100 instances.
87.25
57.00
86.75
Accuracy (%)
$6.50
In each of these two datasets, the training, valida- tion and test sets contain mutually exclusive rela- tion classe sets.
$6.00
65.75
55.50
4.2 Settings
0.0
0.05
0.1
0.15
0.2
0.25
0.5
1.0
10.0
previous FSRE studies, there are mainly two types of model settings, differing in whether the model encoder is udditionally pre-trained using noisy Relation extraction(RE) corpus provided by Zhang and Lu (2022). In our experiments, in previous work, initialize the encoder of our model. For opti- mization, AdamW (Loshchilov and Hutter, 2019) is used with a linear warmup (Goyal et al., 2017) for the first 10% steps. The learning rates of the en- coder and generator are set to le-5, while those of the Aligner A and the generated classifiers are set to 4c-5 and le-2, respectively. Following previous work, set the number of sampled episode M every training step 4. It takes about ten hours for the whole training on single 24 GB NVIDIA RTX 3090 GPU. For testing, we use classification accuracy as the performance metric.
Figure 10-way-1-shot accuracy of our models with different values of FewRel 1.0 validation set.
2022), a label prompi dropoul method that effec- tively exploits the relation description, and filters the pre-training data of CP to conduct more rigor- ous few-shot evaluation.
BERT (Devlin et al., 2019)
4.4 Effect of Hyper-parameters, |
The 8 in Equation 7 is an important hyper- parameter, which balances the loss terms garding the optimization of our encoder. Thus, we conduct an experiment with different values of / on the validation set of FewRel I.0. From Figure 5, we observe that our model achieves the best perfor- mance when 8 is set to 0.1. Hence, we use B = 0.1 in all subsequent experiments.
4.3 Baselines
4.5 Main Results
We compare model with the following base- line methods: Proto-BERT (Snell al., 2017), a BERT-based prototypical network model. 2)
Results on FewRel 1.0. The experimental results on the validation and test sets of FewRel 1.0 are pre- sented in Table In the first part of the table, all the models directly use BERT to initialize their encoder without additional pre-training For these models, we observe that our model consistently outperforms all other contrast models on FewRel 1.0 dataset, es- pecially surpassing the strongest also fine-tunes relation classifier for each testing episode. Moreover, the models in the second part also use BERT initialize their encoder. Notably, they are additionally pre-trained using the noisy RE corpus before the training on Few-shot 1.0. To con- duct more rigorous few-shot evaluation, Zhang and Lu (2022) filters the relations in FewRel 1.0 from the noisy RE corpus. Among these models, our model performs significantly better than contrast models in all Kwwa-KKsshor settings.
MAML (Finn et al., 2017), a typical meta-lcarning method. 3) BERT-PAIR (Gao et al., 2019c), a similarity- based prediction method, in which each query instance is paired with all support instances. 4) REGRAB (Qu et al., 2020). a relation graph- based approach. 5) TD-Proto (Yang et al., 2020), a prototypical network model enhanced by entity description. 6) MTB (Soares et al., 2019), BERT- based model further pre-trained using additional matching the blank objective. 7) CP (Peng et al., 2020), an entity-masked contrastive pre-training framework for FSRE. 8) HCRP (Han et al., 20, an improved Proto-BERT with hybrid attention module and task adaptive focal loss. 9) Sim- pleFSRE (Liu et al., 2022), a prototypical net- work model enhanced by relation description. 10)
that
In addition, it is noteworthy that our model achieves greater gains on the more challenging I-shot setting than that on 5-shot setting. All the above results indicate that our model exhibits
GM GEN (Li and Qian, 2022), graph-based pro- totypical network model with similar testing pro- cedure as our model. 11) LPD (Zhang and Lu,

---

N- --way-K -ssoo
5-way- 1-shot w-ooy-I-soot
5-way-
shoot
10-way-1
1-shot
10-way-5-a0
5-shot
Models (w/o. pre-train)
test
val
test
val
test
val
test
Proto-BERT (Snell et al., 2017)
82.92
80.68
91.32
89.60
73.24
71.48
83.68
82.89
MAML (Fimn et al., 2017)
82.93
89.70
86.21
93.55
73.20
83.17
76.06
88.51
BERT-PAIR (Gao et al,, 2019c)
85.66
88.32
89.48
93.22
76.84
80.63
81.76
87.02
MTB (Soares ct al., 2019)
91.10
95.40
84,30
91.80
REGRAB (Qu ct al., 2020)
87.95
90.30
92.54
94,25
80.26
84.09
86.72
89.93
TD-Proto (Yang et al., 2020)
84,76
92.38
74.32
85.92
HCRP (Han et al., 2021a)
90,90
93.76
93.22
95.66
84,11
89.95
87.79
92.10
LPD (Zhang and Lu, 2022)
88.84
93.79
90.65
95.07
79.61
89.39
82,15
91.08
SimpleFSRE (Liu et al., 2022)
91.29
94,42
94.0S
96.37
86,09
90.73
89.68
93.47
GM GEN (Li and Qian, 2022)
92.65
94.89
95.62
96.96
86.81
91.23
91.27
94.30
Baseline (our)
87.04
91.03
91.82
93.87
80.07
84,98
86,95
91.13
Ours
93.35
95.21
95.94
97.19
87.41
91.59
91.71
94.54
Models (w/ pre-train)
val
lest
val
val
lest
CP (Peng et al., 2020)
88.29
90.85
92.77
95.60
80.50
83.89
88.61
90.61
HCRP-CP' (Han et al., 2021a)
90.89
94.23
92.90
95.77
83.17
89.69
86.43
91.94
LPD' (Zhang and Lu, 2022)
93.51
95.12
94.33
95.79
87.77
90.73
89.19
92.15
Baseline (our)"
91.13
93.71
94.36
95.93
86.11
90.31
89.55
92.49
Ours"
95.46
95.92
96.59
97.48
89.34
92.01
92.46
94.90
Table I: Accuracy (%) on the FewRel 1.0 validation / test set. "w/o. RE pre-train": the models without additional RE pre-training. "w/. RE pre-train": the models whose encoders are additionally pre-trained on the noisy RE corpus (Peng et al., 2020). To conduct more rigorous few-shot evaluation, Zhang and Lu (2022) filters relations contained in FewRE1.0 from the corpus. "f": the results are reported in (Zhang and Lu, 2022).
Model
5-way 1-shot
5-way 5-shot
10-way 1-shot
10-way
Model
5-way
10-way
(w/o. pre-train)
5-shot
(w/o. pre-train)
1-shot
1-shot
Proto-BERT
40.12
51.50
26.45
36.93
93.35
87.41
BERT-PAIR
67.41
78.57
54.89
66.85
w/o. Generator
92.06
85.82
ICRP
76.34
83.03
63.77
72.94
2 w/o, Two-Step Training
91.53
85.36
LPD
77.82
86.90
66.06
78.43
3 w/o, Generator & Two-Step
91.18
84,93
GM_GER
76.67
91.28
64.19
84.84
4 w/o, Aligner A
92.48
86.42
Ours
78.37
91.41
66.54
84.92
5 w/o, EMA
92.76
86.66
Table 2: Accuracy (%) the FewRel 2.0 test set.
Table 3: Ablation results FewRel 1.0 validation set.
higher generalizing capability that can better deal with data scarcity issue in harsher few-shot scenar- ios.
1.59 points in 5-way and 10-way settings, respec- tively. These results indicates that the generalizing capability of a shared classifier is limited, demon- strating the effectiveness of our network genera- tor. Particularly, our generator can generate a suit- able initialized relation classifier for each episode. Subsequently, the fine-tuning of these classifiers encourages the model to effectively Iearn relation- specific knowledge from only a handful of support instances, thus quickly adapting to unseen rela- tions.
Results on FewRel 2.0. As mentioned in Sec- tion 4.1, FewRel 2.0 is more difficult dataset whose training set and test set not only contain mu- tually exclusive relation classes but also come from different domains. From Table 2, we observe that our model still consistently outperforms all con- trast models. These results further demonstrate the superiority of our model.
(2) w/o Two-Step Training. During training, we separate the training of our model model into two steps: the fine-tuning of the generated classi- fiers and the update of the encoder and generator. To verify the effectiveness of this strategy, in this variant, we turn simultaneously optimize the classifier, encoder and generator in each episode. As illustrated in Line 2, this variant causes a sig- nificant performance decline. This suggests that simultaneously optimizing the classifier, encoder,
4.6 Ablation Study
We further conduct extensive ablation studies by removing different components of our model to comprehend their different impacts. We compare our model with the following variants in Table 3. w/o. Generator. this variant, we remove the network generator and employ shared classi- fier across all episodes. As shown in Line 1, this leads to a significant performance drop of 1.29 and

---

class separation in bottom-2 and top-2 layers (Lay- ers 1, .111 and 12) in Figure From the figure, we can observe that, lower layers (i.e. Layers and 1), the degree of class separation shows little difference between the PN model and ours. How- ever, in upper layers (i.e., Layers 11 and 12), the class separation of our model is consistently much lower and vary moderately during the whole train- ing process. This indicates that entire encoder, not just its lower layers, focuses on learning more general relation knowledge, thus exhibiting less bias towards relation classes in the training set. For the details about all the layer-wise class sepuration in our model, please refer to Appendix A.
0.8
Class Separttinn
0.6
1=1(P
(=2(0))
0.4
(-110M
(=12(P5)
(+0e))))
0.2
2(0ns)
10
0.0
7500
10000
12500
15000
17500
Step-Number
Figure 6: The comparison between conventional Proto- typical Network (PN) model and ours in terms of class separation See/ at lower and upper layers.
5 Related Work
and generator leads to the encoder and generator learning more relation-specific knowledge, thus over-adapting to the relations in the training set. (3) w/o. Generator & Two-Step. When we re- move the generator from model and simultane- ously optimize the whole model in each episode, the performance drops by 2.17 and 2.48 points in 5-way and 10-way settings, respectively (See Line 3). These results suggest that our generator and two-step training strategy both crucially contribute to the generalizing ability of our model.
Relation extraction(RE) is critical and funda- mental task in natural language processing (NLP), which aims to identify the semantic relations be- tween two entities within a given text (Xue et al., 2019; Han et al., 2020; Chen et al., 2022; Zhang et al., 2022, 2023b.a). However, conventional ap- proaches are highly dependent on a large amount of labeled data and cannot deal with unseen relation classes well. Therefore, recent studies (Han et al., 2018: Gao al., 2019c) turn Few-Shot Relation Extraction (FSRE) that aims to train a model to classify instances into novel relations with only a handful of training examples.
(4) w/o. Aligner A. The class-agnoste aligner A aims provide our model with more global gen- eral relation knowledge across the whole training process. To verify its effectiveness, we remove from our model and the performance also decreases (See Line 4). This suggests that the aligner A can indeed enhance model generalization.
Most existing studies (Gao al., 2019a; Yang et al., 2020: Zhang and Lu, 2022) employ Proto- type Networks for FSRE, which aims to learn a suitable prototypical vector for each relation using a handful of annotated instances. Gao et al. (2019h) employs an attention mechanism to enhance the ro- bustness of the prototype network to noisy data. Qu al. (2020) proposes Bayesian meta-learning method with external global relation graph model the posterior distribution of relational pro- totypes. Han et al. (2021b) focuses on enhancing the performance of Prototype Network on complex relations through an adaptive focal loss and a hy- brid network. Moreover, some studies (Yang et al., 2020; Wang et al., 2020; Han et al., 2021b) use supplementary information about entities and rela- tions, such as relation descriptions, to enhance the prototype vectors of relations. Despite impressive results achieved, these methods still tends overfit relation classes appearing in the training set, which limits their generalizing capability to new relations. In this paper, inspired by (He et al., 2020; Yin
(5) w/o. EMA. It is noteworthy that the EMA encoder E integrates the historical relation knowl- edge from previously learned episodes during the whole training. In this variant, the EMA operation is removed and the performance becomes inferior to Ours (See Line 5). indicates that, with the help of E, our aligner A is able to provide our encoder E with more global general relation knowledge in more efficient manner.
4.7 Comparison of Class Separation with Our Model
To further verify the effectiveness proposed HyperNetwork-bass decoupling approach, make comparison of the layer-wise class separa- tion Sep' between the encoder of conventional prototypical network (PN) FSRE model and that ours. For clarity, we only depict the degrees of

---

al., 2022), we propose HyperNetwork-basee decoupling method along with two-step training strategy to prevent overfitting of the FSRE model to the relations within the training set.
tional Natural Science Foundation of China under Grant Nos. 62076211, U1908216, 62276219, and 61573294.
On the other hand, some studies (Soares et al.. 2019; Peng et al., 2020; Dong et al., 2021; Wang et al., 2022) focus on further training pre-trained language models (PLMs) using noisy RE datasets. Soares et al. (2019) collect large-scale pre- training dataset and propose matching the blanks pre-training paradigm. Peng et al. (2020) proposes an entity-masked contrastive pre-training frame- work for FSRE. Wang et al. (2022) introduces three structure pre-training tasks to pre-train the large language model (GLM with 10B parameters), al- lowing it to better comprehend structured informa- tion in text. Unlike several other studies, Zhang and Lu (2022) introduces more rigorous few-shot evaluation scenario by filtering relations contained in FewRE 1.0 from the pre-trained corpus. Mean- while, they propose a label prompt dropout method to prevent the model from overfitting to the relation description. These methods are compatible with our model, which can provide our model with better pre-trained encoder.
Limitations
The limitations of our method mainly include fol- lowing two aspects: 1) Our method is only exam- ined on the FSRE task, while whether it is able generalize other tasks, such as intent classifica- tion and image classification, not yet explored this paper. 2) We did not consider scenarios where a query instance may not belong to any class in the support set.
References
Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng. Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. Knowprompt: Knowledge- aware prompt-tuning with synergistic optimization for relation extraction. In Proceedings of WWW.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of NAACL.
6 Conclusion
Manqing Dong, Chunguang Pan, and Zhipeng Luo. 2021. Mapre: An effective semantic mapping ap- proach for low-resource relation extraction. In Pro- ceedings of EMNLP.
In this paper, we propose HyperNetwork-basee Decoupling approach to improve the generalizing capability of FSRE models. Specifically, our model consists of an encoder, network generator (for generating relation classifiers) and the generated classifiers. Our generator aims to generate a prop- erly initialized relation classifier for each episode, allowing our model can quickly adapt to new re- lations. Meanwhile, we design a two-step train- ing strategy along with a class-agnostic aligner, in which the generated classifiers focus on acquiring relation-specn -peciffc knowledge while the encoder is encouraged to learn more general relation knowl- edge. In this way, the roles of upper and lower layers in an FSRE model are explicitly decoupled, thus enhancing its generalizing capability during testing. Experiments on two public FSRE datasets and extensive ablation studies show that our model consistently outperforms all competitive basclines.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of ICML.
Tianyu Gao, Xu Han, Zhiyuan Liu, and Maosong Sun. 2019a. Hybrid attention caatton--bbedd prototypical networks for noisy few-shot relation classification. In Proceed- ings of AAAI
Tianyu Gao, Xu Han, Zhiyuan Liu, and Maosong Sun. 2019h 2b19b. Hybrid siteetion-based prototypical networks for noisy few-shot relation classification. In Proceed- ings of AAAL
Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019c. Fewrel 2.0: Towards more challenging few-shot relation classifi- cation. In Proceedings of EMNLP.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter No- ordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Ac- curate, large minibatch sgd: Training imagenet in hour. arXiv preprint arviv 1006:1700.0000000 02677.
Acknowledgments
The authors would like to thank the three anony- mous reviewers for their comments on this paper. This research was supported in part by the Na-
David Ha, Andrew Dai, and Quoe V Le. 2016. Hyper- networks. arXiv preprint 1rXiv:1609.09106

---

Jiale Han, Bo Cheng, and Wei Lu. 2021a. Exploring task difficulty for few-shot relation extraction. In Proceedings of EMNLP.
Mengge Xue, Weiming Cai, Jinsong Su, Linfeng Song, Yubin Ge, Yubao Liu, and Bin Wang. 2019. Neural collective entity linking based on recurrent random walk network learning. In Proceedings of IJCAI.
Jiale Han, Bo Cheng, and Wei Lu. 2021b. Exploring task difficulty for few-shot relation extraction. In Proceedings of EMNLP.
Kaijia Yang. Nantao Zheng, Xinyu Dai, Liang He, Shu- jian Huang, and Jiajun Chen. 2020. Enhance proto- typical network with text descriptions for few-shot relation classification. In Proceedings of CIKM.
Jiale Han, Bo Cheng, and Xizhou Wang. 2020. Two- phase hypergraph based reasoning with dynamic re- lations for multi-hop kbqa. In Proceedings of IJCAI.
Li Yin, Juan Perez-Rua, and Kevin J Liang. 2022. Sylph: A hypernetwork framework for incremental few-shot object detection. In Proceedings of CVPR.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang. Yuan Yao. Zhiyuan Liu, and Maosong Sun. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of art evaluation. In Proceed- ings of EMNLP.
Shuang Zeng, Runxin Xu, Baobao Chang, and Lei Li. 2020. Double graph hased reasoning for document- level relation extraction. In Proceedings of EMNLP.
Kaiming He. Haoqi Fan. Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsu- pervised visual representation learning. In Proceed- ings of CVPR.
Liang Zhang, Zijun Min. Jinsong Su, Pei Yu, Ante Wang, and Yidong cheng. 2023a. Exploring effec- tive inter-encoder semantic interaction for document- level relation extraction. In Proceedings of IJCAL.
Simon Komblith, Ting Chen, Honglak Lee, and Moham- mad Norouzi. 2021. Why do better loss functions lead to less transferable features? In Proceedings of NIPS.
Liang Zhang, Jinsong Su. Yidong Chen, Zhongjian Miao, Min Zijun, Qingguo Hu, and Xiaodong Shi. 2022. Towards better document-level relation ex- traction via iterative inference. In Proceedings of EMNLP.
Wanli Li and Tieyun Qian. 2022. Graph-based model generation for few-shot relation extraction. In Pro- ceedings of EMNLP.
Liang Zhang, Jinsong Su, Zijun Min, Zhongjian Miao, Qingguo Hu, Biao Fu, Xiaodong Shi, and Yidong Chen. 2023b. Exploring self distillltlation based rela- tional reasoning training for document-level relation extraction. In Proceedings of AAAI.
Yang Liu, Jinpeng Hu, Xiang Wan, and Tsung- Hui Chang. 2022. simple yet effective relation informa- tion guided approach for few-shot relation extraction. In Proceedings of Findings of ACL.
Peiyuan Zhang and Wei Lu. 2022. Better few-shot relation extraction with label prompt dropout. In Proceedings of FMNLP.
Tlya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proceedings of ICLR.
Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan I. Liu, Maosong Sun, and Jie Zhou. 2020. Learning from context or names? an empirical study on neural relation extraction. In Proceedings of ACL.
Meng Qu Tianyu Gao, Louis-Pascal Xhonneux, and Jian Tang. 2020. Few-shot relation extraction via bayesian meta-learning relation graphs. In Pro- ceedings of ICML.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. Pro- ceedings of NIPS.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling. and Tom K Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning, In Pro- ceedings of ACL.
Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong. Jie Tang, and Dawn Song. 2022. Deepstruct: Pre- training of language models for structure prediction. In Proceedings of Findings of ACL.
Yuxia Wang, Karin Verspoor, and Timothy Baldwin. 2020. Learning from unlabelled data for clinical semantic textual similarity. Proceedings of Clini- caINLP.

---

0.8
Class Seporatin
06
1=6
0.
148
1=10
1w12
0.2
2500
10000
15000
Ntmber
Figure 7: Laser-wise class separation Sep' in our 12- layer encoder E with respect the training steps.
Appendix
A Layer-wise Class Separation of our model
In this section, we investigate the relation classes separation within our encoder in detail, as we done in the preliminary study section. As illustrated in Figure 7, the degree of class separation Sep' in each layer of our encoder remains stable through- out the training process. In particular, at 12-th layer, our encoder has significantly lower the class sepa- ration than the prototype network. This intuitively demonstrates that our training strategy can effec- tively train our entire encoder, not just its lower layers, to learn more general relation knowledge.


Reviews:
[{"review_id": "mITOwoKCfI", "reviewer": "Reviewer_TfK4", "summary": "", "questions": "1. Given that the experimental improvements are not that significant, how many times did you run the experiments?\n2. Please conduct statistical significant test with multiple experiment runs\n3. deepstruct is a popular and open-sourced paper. Why deepstruct was not cited or compared?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> $\\textbf{Q1}$ : This paper adopts existing components in FSRE, however, the novelty of applying previously proposed techniques is limited.\n\nThe contributions of this paper are summarized as follows:\n1. We propose a hypernetwork-based decoupling approach for FSRE, which uses a hypernetwork (the network generator) to explicitly decouple the lower layer (the encoder) and upper layer (the generated relation classifiers) of the model. As illustrated in Line 1 of Table 3, this decoupling model architecture can effectively enhance the generalization of our model to novel relations. **It is worth noting that** this hypernetwork-based decoupling architecture has not been explored in existing FSRE studies. Moreover, unlike previous hypernetwork-based methods where the generated module are usually not trainable, our generated classifiers are further fine-tuned before actually being used, which enables our model to quickly adapt to new relations.\n\n2. We propose a two-step training strategy along with a class-agnostic aligner, in which the generated classifiers focus on acquiring relation-specific knowledge while the encoder is encouraged to learn more general relation knowledge. In this way, we can effectively prevent the encoder from overfitting the relations in the training set, enabling our model to better generalize to new relations in the test set. This is effectively demonstrated in Figure 5 and Line 2 of Table 3. **As far as we know**, this training strategy is the first attempt in FSRE to alternately train the encoder and classifiers in a decoupling manner. \n\n\n\n> $\\textbf{Q2}$ : Compared with *GM_GEN*, the improvements of the proposed method are marginal. Please conduct statistical significant test and multiple experiment runs.\n\nAccording to the open-source code from *GM_GEN*, we find that *GM_GEN* employs distinct distance functions in different N-way-K-shot settings. To ensure a fair comparison, we adopt the same distance function as *GM_GEN* and re-report the performance of our model as follows:\n\n|   |    |    |    |    |\n| :--- | :----: | :----: | :----: | :----: | \n| Model | 5-way-1-shot | 5-way-5-shot | 10-way-1-shot | 10-way-5-shot |\n|  |val / test | val / test | val / test | val / test|\n| Baseline (our)| 87.04 / 91.03 | 91.82 / 93.87 | 80.07 / 84.98 | 86.95 / 91.13 |\n|   |    |    |    |    |\n| GM_GEN | 92.65 / 94.89 | 95.62 / 96.96 | 86.81 / 91.23 | 91.27 / 94.30 |\n| Ours-old | 93.35 / 95.21 | 95.94 / 97.19 | 87.41 / 91.59 | 91.71 / 94.54 |\n| ∆ | +0.70 / +0.32 | +0.32 / +0.23 | +0.60 / +0.36 | +0.44 / +0.24 |\n|   |    |    |    |    |\n| GM_GEN | 92.65 / 94.89 | 95.62 / 96.96 | 86.81 / 91.23 | 91.27 / 94.30 |\n| Ours-new | 93.75±0.14 / 95.53±0.05 | 96.86±0.21 / 97.55±0.10 | 87.80±0.19 / 91.78±0.07 | 92.50±0.27 / 94.89±0.13 |\n| ∆ | +1.10 / +0.64 | +1.24 / +0.59 | +0.99 / +0.55 | +1.23 /+0.59 |\n| $p$ value | 0.0036 / $-$ | 0.0027 / $-$ | 0.0041 / $-$ | 0.0023 / $-$ |\n|   |    |    |    |    |\n\nWe run experiments 5 times with different random seeds and report the mean and standard deviation on the FewRel 1.0 validation and test sets. Meanwhile, we perform statistical significance tests and report the corresponding $p$-values.\n\nAs shown in the table above, using the same distance function, our model significantly outperforms *GM_GEN* ($p$<0.01) by a larger margin (See ∆).\nMoreover, given the apparent discrepancy between our model and *GM_GEN*, we believe that the performance improvement of our model over our Baseline is more reliable indicator of the efficacy of our approach.\n\n\n> $\\textbf{Q3}$ : Why *Deepstruct* was not cited or compared?\n\nWe appreciate your reminder. In the revised version of our paper, we will cite this work.\n\nNext, we provide a detailed comparison between *Deepstruct* and our approach：\n1. The primary focus of *Deepstruct* and our approach are notably dissimilar. Similar to *CP* and *LPD*, *Deepstruct* focuses on further training pre-trained language models (PLMs) to improve its performance on structured prediction tasks. In contrast, our approach aims to prevent the PLM-based encoder from overfitting the relations in the training set, thus enabling our model to better generalize to new relations in the test set. Moreover, our approach usually is compatible with such methods that further train PLMs. As shown in the bottom cell of Table 1, our method can effectively enhance the performance of *LPD*.\n2. *Deepstruct* constructed its pre-training dataset in a similar manner to the FewRel 1.0 dataset, and did not exclude relations present in the FewRel 1.0 test set from its pre-training data. *LPD* has verified that the performance of the model will be overestimated under this setting. Instead, we follow *LPD* to conduct a more rigorous few-shot evaluation by excluding relations in the FewRE1.0 test set from the pre-training data. Consequently, it would be unfair to conduct a direct comparison between the performance of *Deepstruct* and that of our approach.\n3. *Deepstruct* focuses on enhancing the ability of large PLM (10B GLM) to understand structural information within text, while our method aims to enhance the generalization of small PLM (BERT-base) to new relations. The large gap in the number of parameters between *Deepstruct* and our approach makes a direct comparison between the two unfair.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper adopts a relatively new approach in the context of few-shot relation extraction.", "reasons_to_reject": "This paper adopts existing components in FSRE, however, the novelty of applying previously proposed techniques is limited. The experimental improvements of the proposed approach are also marginal, especially when comparing against GM_GEN. Besides, this paper did not cite the existing SOTA deepstruct, which also adopts relation pre-training. I think it is necessary to justify the reason for not citing this paper.", "questions_to_authors": "1. Given that the experimental improvements are not that significant, how many times did you run the experiments?\n2. Please conduct statistical significant test with multiple experiment runs\n3. deepstruct is a popular and open-sourced paper. Why deepstruct was not cited or compared?", "paper_topic_and_main_contributions": "This paper proposed a hypernetwork-based decoupling approach for few-shot relation extraction. This approach has demonstrated effectiveness in FewRel 1.0 and 2.0 datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "https://aclanthology.org/2022.findings-acl.67v2.pdf"}, "extra_text": {}}, {"review_id": "eODmEe14xm", "reviewer": "Reviewer_bDys", "summary": "", "questions": "Please see \"Reasons to Reject\" above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> $\\textbf{Q1}$: The contributions of this work are suggested to be explicitly highlighted in the introduction. The audiences need to take time to dig out them.\n\nThank you very much for your valuable suggestion. In the revised version of our paper, we will include our contributions within the introduction section.\n\n> $\\textbf{Q2}$: The differences between the proposed approach and “Sylph: A hypernetwork framework for incremental few-shot object detection” should be discussed. \n\nThank you for your reminder. In the revised version of our paper, we will provide a comparison between *Sylph* and our approach.\n\nThe differences between *Sylph* and our approach can be summarized as follows:\n1. *Sylph* focuses on the incremental few-shot object detection, where the model is required to adapt to novel classes without forgetting previously seen classes. Instead, our approach aims to enhance the model's generalization ability to novel relations in the test set by preventing the model from overfitting the relations (previously seen classes) in the training set.\n2. Although both *Sylph* and our approach employ hypernetwork, the structures of the two hypernetworks are notably distinct, and the modules generated by them also exhibit significant differences. *Sylph* employs a multi-layer convolutional neural network as a hypernetwork, and the classifiers generated by this hypernetwork are non-trainable. We apply a simple multi-layer perceptron as a hypernetwork, and the generated classifiers are further fine-tuned before actually being used. This trainable classifier enables our model to quickly adapt to new relations during testing. \n3. Our training strategy differs from that adopted by *Sylph*. *Sylph* first pre-trains the image encoder, and then freezes the encoder and train its hypernetwork to produce classifiers. Instead, we propose a two-step training strategy to train our model, where the classifiers and the encoder are alternately trained in a decoupling manner. As demonstrated in Figure 5 and Line 2 of Table 3, our training strategy can effectively prevents the encoder from overfitting the relations in the training set, enabling our model to better generalize to new relations in the test set. \n\n>  $\\textbf{Q3}$: Typos and Missing References\n\nThanks. We will fix them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.  the motivation of this paper is clear, and the proposed method seems novel to me. \n2.  this paper is easy to follow\n3.  extensive experiments with promising results", "reasons_to_reject": "1.  The contributions of this work are suggested to be explicitly highlighted in the introduction. The audiences need to take time to dig out them.\n\n2. The differences between the proposed work and the below work should be discussed. \n\nYin, Li, Juan M. Perez-Rua, and Kevin J. Liang. \"Sylph: A hypernetwork framework for incremental few-shot object detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "questions_to_authors": "Please see \"Reasons to Reject\" above.", "paper_topic_and_main_contributions": "Most existing approaches to few-shot relation extraction(FSRE) employ Prototypical Networks, which usually overfit the relation classes in the training set and perform poorly on unseen relations. This paper explores what restricts the model's generalizing capability and proposes a HyperNetwork-based Decoupling approach for FSRE. The model consists of three components: an encoder for obtaining the representation of relations and instances, a network generator for generating the initialized relation classifiers, and the generated classifiers to predict the relation of each query instance. With a two-step training strategy and a class-agnostic aligner, the proposed model alleviates the overfitting issue and exhibits higher generalization. Experimental results and ablation studies demonstrate the effectiveness of the method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 595: \"the roles of upper and lower layers in an FSRE model is explicitly decoupled\". Here \"is\" should be \"are\"."}, "extra_text": {"Missing_References": "Yin, Li, Juan M. Perez-Rua, and Kevin J. Liang. \"Sylph: A hypernetwork framework for incremental few-shot object detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022."}}, {"review_id": "DAECLL6u0q", "reviewer": "Reviewer_ePtg", "summary": "", "questions": "(1) How would the aligner itself perform if used as the few-shot classifier? I would be interested to see how well contrastive learning can do on few-shot classification tasks. \n\n(2) For the two-step optimization: it makes sense to do the theta_cm^ + (theta_cm - theta_cm^) trick (I imagine the second term has stop gradient), but if the embedding fine-tuning was already very good, doesn't this stop optimizing the generator at all (because theta_cm - theta_cm^ already did all the heavy lifting)? Would something like using gradient of the gradient make more sense (but of course it's way more complicated)?\n\n(3) I didn't find the details for how to fine-tune the classifier embedding (sorry if I missed it). Can you elaborate?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> $\\textbf{Q1}$: I would be interested to see how well the aligner and large language models perform on FSRE tasks.\n\nWe report the performances of the aligner and ChatGPT as follows:\n\n|   |    |    |    |    |\n| :--- | :----: | :----: | :----: | :----: | \n| Model | 5-way-1-shot | 5-way-5-shot | 10-way-1-shot | 10-way-5-shot |\n|  |val / test | val / test | val / test | val / test|\n| Aligner | 90.12 / 93.31 | 92.93 / 95.33 | 83.87 / 88.72 | 87.55 / 91.94 |\n| ChatGPT | 81.33 / 83.01 | 83.97 / 86.56 | 71.22 / 75.19 | 74.71 / 78.11 |\n| Ours | 93.35 / 95.21 | 95.94 / 97.19 | 87.41 / 91.59 | 91.71 / 94.54 |\n|   |    |    |    |    |\n\nwhere Ours denotes the performance of our model as reported in our paper.\n\nWe evaluate ChatGPT on 1,000 episodes randomly sampled from the FewRel 1.0 validation and test sets. Meanwhile, we investigate four commonly-used prompt templates and report the performance of the optimal template.\n\nAs shown in the table above, our model outperforms the aligner and ChatGPT, which further demonstrates the superiority of our method.\n\n\n> $\\textbf{Q2}$: For the two-step optimization: it makes sense to do the $\\boldsymbol{\\hat{\\theta}}\\_{\\mathcal{C}\\_m} = \\boldsymbol{\\hat{\\theta}}\\_{\\mathcal{\\widetilde{C}}\\_m}+(\\boldsymbol{\\hat{\\theta}}\\_{\\mathcal{C}\\_m} - \\boldsymbol{\\hat{\\theta}}\\_{\\mathcal{\\widetilde{C}}\\_m})$ trick, but if the embedding fine-tuning was already very good, doesn't this stop optimizing the generator at all? Would something like using gradient of the gradient make more sense?\n\nAt each model optimization, our generator first generates a new initial classifier for each episode, and then the generated classifier is fine-tuned on the support set of its corresponding episode. Finally, the generator is optimized using the loss of the fine-tuned classifier on the query set. Since the classifier is regenerated each time and fine-tuned on the support set (rather than the query set), this loss for training the generator is usually non-zero. Therefore, our generator can be optimized continuously.\n\nMoreover, we believe that using gradient of the gradient is an interesting idea. In future work, we will attempt to employ this approach. \n\n> $\\textbf{Q3}$: I didn't find the details for how to fine-tune the classifier embedding (sorry if I missed it). Can you elaborate?\n\nThe fine-tuning process of the generated classifier is described in detail on Lines 277 to 289 of our paper. Specifically, for each episode, we first use the encoder to obtain the representations of the instances and relation descriptions in the episode. Based on these representations of relation descriptions, we then employ the generator to produce an initial relation classifier for the episode. Finally, we fine-tune the classifier by utilizing its loss on the episode's support set (See Equation 3 in our paper).\n\n\n> $\\textbf{Q4}$: Typos and Missing References\n\nThanks. We will fix them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The proposed method is based on a sold preliminary observation, is well motivated, and is intuitive. \n\n(2) The experiment is very comprehensive and the results are very strong, especially that this method can still bring improvement over RE-pre-trained models.\n\n(3) The ablation study shows that every component is essential to the success of the method.", "reasons_to_reject": "(1) This is nitpicking, but it would be interesting to see how large language models would perform on such a task. Also, even with a small model like BERT, having a prompt would probably make the performance better.\n", "questions_to_authors": "(1) How would the aligner itself perform if used as the few-shot classifier? I would be interested to see how well contrastive learning can do on few-shot classification tasks. \n\n(2) For the two-step optimization: it makes sense to do the theta_cm^ + (theta_cm - theta_cm^) trick (I imagine the second term has stop gradient), but if the embedding fine-tuning was already very good, doesn't this stop optimizing the generator at all (because theta_cm - theta_cm^ already did all the heavy lifting)? Would something like using gradient of the gradient make more sense (but of course it's way more complicated)?\n\n(3) I didn't find the details for how to fine-tune the classifier embedding (sorry if I missed it). Can you elaborate?", "paper_topic_and_main_contributions": "This paper tackles the few-shot relation extraction (FSRE) problem (to be more specific, FewRel and FewRel 2.0 datasets). The task setting is that during test time, an episode of N-class K-shot data will be sampled and the model will be tested on the (previously unseen) N tasks. \n\nThe authors first conduct preliminary studies to show that in FSRE models, the representations at early layers are usually quite similar across classes but are more distinct at last layers. Then they propose to use a hypernetwork (an MLP) to generate the class embeddings (for the final linear classification), in contrast to conventional approaches like Prototypical Network (averaging class instances to get the class embedding). There are three key components of the proposed method: (1) the generator which takes the encoded relation description as input and outputs the relation embedding; (2) two-step training where they first fix the encoder and only fine-tune the relation embedding, and then then fix the embedding to fine-tune the generator and encoder; (3) an \"aligner\" which is used for a momentum-encoder-style contrastive learning.\n\nThe authors conduct extensive experiments on FewRel and FewRel 2.0 and demonstrate very strong results compared to a number of baselines. Even compared to models using RE pre-training, the proposed method is still very effective. There is also a comprehensive ablation study to demonstrate the effectiveness of each component. \n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The authors should cite MoCo since they are using contrastive learning + EMA: \n\nHe et al. Momentum Contrast for Unsupervised Visual Representation Learning\n\n", "Typos_Grammar_Style_and_Presentation_Improvements": "173: as explained in (Kornblith et al., 2021) -> as explained in Kornblith et al., 2021\n\nEquation (5): I believe it's exp(sim) instead of sim?\n\nTable 1: line 1 and line 3 have the same values?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This research paper introduces a novel approach to tackle few-shot relation extraction using a hypernetwork-based decoupling method. The model comprises three key components: an encoder, responsible for extracting representations of relations and instances; a network generator, which produces the initial relation classifiers; and the generated classifiers themselves, used to predict relations for each query instance. By employing a two-step training strategy and a class-agnostic aligner, the proposed model effectively addresses the problem of overfitting and demonstrates enhanced generalization capabilities.
Decision: Accept-Main