Density-Aware Prototypical Network for Few-Shot Relation Classification
Authors: Jianfeng Wu
Mengting Hu
Yike Wu
Bingzhe Wu
Yalan Xie
Mingming Liu
Renhong Cheng
Venue: EMNLP 2023 Findings (2023)
Keywords: Few-shot relation classification
None-of-the-above challenge
Density estimation
TL;DR: To address the none-of-the-above (NOTA) challenge in few-shot relation classification, we introduce NCA loss and NOTA loss to optimize the instance distribution, and design a novel density estimation method to detect NOTA.

Abstract: In recent years, few-shot relation classification has evoked many research interests. Yet a more challenging problem, i.e. none-of-the-above (NOTA), is under-explored. Existing works mainly regard NOTA as an extra class and treat it the same as known relations. However, such a solution ignores the overall instance distribution, where NOTA instances are actually outliers and distributed unnaturally compared with known ones. In this paper, we propose a density-aware prototypical network (D-Proto) to treat various instances distinctly. Specifically, we design unique training objectives to separate known instances and isolate NOTA instances, respectively. This produces an ideal instance distribution, where known instances are dense yet NOTAs have a small density. Moreover, we propose a NOTA detection module to further enlarge the density of known samples, and discriminate NOTA and known samples accurately. Experimental results demonstrate that the proposed method outperforms strong baselines with robustness towards various NOTA rates. The code will be made public after the paper is accepted.

Density-Aware Prototypical Network for Few-Shot Relation Classification
Jianfeng Wu'
Mengting Hu Yike Wi Bingzhe Wu*
Yalan Xie!
Mingming Liu² Renhong Cheng'
College of Computer Science, Nankai University, 2 College of Software, Nankai University 3 School of Journalism and Communication, Nankai University, Tencent Al Lab wjf@mail.. nankai. cnu
cn mthuênankai.c edu.ccn
Abstract
Support Sct
In recent years, few-shot relation classification has evoked many research interests. Yet a more challenging problem, i.e. none-of-the-above (NOTA), is under-explored. Existing works mainly regard NOTA as an extra class and treat it the same known relations. However, such a solution ignores the overall instance dis- tribution, where NOTA instances are actually outliers and distributed unnaturally compared with known ones. In this paper, we propose density-aware prototypical network (D-Proto) to treat various instances distinctly. Specifi- cally, we design unique training objectives to separate known instances and isolate NOTA in- stances, respectively. This produces an ideal in- stance distribution, where known instances are dense yet NOTAs have small density. More- over, we propose a NOTA detection module to further enlarge the density of known samples, and discriminate NOTA and known samples ac- curately. Experimental results demonstrate that the proposed method outperforms strong base- lines with robustness towards various NOTA rates.!
(1) contains
California is state in the United States.
(2) founded by
Steve Jobs was one of the founders of Apple.
Query Set
Rill Gates founded Microsoft in 1975
NOTA
Einstein was born in 1879.
Table example for 2-way 1-shot episode with 50% NOTA rate, where 2 indicates the number of classes and each class has one instance. The head entity marked in red, and the tail entity marked in blue.
recognized by reference to this support set. But for the second query instance, the relation between "Einstein" " and "1879" "born in".. It cannot find a reference from the support set.
Gao et al. (2019b) define such relation none- of-the-above (NOTA) and propose new task set- ting. Its challenge relies the accurate detection of both the known instances and NOTA ones. Previ- ous works extend N W-way to (N+1)- regarding NOTA relation as an extra class. Then the prob- lem is further solved by classification (Gao et al., 2019b), or computing distance with pre-defined learnable vectors (Sabo et al., 2021). or multiple choice (Liu et al., 2022). Despite their popularity and superiority, they only focus detecting NOTA effectively rather than refining the distribution of NOTA instances, leaving the overall distribution of all instances ignored. However, NOTA not a sim- ple (N + 1)-th class, which tends to be outliers for the known N relations. As an illustration, MNAV may lead some NOTA instances to cluster around a particular vector. However, the clustering of these instances from different categories together is un- natural. Therefore, motivated by this, we study from the view of instance distribution.
1 Introduction
Relation classification is a fundamental task in nat- ural language processing field, aiming to recognize the relation between two entities in a given sen- tence. Recently, many works formulate this task in the few-shot learning scenario (Dong et al., 2020; et al., 2020; Han al., 2021; Brody et al., 2021). They focus on obtaining fast adaptation with a few samples by learning multiple episodes. Usually, each episode is composed of a support set and a query set, where the support set has N-way K- shot instances. As an example shown in Table 1, the support set provides two known relations, i.c. "contains" and "founded by". The first query in- stance has "founded by" relation, which can be
MMengting Hu the corresponding author.
'The code is released on GitHub at https://githhbb com/Pisces 29/DProto.
In this paper, our initial research question is what

---

instances distribution is ideal. For known samples, intra-class needs be well clustered and inter- class is clearly separated. And NOTA samples, without gathering. should be distributed away from all known instances. In light of these properties, we propose a density-aware prototypical network (D-Proto) that takes into account the density of dif- ferent types of instances. Specifically, to guarantec the ideal instance distribution of known instances, we introduce neighborhood component analysis (NCA) (Laenen and Bertinetto, 2021), which in- volves all sample pairs out the support and query sets. For NOTA instances, we propose a NOTA loss, which aims at pushing them away from all non-NOTA ones. By making NOTA instances iso- lated, both of them become more distinguishable in the embedding space.
Staves Sample
Suppprt
Encodkr
Exalition
Querr
Zusting
NOSA Pelection
Sugport iassaro
The framework overview of D-Proto. In this example, the support set is 2-way 3-shot. During train- ing, two different query samples are for class and NOTA, respectively. During testing, queries are first predicted via distances with prototypes and further re- vised with NOTA detection module.
Figure 1:
the effectiveness and robustness of our ap- proach on both fixed and random episode sam- pling with various NOTA rates.
Moreover, the ideal distribution will lead to dis- tinct densities for NOTA and known samples. That is to say, known instances are distributed densely since the intra-class is clustered together. NOTA's density is small as it isolated from others. There- fore, we propose a NOTA detection module by fully enhancing the density properties. Concretely, to promote the density of known instances, we de- sign intra-class prototype enhancement. By adding prototypes and degenerative prototypes, their den- sities could be further enlarged. Based on this, we propose a prototype enhanced local outlier factor (PLOF) to calculate the outlier extent of an instance through distances with its neighbors. Finally, we propose base score that uses the extremal dis- tance ratio between query samples and N proto- types. By fusing PLOF and base score, the final NOTA score is obtained to accurately discriminate between NOTA instances and known ones. In summary, the main contributions our work are as follows:
2
Methodology
2.1
Formulation and Method Overview
Given the training set Dtrain, we sample multi- ple episodes, where each of them is composed of {8, Q, R}. The relation set R {r+, ..... r mentions N relations. The support set S follows N-way K-shot setting, which contains N relations, and each relation rd has K instances.
S = ((6)) ((6)))))), 1 N, sissss
where st denotes the j-th sample belonging the relation r² The query set Q is composed of two kinds of instances: known instances, which belongs to the relation set R of the support set; NOTA instances, without references from the support set.
@ = ((91,)), (4m, Ta), (71, ,),.., (Tm, F)}
We propose D-Proto to address the more chal- lenging few-shot relation classification task.
where ((gj, r;)} the known instances and {Q,, 7) are the NOTA instances. Here de- notes the NOTA relation. The query set Q is pre- dicted with the help of support set S. Our goal is to recognize all samples in Q accurately.
To the best of our knowledge, this work the first to model the overall instance distribution in an episode.
The proposed approach is depicted in Figure 1. A sample is encoded with BERT E-)) (Devlin et al., 2019), yielding the vector representation of [CLS]. In the embedding space, we design two training ob- jectives: 1) known sample separation Cnca aims to increase the separability of known samples by reducing intra-class distances and enlarging inter- class distances; 2) NOTA sample isolation E
We introduce the NCA objective and propose NOTA loss to achieve ideal instance distri- bution. Based on its idcality, we further pro- pose a prototype enhanced local outlier factor (PLOF) and a base score to fully leverage the density properties.
Extensive experimental results demonstrated

---

push NOTA samples be distinct from known ones, making them more easily identifiable as out- liers. Ultimately, our model is optimized by the combination of two training objectives.
Adding proratypus
C neca anooota
(1)
Adding degenerative
prototypes
where is hyper--aane papem--rramtter to balance the contri- bution of NOTA loss.
Support instances
In the testing phase, we propose NOTA detec- tion module, fusing PLOF and base score into NOTA score. Each sample's NOTA score then reveals its outlier extent. NOTA samples will be easily discriminated from known samples. Next, we will introduce each module in detail.
Prototypes
INYZA
Degmerativee protetypes
Query
Illustration of expanding instance sets for each relation PLOF. We specifically augment with prototypes and degenerative prototypes.
Figure 2:
2.2 Known Sample Separation
Although considering the overall instance distribu- tion is intuitive, its ideal is not casy to attain in the few-shot scenario. Due to the limitation of samples, a sample and its neighbors are all important for quantifying density. Therefore, we introduce neigh- borhood component analysis (NCA) loss (Laenen and Bertinetto, 2021). It considers the distances between sample pairs from both the support set and the query set. In other words, K samples of some relation are pulled together interactively and away from all other K x (N 1) samples.
2.3 NOTA Sample Isolation
Since NOTA samples are outliers for N known re- lation clusters, they should be isolated from known ones. Meanwhile, if query set has multiple NO- TAs. they might belong to different relations like the second query instance of Table 1. Aggregat- ing NOTA samples into the (N + 1)-th class may cause negative impact on distinguishing rela- tions. Therefore, we propose NOTA loss to push each NOTA instance away from all the known samples. The NOTA instances are denoted as Bnota = { (z;), r:) E B, = F). Lnota
Assume all instances in an episode are B { (zi, rx)((z;; ri) E {S U Q),15is)). then all the known instances are defined as B₁ (ai rs)((2)))s)) rs) E Br + T}. where 4; is a in- stance, ry is its relation, F represents the NOTA relation. NCA loss is formulated as below.
is defined as below.
Enota
Σ log VEI.....b n==
|Botal
-1
(3)
Enca
log iEL....
BB
Σ jEL.....
exp( exp(-d((xi, 2)),
Σ
exp(-dt (I,, x.2))
jCT....b
(2)
574
In this way, each NOTA sample is isolated from all others, which is beneficial for detecting known relations and NOTA via density.
:==r
Σ kcT.....b
exp(- (d2 (b:)
Aigh
FF
2.4 NOTA Detection
where x, E(x,) ) is the vector representation for the instance 24. -) represents the Buclidean dis- tance. [B&] indicates the number of samples in Bt.
To further quantify densities accurately for few- shot samples, we promote the testing phase with PLOF and base score. They are aggregated into the final NOTA score, which indicates an instance's s outlier degree through its nearest instances and pro- totypes. Next, we will introduce them individually.
With Lncar the intra-class clustered together and the inter-class is away from each other, making a good distribution for the density of known samples.

---

Wanot
NOTA
Closs
Roret
Neane
NOTA
Class
Support bustances
Procctypss
Query instanccs
Query indionees
Figure A 3-neighborhood example. The radius of each circle represents 3-distance.
Figure 4: An example of 2-way 3-shot scenario. For a known query, its score is much smaller than that of NOTA query.
Prototype Enhanced Local Outlier Factor (PLOF) Different from the naive prototypical net- work (Snell et al., 2017), we compute the sample pair distances as Eq. (2). However, during infer- ence, we argue that the prototypes can be leveraged to enlarge the density. Therefore, as depicted in Figure 2, we first calculate the prototype for each relation below and expand the relation's vector set with its prototype.
Then given a positive integer le (k [Bpior]). the k-distance dx(q) of vector q. defined as that for an instance are at least (I: - 1) instances of which holds that d(q, v) < d(q, where E Baos o}. Here the symbol \ indicates except. After find- ing such instance o, d4(4) d(q, o). The Eu- clidean distance between and all the chosen in- stances are less than or equal to dg(q). As shown in Figure 3, dashed circles surround the chosen instances for each target q. These instances called the k-neighborhood (g(g) (v E Bptof Va v) S de(g)).
E Bwoo\(e) there
B(s))
(4)
of q. denoted
To further boost the density and improve the identification of NOTA samples, we propose a spe- cial type of prototype, referred to as ddegenerative prototype", which is computed with degenerative set of instances.
Based on k-neighborhood. the local reachability distance Ird defined as follows:
[N((a)
Ird(q) EoEN. max(d)(()) o))
(6)
Σ E(s)),15m Il....
(5)
Obviously, Ird(g) is the inverse of the average reachability distance based the k-neighborhood of q. Based on it, we further define PLOF the average of the ratio of the local reachability density of q and those of q k-neighborhood.
K-
jmm
Intuitively, each degenerative prototype eri is a vector obtained by averaging (K -K-1) support embeddings belonging to its relation r' after re- moving support instance stm- As shown in Figure 2, the augmentation of prototypes and degenerative prototypes significantly increases the density of known instances and the outlier status NOTAs, thus improving the detection.
Ird(w) Tra(())
PLOF(g) = EnE (e(N
(7)
[Nx(4))
By considering the neighbors, PLOF better re- flects the density. Meanwhile, the higher the PLOF value of q, the more probable that q is an outlier (e.g. a NOTA sample).
Before calculating the PLOF of each instance, the original instance set B is expanded to Byosf B BUPUT UP..P.
Base Score Query instances are recognized with the help of support prototypes. We observe interesting property on the distances between the query instances and support prototypes. As de- picted in Figure 4, the non-NOTA query is close to its own prototype but far away from the other prototypes. Yet the phenomenon different for the
P = ( JiliiNN
DP = (crt) 1 N, m SSK
where P is the prototype set and DP is the degen- erative prototype set.

---

NOTA query, which is far away from all prototypes. This occurs because the proposed NOTA loss Entta pushes NOTA instances away from those known instances. Based upon this distance property, we propose a base score, which calculates the ratio of extremal distance to prototypes.
Random Sampling Sabo et al. (2021) propose fix the number of instances in the query set and assign probability p to each instance for sampling. If p is less than a threshold Tho- the instance is sampled as a known query. Otherwise, it is sampled as a NOTA. As such, the value of Tiur determines the NOTA rate. This sampling strategy results in dynamic NOTA rate for each episode. For example, some episodes may have NOTA rate of 100%, while others may not have any NOTA instances, resulting in more realistic simulation real-world scenarios.
min Elg,c"' i=
Scorebase (q)
llgg,
max !
This definition could also help to distinguish non-NOTA and NOTA samples. The base score of a known query smaller, while it is larger for a NOTA query.
3.3 Baseline Methods
NOTA Score Finally, the NOTA score is the com-
The following baseline methods are chosen for comparison. O-Proto (Tan et al., 2019) aims to solve few-shot out-of-domain detection. It is based on the prototypical network, which detects NOTA by cosine similarity. BERT-Pair (Gao al., 2019b) adopts BERT match a query instance and a support instance, which then yields a score indicating whether they share the same relation. MNAV (Sabo et al., 2021) is also based on the pro- totypical network. NOTA class is represented by some learnable vectors. A query instance is evalu- ated by its similarity with the NOTA vectors in the embedding space. MCMN (Liu et al., 2022) uses pre-training and fine-tuning paradigm that converts all candidate relation descriptions into multiple- choice prompts.
bination of PLOF and base score.
Score (q) = (L)F Scoretase (q)
(9)
During inference, query instance can be de- tected by comparing its NOTA score with thresh- old т. Since the proposed NOTA score is well- designed, using this score can more effectively recognize instances. In the real application, the threshold can be estimated with k-fold validation. In our experiments, we choose a fixed threshold for simplification.
3 Experiments
3.1 Datasets
FewRel 1.0 (Han al., 2018) is relation classi- fication dataset containing 100 relations extracted from Wikipedia. Gao al. (2019h) update it to FewRel 2.0 and propose the NOTA detection chal- lenge. Our experiments follow the splits used in official FewRel benchmarks, which split the dataset into 64 classes for training, 16 for validation, and 20 for testing.
3.4 Experimental Results
3.4.1 Overall Results
The results of two sampling strategies are presented in Table 2. It can be observed that D-Proto out- performs strong basclines and achieves the best average results on both sampling strategies. Espe- cially, D-Proto significantly outperforms MCMN by +2.75% on accuracy and +2.66% F1 under random sampling. The results indicate the effec- tiveness of D-Proto, with great stability and ro- bustness. As mentioned in $3.2, random sampling better reflects practical applications. It can be seen that under fixed sampling, the proposed D-Proto achieves competitive results. And under random sampling, the improvements more significant. This implies that D-Proto is more robust in real- world situations.
3.2 Episode Sampling
In 1 we introduce the N-way K-shot setting. Each episode sampling support set follow the N- way Khshot setting. But there are two ways to sample a query set.
Fixed Sampling Gao et al. (2019b) propose fixing the NOTA rate of query set to some spe- cific value. For example, if query set has 5 known queries and 5 NOTA ones, the NOTA rate of the query 5/(5 + 5) x 100% = 50%. Then the total num- ber of instances in that query set will be adjusted based on the NOTA rate.
would be calculated as
Moreover, under random sampling, we see that the accuracy and Fl of MCMN decrease by 13.36% and 16.05%, respectively when the NOTA rate is

---

Random Sampling
15%NOTA
30%NOTA
50%NOTA
Average
Methods
Acc
F1
Acc
F1
Acc
F1
Acc
F1
O-Proto (Tan et al., 2019)
80.02
72.84
80.55
73.73
81.16
72.41
80.58
72.99
BERT-Pair (Gao et al., 2019b)
80.77
73.99
81.53
75.06
83,24
75.14
81.85
74.73
MNAV (Sabo al., 2021)
85.02
78.81
83.67
77.78
81.59
73.94
83,42
76.84
MCMN (Liiu al., 2022)
88.24
83.54
82.50
76.39
74.88
67.49
81.87
75.80
ChatGPT
64.57
56.63
63.30
55.81
61 .666
54.16
63. 18
55.54
D-Proto (Ours)
85.76
79.79
84.80
79.27
83.30
76.32
84.62
78.46
Fixed Sampling
15%NOTA
30%NOTA
50%NOTA
Average
Methods
Acc
F1
Acc
F1
Acc
F1
Acc
F1
O-Proto (Tan et al., 2019)
81.38
80.45
81.42
81.04
81.38
79.39
81.38
80.29
BERT-Pair (Gao et al., 2019b)
82.83
81.98
83.76
83.00
85.49
82.62
84.02
82.53
MNAV (Sabo et al., 2021)
85.05
84.22
84.36
84.45
82.91
82.15
84.10
83,60
MCMN (Liu et al., 2022)
87.56
84.04
83.16
83.40
75.07
79.14
81.93
82.19
ChatGPT
63.59
56,26
61.83
58.41
51.14
61.28
53.95
D-Proto (Ours)
85.37
84.67
84.87
84.90
83.05
82.66
84.43
84.08
Table 2: Evaluation results of baseline methods and the proposed D-Proto, in terms of accuracy (%) and Fl (%), on the FewRel dataset. The setting is 5-way 5-shot.
set from 15% to 50%. On the contrary, D-Proto only sees decreases of 2.46% and 2.01% %, respec- tively. A possible reason is that MCMN needs to meta-adapt with the support set. Yet support set only has N-way K-shot samples, while NOTA ex- ists the query set. This tends to cause overfitting on the known instances but fails to detect NOTA. This further demonstrates the effectiveness and ro- bustness of the proposed D-Proto model toward various NOTA rates.
around 0.1. This leads to difficulty in choosing a good threshold since the range of proper thresh- olds is small. Thus, base scores alone may not be sufficient distinguish between the two classes accurately.
Secondly, without adding prototypes and degen- erative prototypes, we have the naive local outlier factor (LOF).. It can be seen that both LOF and PLOF show numerical overlap for both non-NOTA and NOTA instances. However, in LOF, the scores for non-NOTA instances are centered around 0.9, while the scores for NOTA instances range from 1.0 to 1.2. The available range of threshold is lim- ited. Contrarily, in PLOF, the scores for non-NOTA samples are in the range of to 3, while the scores for NOTA instances are significantly larger than 4. Finally, combining PLOF and base NOTA score forms our final NOTA score. is found that the NOTA score can make two types of instances have a clear difference. The gap between medians be- comes more obvious, and the NOTA score for the NOTA instance becomes larger. These all con- tribute the detection of NOTA instances, show- ing the effectiveness of our proposed NOTA score.
Furthermore, evaluate the performance of the ChatGPT2 dialog system in the NOTA chal- lenge. As demonstrated in Table 2, ChatGPT's performance on the NOTA task is far below other models. Therefore, we conclude that the model targeting the NOTA challenge in few-shot relation classification is still necessary. More details can be found in Appendix A.9.
3.4.2 NOTA Score Analysis
We further investigate the scores computed by D- Proto, which are depicted through box line dia- grams in Figure 5. We first observe that the base scores alone are able to distinguish between known instances and NOTA ones, but the numerical values for the two classes close. Specifically, the base scores of NOTA instances are generally greater than 0.4, while those of non-NOTA samples are
3.4.3 Ablation Study
To explore the impact of individual training objec- tives, we perform ablation study. The results are reported in Table 3. can be seen that removing Lnoto causes sigmificant decreases in all evaluation metrics. This verifies that NOTA loss effectively
ht tps: http://chaa..ppaa....cco
'In the context of the section, non-NOTA samples can he considcred equivalent to known ones.

---

15%NOTA
30%NOTA
50%NOTA
Average
Methods
Ace
F1
Acc
Acc
F1
Acc
F1
D-Proto w/o Cnota
84.95
78.73
84.04
78.27
82.72
75,41
83.90
77.47
D-Proto w/o Scorchase
78.44
70.95
78.23
71.39
77.52
68.99
78.06
70.44
D-Proto w/o
84.24
78.00
83,93
78.27
83.48
76.13
83.88
77.47
D-Proto w/o DP
85.40
79.20
83.39
77.55
80.32
73.01
83.03
76.58
D-Prote w/o {P UPUDP)
83.27
76.83
83.04
77.10
82.37
74.73
82.89
76.22
D-Proto 20%DP K-2
84.04
77.76
83,80
78.10
83.46
76.10
83.77
77.32
D-Proto 40%DDF
83.60
77.29
83,49
77.47
83.75
76.36
83.61
77.04
D-Proto 60%DPxx x-
83.35
76.99
83.63
77.88
83.88
76.49
83.62
77.12
D-Proto 80%DPK-2
83.19
76.81
83.58
77.83
84,01
76.62
83.59
77.08
D-Proto 100%DPx x-2
83.70
77.37
83.74
78.02
83.72
76.37
83.72
77.25
D-Proto(OUrs)
85.76
79.79
84,80
79.27
83.30
76.32
84.62
78.46
Table 3: Ablation study results and degenerative prototypes study (%) of D-Proto on FewRel benchmark.
Base
Factor
mance of the model? To answer this question, we design the K - 2 degenerative prototype set D- DPk-2 consists of all K - 2 degen- erative prototypes, where each one is computed by averaging the embeddings of the K - 2 sup- port instances belonging its relation r² after ex- cluding two support instances. It is worth noting that D-Proto only adds the degenerative prototypes DP Will including DP--2 bring further gains?
0.0
NOTA
PLOFF
3.8
NOTA Score
2.4
In this experiment, we add various proportions of DP into the support set. The results are displayed in the second part of Table Our expec- tation is that DPx-2 will increase the density of known samples and thus improve the performance. However, the results of this experiment show that while the performance of the D-Proto method does improve when DP is added under a NOTA rate of 50%, the performance decreases for NOTA rates of 15% and 30%. This suggests that while the use of DP K-2 can indeed improve the detection of NOTA instances at higher densities, also leads to the false detection of known instances as NOTA, particularly at lower NOTA rates.
Non NOTA
NOTA
Non NOTA
NOTA
Figure 5: Comparison of all scores presented in this paper under 50% NOTA rate on the FewRel dataset. The dash-dotted line indicates the median of each score.
makes NOTA isolated, helping to recognize rela- tions with densities.
Then removing the Base Score significantly im- pacts the model's performance. The Base Score is essential as it serves the basis of PLOF. It also be observed that the removal of either the prototype set P the degenerative prototype set DP results significant decline in the evaluation metrics across different NOTA rates. Additionally, the removal of both P and DP results in an even more significant drop in the metrics. These findings suggest that both the prototype set and the degen- erative prototype set are important components of the PLOF method and contribute significantly to its performance.
3.4.5 Visualization
To further evaluate the representations learned by different models, we visualize the embeddings us- ing the t-SNE algorithm (Van der Maaten and Hin- ton, 2008) in Figure 6. Under both the 15% and 30% NOTA rates, compared with O-Proto and MNAV, method can make each relation cluster more densely. For instance, in Figure 6 (a), the lation 1 is scattered around in O-Proto and MNAV. However, in our method, this class is well aggre- gated. This shows that our method can learn more separable representations for known instances.
3.4.4 Degenerative Prototypes Analysis
We have a research question: does adding more degenerative prototypes also increase the perfor-

---

Thus few-shot scenario has drawn much attention recently. By training on multiple episodes, mod- els can learn transferrable knowledge for unseen testing episodes (Hu et al., 2021). Although many efforts (Hendrickx et al., 2010; Zhang et al., 2017;
Proto
O-Prato
Obamuyide and Vlachos, 2019; Gao et al., 2019a; Baldini Soares et al.. 2019; Ye and Ling, 2019; Dong al., 2020) have been devoted to the few- shot relation classification. However, real-world issue that has been proposed is the NOTA chal- lenge (Gao et al., 2019b). This issue breaks the assumption the traditional N N-way K-shot set- ting. Therefore, detecting NOTA instances during inference needs to be explored.
MNAA
2
1
11
1
D-Protofours)
D-Protojours)
Promising works have been proposed to detect NOTA instances in few-shot relation classification. One approach that has been widely adopted is the N + classification. This approach has been pro- posed by several works. One work is to use a sentence-pair model for classification, as proposed by Gao et al. (2019h) in their work on few-shot learning. Another work, Sabo et al. (2021) in their work on revisiting NOTA challenge, is to compute the distance between learnable vectors. The third one, proposed by Liu et al. (2022), adopts the pre- training fine-tuning paradigm and selects the cor- rect relation from a set of options that includes NOTA. The above three methods, by considering NOTA instances as an additional class, are able to learn the boundary between the known relations and the NOTA instances, allowing the model to learn more comprehensive representation of the data.
$6
3g1
(a) 15% NOTA rate
(b) 30% NOTA rate
Figure 6: Embedding distributions of different models the same episode on the FewRel dataset. NOTA rates are 15% and 30% respectively. The numbers 1 to 5 represent the relation. The red inverted triangles are NOTA instances.
Moreover, in Figure 6 (a), we that the NOTA instance is far away from known samples in our method. This observation is visible in the plot (b). This shows the validity of the proposed NOTA loss Lnotor making NOTA instances more isolated. In MNAV, the two NOTA instances are close to a known sample. One possible reason is that MNAV makes multiple NOTA instances approach learn- able vectors. Different classes may aggregate with each other. This tends to cause negative effects on distinguishing samples, embeddings not well separated. In summary, our method learns the best representations. With the well-clustered known samples and the isolated NOTA samples, D-Proto can both recognize them through the density-aware NOTA detection module accurately.
Conclusion
In this paper, we propose D-Proto, a density-aware prototypical network for few-shot relation classi- fication. We focus the overall instance distri- bution and design special training objectives for distinct samples. For the known samples, intra- class becomes well-clustered and inter-class is well- separated. The NOTA ones are isolated as outliers. We further propose a NOTA detection module, which considers the local density and distance prop- erty, to distinguish both types of instances. And prototypes and degenerative prototypes to enhance the density of known instances to better identify NOTA. Experiments on a popular dataset demonstrate that D-Proto outperforms strong base- lines. We also provide score analysis and visualiza- tion to verify the effectiveness of our method.
4 Related Work
Previous works (Zelenko et al., 2003; Mooney and Bunescu, 2005; Zeng et al., 2014; Gormley al., 2015; Zhou et al., 2016; Kumar, 2017; Zhang et al., 2018) for relation classification usually train mod- cls on labeled dataset with a fixed number of classes, which cannot deal with unseen relations.

---

Limitations
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre- veetrrainng of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistiex: Human Language Technologies, Volume (NAACI-III.T), pages 4171- 4186.
In this work, we study the few-shot relation classi- fication task, which contains NOTA instances. The proposed D-Proto method achieves some gains on this problem. Yet our work still has the following two limitations.
The first limitation is that NOTA instances are detected with a threshold r. This threshold relies on human experiences or k-fold validation. How- ever, using a threshold in the binary way methods, including O-Proto and ours, is not a big problem. Our viewpoint is that N +1 +1 classification can ef- ficiently accomplish the NOTA detection task by treating the NOTA as an additional class to classify with the known N classes. But the N + 1 classifi- cation method ignores the distribution of the data. Therefore, despite that we need a threshold, the ad advantages of our method are also obvious.
Bowen Dong, Yuan Yao. Ruobing Xie, Tianyu Gao. Xu Han, Zhiyuan Liu, Fen Lin, Leyu Lin, and Maosong Sun. 2020. Meta-information guided meta- learning for few-shot relation classification. In Pro- ceedings of the 28th International Conference Computational Linguistics (COLING), pages 1594- 1605.
Tianyu Gao, Xu Han, Zhiyuan Liu, and Maosong Sun. 2019a. Hybrid attention- based prototypical networks for noisy few-shot relation classification. In Proceed- ings of the AAAI Conference on Artificial Intelligence (AAAI), volume 33, pages 4407-6414.
Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019b. Fewrel 2.0: Towards more challenging few-shot relation classi- fication. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJJCN) pages 6250-6255.
The second limitation is that our experiment is not absolutely fair. The reason is that BERT-Pait (Gao et al., 2019b) requires high GPU memory. In the fixed sampling method of the episode, we used two NVIDIA RTX A6000 with 48G memory each to train BERT-Pair. The other baselines are trained with an NVIDIA TESLA V100-32G. Except for this difference, all other settings are the same.
Matthew R. Gormley, Mo Yu, and Mark Dredze. 2015. Improved relation extraction with feature-rich com- positional embedding models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1774- -1744
A Akknowledgment
We sincerely thank all the anonymous reviewers for providing valuable feedback. This work is sup- ported by the National Natural Science Foundation of China (Grant No. 62302245). the Ministry of Education of the People's 's Republic of China Hu- manities and Social Sciences Youth Foundation (Grant No. 23YJCZH240), and the youth program of National Science Fund of Tianjin, China (Grant No. 22JQQNIC01340)
Jiale Han, Bo Cheng, and Wei Lu. 2021. Exploring task difficulty for few-shot relation extraction. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2605-2616, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao. Zhiyuan Liu, and Maosong Sun. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of -f-the-art evaluation. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4803 4809.
References
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Pro- ceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguisties (ACL), pages
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Pad6, Marco Pennacchiotti. Lorenza Romano, and Stan Szpakowicz. 2010. Semeval- 2010 task 8: Multi- way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 33-38.
Sam Brody, Sichao Wu, and Adrian Benton. 2021. To- wards realistic few-shot relation extraction. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 5338- 5345, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mengting Hu, Shiwan Zhao, Honglei Guo, Chao Xue, Hang Gao, Tiegang Gao, Renhong Cheng, and Zhong Su. 2021. Multi-label few-shot learning for as- pect category detection, In Proceedings of the 59th

---

Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference Natural Language Processing (ACI- IJCNLP), pages 6330-6340.
Zhi-Xiu Ye and Zhen- Ling. 2019. Multi-level matching and aggregation network for few-shot rela- tion classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics (ACL), pages 2872-2881.
Shantanu Kumar. 2017. survey of deep learning methods for relation extraction. arXiv preprint arXiv:1705.03645.
Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2003. Kernel methods for relation extraction. Journal of machine learning research (JMLR), 108-)):0083-1100
Steinar Laenen and Luca Bertinetto, 2021. On episodes, prototypical networks, and few-shot learning. Ad- vances in Neural Information Processing Systems (NeurIPS), 34.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via con- volutional deep neural network. In Proceedings of the 28th International Conference on Computational Linguistics (COLING), pages 2335-2344.
Fangchao Liu, Hongyu Lin, Xianpei Han, Boxi Cao, and Le Sun. 2022. Pre-training to match for unified low- shot relation extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5785- 5795, Dublin, Ireland. Association Computational Linguistics.
Ningyu Zhang. Shumin Deng. Zhanling Sun, Xi Chen, Wei Zhang. and Huajun Chen. 2018. Attention-base capsule networks with dynamic routing for relation extraction. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing (ACL), pages 986-992.
Raymond Mooney and Razvan Bunescu. 2005. Sub- sequence kemnels for relation extraction. Advances neural information processing systems (NeurIPS). 18.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D Manning. 2017. Position-aware attention and supervised data improve slot filling. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 35 45.
Abiola Obamuyide and Andreas Vlachos. 2019. Model-
agnostic meta-learing for relation classification with limited supervision.
Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics (ACL), pages 5873-5879, Florence, Italy.
Peng Zhou, Wei Shi, Jun Tiam, Zhenyu Qi, Bingehen Li. Hongwei Hao, and Bo Xu. 2016. Attention- bidirectional long short-term memory networks for relation classification. Annual Meeting of the Association for Computational Linguistics (ACL), pages 207-212.
Association for Computational Linguistics.
Proceedings of 54th
Meng Qu, Tianyu Gao, Louis-Pascaa Xhonneux, and Jian Tang. 2020. Few-shot relation extraction via relation graphs. In Pro-
bayesian meta-learning ceedings of the 37th International Conference on Machine Learning (ICML), pages 7867-7876.
Ofer Sabo, Yanai Elazar, Yoav Goldberg, and Ido Da- gan. 2021. Revisiting few-shot relation classification: Evaluation data and classification schemes. Transac- tions of the Association for Computational Linguis- tics (TACL), 9:691-706.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning- Ad- vances in neural information processing systems (NeurIPS), 30.
Ming Tan, Yang Yu. Haoyu Wang. Dakuo Wang Saloni Potdar, Shiyu Chang, and Mo Yu. 2019. Out-of- domain detection for low-resource text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNILP), pages 3566-3572.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research (JMLR), 911))

---

Random Sampling
15%NOTA
30%NOTA
50%NOTA
Average
Methods
Acc
F1
Acc
F1
Acc
F1
Acc
F1
O-Proto
80.02:10.65
02.84
80.55+10.97
73.73+11.18
81.16:1..04
12.41=1.22
80.58
72.99
BERT-Pair
80.77:12.35
73.9912.67
81.5311.60
75.06:11.92
83.24110.85
15.14
81.85
74.73
MNAV
85.02-00.39
78.8140.47
83.67-0.66
77.78+0.90
81,59+1.02
73.94=1.11
83.42
76.84
MCMN
1..24+1.83
83.54+2..45
82.50++3.40
76.39+44.34
74.88+5.83
67,49-6.64
81.87
75.80
ChatGPT
64.57-10.31
56.63-+0.41
63.30-00557
55.81
61.66+00.76
54.16=00.45
63.18
55.54
D- P-Proto
85.76++0.72
79.79
84,80+00.92
1.12
$3.30-+1.22
76,32 1.48
84.62
78.46
Fixed Sampling
15%NOTA
30%NOTA
50%NOTA
Average
Methods
Acc
F1
Ace
FI
Acc
FI
Acc
F1
O-Proto
81.3811.226
80.45:11.29
81.42.1.111
81.0411.19
81.98
79.3911.00
81.38
80.29
BERT-Pair
82.83:1:062
81.98:10.22
83.76:1:0.6
83.00=0.554
85,49:0.299
82.62=#0.45
84.02
82.53
MNAV
85.05.10.97
84.22.11.03
84.36.11.34
84.4511.115
82.91
82.15
84.10
83.60
MCMN
1..5
84.04:22.52
83.16:13.37
83.40=33.08
75.07=45.79
79.14
81.93
82.19
ChatGPT
63.5910.50
56.2610.27
61.8311.41
54.46110.22
58.41:10.51
51.14:10.43
61.28
53.95
D-Proto
85.37+00.61
84.67:+.611
84.87=0071
84.90=0.66
83.05 1.00
82.66=
84.43
84.08
Table 4: Evaluation results of baseline methods and the proposed D-Proto, in terms of accuracy (%) and Fl (%), on the FewRel dataset. This table includes the standard deviation of baseline methods and D-Proto.
A Reproducibility
Model
Number of parameters
A.1 Implementation Details
O-Proto
109482240
MNAV
109497600
Following Gao et al. (2019b), we exploit the same hyper-parameters. For fair comparison, all base- lines and our work use the same encoder. We choose BERT-base-uncased (Devlin et al., 2019) the encoder. It is initialized by the pre-trained parameters and optimized during training. For the FewRel dataset, we the batch size 2, which means we feed two episodes into the model per batch. Each model is trained for 30 epochs with 1000 batches per epoch and tested with 10000 batches in the testing. An early stopping strategy is adopted, indicating the model will stop training when the performance the validation set does not improve for 6 epochs. The best model the validation set saved for evaluation.
BERT-Pair
109483778
D-Proto
109482240
Table 5: model.
The number hyperparameters for each
training follows Liu et al. (2022), while the sam- pling method in the testing is fixed sampling random sampling.
A.2 Computing Infrastructure
Since BERT-Pair (Gao al., 2019b) requires high GPU memory in the fixed sampling of 5-way 5- shot setting, we use NVIDIA A6000s with 48G memory each to train the model. The rest of the baselines and our model are trained on NVIDIA TESLA V100 with 32G of GPU memory.
In Lgq (1), we set the hyper-parameters to le-5 on the FewRel. For & of the local outlier factor in Eq. (7), we set it to be K. The reason is that in the N-way K-shot setting, support instance has other (K 1) neighbors with the same relation. And each relation has at least one prototype. This is consistent with the definition of k-neighborhood. All the reported results using the FewRel dataset are the average of five runs with fixed seeds [5, 10, 15, 20, 25J.
A.3 Number of Parameters per Model
Table shows the number of hyperparameters for each model.
A.4 Evaluation Metries
After the hyperparameter search, the threshold is set to 0.9 in the experimental setting of FewRel. For MCMN, the sampling method the pre-
We use two primary evaluation metrics, accuracy and macro F1. The basic implementation of accu-

---

Method
Number of instances
A.7
Links to Toolkit
D-Proto w/o p
53
BERT:
https://hugiingaacc. co/
D-Proto w/o DP
33
5//oe//..././/eodo/o////eee htmlfbbrrtoo tmodel
transformers/v4.
D-Proto w/o
28
D-Proto 20%DPxx2
68
D-Proto 40%DPk-
78
D-Proto 60%DPx-
88
Tokenizer: https://hugginngfacc co/ htt.rssforrrrr//...... 5.1//odee_doc// htw.thtttteerttoeeeeee
BERT
D-Proto 80%DPk-
98
D-Proto 100%DPx>
108
D-Proto(Ours)
58
Table 6: The number of instances for each method in an episode. The experiment setting is 5-way 5-shot. The number query instances is 3.
Local
Outlier Factor(L.OF):
https:
/ heaplktt-eeern.ogggsatlllleeeeee
generated/skklarr.neigg Localoutlierfactor. html
racy is as follows:
Maero Fl: https///sikitt-eem..rrgg
TP TN TP + FP + TN + FN
stailee/nmouulsssseeerrrt///slss metrics, fw_.scls.eee__eeee html
Accuracy
where T are true positives, FP are false positives, TN are true negatives, and FN are false negatives. The definition of FI as follows:
A.8 Number of Instances per Method
Table shows the number of instances included an episode for each method in ablation studying. The episode sampling method is random sampling.
TP P TP + FP TP TP + FN
A.9 Prompt of ChatGPT
We demonstrate the prompt for using ChatGPT directly in the NOTA task, follows:
2 P F1 P+R
Macro Fl calculates FI for each class and finds their unweighted mean.
The purpose of the relation clas- sification task is to identify the relation be- tween two entities (one is the head entity and the other is the tail entity) in a given sen- tence. For example, the following sentence "California is state in the United States". Through the derivation of the whole sen- tence, the relation between the head entity "California" and the tail entity the "United States" is "contains". While the few-shot relation classification task consists of sup- port set and a query set. The support set generally contains N relations, and each relation contains K instances. This setup is called N-way K-shot. We need predict the relation between two entities in each sentence given in the query set based on the support set. Now need you to complete a few-shot relation classification task, which set to 5-way 5-shot. The five categories in the support set are numbered from O to 4, respectively. Five sentences are given after
Question:
A.5 Hyperparameters
Following Gao et al. (2019h), the hyperparameters not mentioned in Section A.1 are as follows:
Learning rate: 2e-5.
Weight decay: 0.01.
Warmup steps: 300.
Gradient accumulation: 1.
Max length: 128.
Hidden size: 768.
Test batch: 10000.
A.6 Links to Dataset
FewRel (Han et al., 2018): https://thunl/: github.

---

each category. The head entity of each tence is surrounded by two special charac- ters [head] and [/head], and the tail en- tity is surrounded by two special characters [tail] and [/tai]]. You need to predict the relationship between two entities for each sentence in the query set from the support set. That is, if the relation of query in- stance is similar that of all instances of category 0 in the support set, then the cate- gory predicted by the query instance is car- egory O. Notice! If the relation between the two entities of a sentence in the query set cannol be referenced in the support set, the relation between the two entities of the sen- tence is called NOTA (None-of-the -f---e--bbveee When the NOTA relation is encountered, the category number of the predicted result is 5. Warning! You just need return the class number for each instance predicted in the query set. For example, the relation of the first instance in the query set is rela- tion O, and the second instance is relation 5 (NOTA). Then your reply is: [0, 51. If you get it, then I'll give you support sets and query sets next.
ChatGPT: Yes, I understand the task. Please provide me with the support set and query set.
Question: The following is the support set (5-way 5-shot. Instances in each category are separated by special character (sep].):
The following is the query set (The query set contains one instance.):


Reviews:
[{"review_id": "vW2pK4C2WI", "reviewer": "Reviewer_vjzP", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and suggestions.\n\n__Q1:__ Limited evaluation: The proposed method is evaluated on only one dataset, which may limit the generalizability of the results. Further evaluation on other datasets is needed to validate the effectiveness of the proposed method.\n\n__A1:__ We appreciate your kind suggestions. Although our experiments only used the FewRel dataset, which is a publicly available dataset, it is currently the highest-quality and most balanced dataset for the FSRC task. Additionally, our experiments employed multiple data sampling methods and different NOTA rates. The extensive ablation study also serves as evidence for the effectiveness of our approach. We will conduct more experiments by using more FSRC datasets in the camera-ready version.\n\n__Q2:__ Although the paper discusses the limitations of the proposed method, the discussion is relatively brief and lacks a detailed analysis of the limitations. A more in-depth discussion of the error analysis can help readers better understand the potential challenges and future directions of the research.\n\n__A2:__ Thank you for pointing out this issue. In the camera-ready version, we will introduce more analysis, such as conducting further error analysis. Below, we provide an example of a misclassified relation by the D-Proto model sampled from the FewRel dataset. The true category of query instance $1$ is “Country of origin”, but due to the semantic similarity between this instance and the relation “Country” in the support set, D-Proto failed to correctly detect it as the NOTA category.\n\nSupport set:\n\n{\n\n“Record label”: {……}\n\n“Operator”: {……}\n\n“Moment”: {……}\n\n“Followed by”: {……}\n\n“Country”: {\n1. [tail] Belgium [\\tail]'s highest point is the [head] Signal de Botrange [\\head] at 694 meters above the sea level.\n2. [head] Syria [\\head] was an early Roman province, annexed to the [tail] Roman Republic [\\tail] in 64 BC by Pompey in the Third Mithridatic War, following the defeat of Armenian King Tigranes the Great.\n3. The [head] Dniester [\\head] rises in [tail] Ukraine [\\tail], near the city of Drohobych, close to the border with Poland, and flows toward the Black Sea.\n4. The Ulster House Hotel, formerly the Wellington Hotel, is located on Main Street in Pine Hill, New York, United States.\n5. [head] Pico Turquino [\\head] is the highest summit of the island and [tail] Republic of Cuba [\\tail].\n\n}\n\n}\n\nQuery set:\n\n{\n\n1.\t[head] Miss Navajo [\\head] Nation is a pageant that has been held annually on the Navajo Nation, [tail] United States [\\tail], since 1952.\n\n……\n\n}\n\n__Q3:__ In the experimental section, it would be clearer to separately highlight the improvements in NOTA cases attributed to the proposed D-proto. A more in-depth analysis of how D-proto addresses NOTA cases would offer a clearer explanation of the model's effectiveness.\n\n__A3:__ Thanks for your suggestion. In Figure 6, we present specific examples by visualizing their representations. This figure demonstrates the improvements brought by our proposed D-Proto compared to other prototype-based baselines in terms of density and the distribution of NOTA. In the camera-ready version, we will provide further analysis with more cases.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novelty: The paper proposes a novel density-aware prototypical network (D-Proto) for few-shot relation classification, which addresses the challenging problem of none-of-the-above (NOTA) relation classification, which is a fundamental task in natural language processing.\n\n2. Effectiveness: The experimental results demonstrate that D-Proto outperforms strong baselines with robustness towards various NOTA rates. This suggests that D-Proto can be a promising solution for few-shot relation classification tasks when it comes to NOTA problem.\n\n3. Clarity: The paper is well-written and organized, with clear explanations of the proposed method and experimental results. The authors also provide score analysis and visualization to verify the effectiveness of their method.\n\n4. Reproducibility: The authors provide a clear reproducibility for further replication and plan to make the code public after the paper is accepted, which can facilitate further research and development in this area.", "reasons_to_reject": "The weaknesses of this paper are as follows:\n\n1. Limited evaluation: The proposed method is evaluated on only one dataset, which may limit the generalizability of the results. Further evaluation on other datasets is needed to validate the effectiveness of the proposed method.\n\n2. Although the paper discusses the limitations of the proposed method, the discussion is relatively brief and lacks a detailed analysis of the limitations. A more in-depth discussion of the error analysis can help readers better understand the potential challenges and future directions of the research.\n\n3. In the experimental section, it would be clearer to separately highlight the improvements in NOTA cases attributed to the proposed D-proto. A more in-depth analysis of how D-proto addresses NOTA cases would offer a clearer explanation of the model's effectiveness.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The D-Proto network addresses the problem of none-of-the-above (NOTA) relation classification by treating various instances distinctly. Specifically, the authors design unique training objectives to separate known instances and isolate NOTA instances, respectively. This produces an ideal instance distribution, where known instances are dense yet NOTAs have a small density. Moreover, the authors propose a NOTA detection module to further enlarge the density of known samples, and discriminate NOTA and known samples accurately. The experimental results of the proposed method show that D-Proto outperforms strong baselines and achieves the best average results on both fixed and random sampling strategies.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eZIweUMPlB", "reviewer": "Reviewer_sQUS", "summary": "", "questions": "Have the authors considered testing their framework under common scenarios (i.e., without NOTA samples)? The datasets included in the experiments are also insufficient. Do the authors consider adding more datasets for comparison?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and suggestions.\n\n__Q1:__ The authors mainly focus on the scenario where there exist a substantial number of NOTA samples. However, such a scenario may not be very common in real world. Specifically, in the experiments, the authors show results with more than 15% NOTA samples, which can be a lot in realistic scenarios. More importantly, the authors do not provide justification such as references of such existence of a high ratio of NOTA samples.\n\n__A1:__ Thanks for pointing this out. FewRel 2.0[2] first introduced the NOTA problem and provided standard experimental settings with NOTA rates of 15%, 30%, and 50% respectively. Our experiments followed these standard settings. We will conduct further experiments to explore the effectiveness of our proposed method at lower NOTA rates.\n\n__Q2:__ The experimental part is not solid. Specifically, the compared methods are not enough (only four few-shot relation classification baselines compared with two before 2020). Moreover, these methods are not specifically proposed for scenarios with such a high ratio of NOTA samples, which will reduce the fairness of the experiments.\n\n__A2:__ As mentioned in answer __A1__, the baseline we selected is also based on the standard settings of FewRel 2.0[2]. We will provide further results in the camera-ready version.\n\n__Q3:__ Most importantly, although the authors claim that they focus on scenarios with NOTA samples, they do not compare the results of their framework and other baselines regarding the scenario without NOTA samples, which should be more common in realistic scenarios. The lack of such comparison harms the soundness of experiments\n\n__A3:__ This is a good question. The absence of NOTA instances is a classic setting in the FSRC task proposed by FewRel 1.0[1]. The new NOTA problem introduced by FewRel 2.0 [2] is currently receiving the most attention, as the NOTA task aligns more with real-world situations and presents greater difficulty.\n\n__Q4:__ Have the authors considered testing their framework under common scenarios (i.e., without NOTA samples)? The datasets included in the experiments are also insufficient. Do the authors consider adding more datasets for comparison?\n\n__A4:__ We appreciate your kind suggestions. As mentioned in answer __A3__, we will further investigate the performance of D-Proto on traditional FSRC tasks and provide more experimental results in the camera-ready version.\n\nReferences:\n\n[1] Han X, Zhu H, Yu P, et al. FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation[J].\n\n[2] Gao T, Han X, Zhu H, et al. FewRel 2.0: Towards More Challenging Few-Shot Relation Classification[J].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work studies the important problem of few-shot relation classification under the scenario with existence of NOTA samples. The experiments also validate the effectiveness of the framework.\n\n2. The authors design specific strategies that are inspiring for future work on few-shot relation classification. The authors also investigate the challenges in few-shot relation classification, which can be crucial for future research.\n\n3. The paper is well-written and easy to follow.", "reasons_to_reject": "1. The authors mainly focus on the scenario where there exist a substantial number of NOTA samples. However, such a scenario may not be very common in real world. Specifically, in the experiments, the authors show results with more than 15% NOTA samples, which can be a lot in realistic scenarios. More importantly, the authors do not provide justification such as references of such existence of a high ratio of NOTA samples.\n\n2. The experimental part is not solid. Specifically, the compared methods are not enough (only four few-shot relation classification baselines compared with two before 2020). Moreover, these methods are not specifically proposed for scenarios with such a high ratio of NOTA samples, which will reduce the fairness of the experiments.\n\n3. Most importantly, although the authors claim that they focus on scenarios with NOTA samples, they do not compare the results of their framework and other baselines regarding the scenario without NOTA samples, which should be more common in realistic scenarios. The lack of such comparison harms the soundness of experiments", "questions_to_authors": "Have the authors considered testing their framework under common scenarios (i.e., without NOTA samples)? The datasets included in the experiments are also insufficient. Do the authors consider adding more datasets for comparison?", "paper_topic_and_main_contributions": "This paper studies the important problem of few-shot relation classification where there exist a substantial number of none-of-the-above (NOTA) samples, which do not belong to any classes. The authors propose to identify NOTA samples as outliers that differ from the distribution of known samples. By designing a unique training objective that separate known samples and NOTA samples, the authors manage to accurately detect the existence of NOTA samples. The authors also conduct extensive experiments to verify the effectiveness of their framework.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cTZP0b5ZTe", "reviewer": "Reviewer_sq7x", "summary": "", "questions": "Question A: Could you provide more informative arguments or illustration to demonstrate the distribution imbalance of the chosen dataset distribution or any other dataset?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and suggestions.\n\n__Q1:__ The authors only tested the proposed method on the FewRel dataset. Considering the specificity of the NOTA detection task, more experiments on other datasets should be included to validate the effectiveness of D-Proto.\n\n__A1:__ We appreciate your kind suggestions. Although our experiments only used the FewRel dataset, which is a publicly available dataset, it is currently the highest-quality and most balanced dataset for the FSRC task. Additionally, our experiments employed multiple data sampling methods and different NOTA rates. The extensive ablation study also serves as evidence for the effectiveness of our approach. We will conduct more experiments by using more FSRC datasets in the camera-ready version.\n\n__Q2:__ In addition, the article lacks a comparison of the distributional differences of the original data, although a comparison between the proposed D-Proto and the two previous amplifications is provided in the visualization section, the analysis of the original dataset is even more important since it is the main source of this research’s intuition, which should have been given more justification and explanation.\n\n__A2:__ Thank you for pointing out the issue that we overlooked. The FewRel dataset proposed by FewRel 1.0[1] contains $700$ instances for each category, so its distribution is balanced. When we mention imbalanced distribution, we are referring to the phenomenon of imbalance within each sampled episode. For the specific cases, please refer to answer __A3__.\n\n__Q3:__ Could you provide more informative arguments or illustration to demonstrate the distribution imbalance of the chosen dataset distribution or any other dataset?\n\n__A3:__ We appreciate your kind suggestions. Below, we provide an example sampled from the FewRel dataset. The query set includes $5$ instances of non-NOTA (not given) and $5$ instances of NOTA, each belonging to $5$ different categories. If we don't consider the distribution and cluster these NOTA instances of different categories together (like in MNAV), it will inevitably lead to an imbalance in the instances of this episode. In the camera-ready version, we will provide further analysis to illustrate this imbalance, such as a case study.\n\nSupport set:\n\n{\n\n“Record label”: {……}\n\n“Operator”: {……}\n\n“Moment”: {……}\n\n“Followed by”: {……}\n\n“Country”: {……}\n\n}\n\nQuery set:\n\n{\n\n……\n\n1. [head] Miss Navajo [\\head] Nation is a pageant that has been held annually on the Navajo Nation, [tail] United States [\\tail], since 1952.\n2. The [tail] Wildcats [\\tail] moved off campus from Memorial Coliseum to [head] Rupp Arena [\\head] in the downtown metroplex.\n3. Actress Sally Hawkins was asked to play \"[head] Persuasion [\\head] 's\" protagonist [tail] Anne Elliot [\\tail].\n4. The songs were re-mastered to vinyl by Robert Hadley and former lead singer of [head] Journey [\\head], [tail] Steve Perry [\\tail].\n5. [head] William Trevor [\\head], \"[tail] The Story of Lucy Gault [\\tail]\" , 2002, page 14.\n\n}\n\nReferences:\n\n[1] Han X, Zhu H, Yu P, et al. FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation[J].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "For the NOTA problem, D-Proto innovatively starts from the data distribution and rigorously defines a series of new loss functions and methods to answer the question \"what instances distribution is ideal\". It progressively combines known sample separation, NOTA sample isolation, and NOTA detection to leverage the density properties. Detailed experiments and graphs illustrate the effectiveness of each of the proposed approaches on both the Random and Fixed sampling methods, where the F1 score and Accuracy outperform the previous studies.", "reasons_to_reject": "The authors only tested the proposed method on the FewRel dataset. Considering the specificity of the NOTA detection task, more experiments on other datasets should be included to validate the effectiveness of D-Proto. In addition, the article lacks a comparison of the distributional differences of the original data, although a comparison between the proposed D-Proto and the two previous amplifications is provided in the visualization section, the analysis of the original dataset is even more important since it is the main source of this research’s intuition, which should have been given more justification and explanation.", "questions_to_authors": "Question A: Could you provide more informative arguments or illustration to demonstrate the distribution imbalance of the chosen dataset distribution or any other dataset?", "paper_topic_and_main_contributions": "This work investigates the subproblem none-of-the-above (NOTA) under the field of few-shot. The authors notice that previous approaches have not focused on the discrepancy between the data distribution of NOTA and the known categories in small samples, and based on this they propose density-aware prototypical network (D-Proto) to leverage the distributional difference, claiming that is the first to model the overall instance distribution in an episode. They also propose NOTA loss to balance the distributional differences between the two heterogeneous data, then further use a prototype enhanced local outlier factor (PLOF) and a base score to fully leverage the density properties. Extensive experimental results demonstrated the effectiveness and robustness of the proposed method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No", "Typos_Grammar_Style_and_Presentation_Improvements": "No"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to deal with few-shot relation extraction, especially to solve the problem of none-of-the-above (NOTA) relation classification. Specifically, this paper adopts an additional training objective to classify known and out-of-distribution instances, making the NOTA relations easier to identify. The structure of the article is well organized, and the presentation is relatively straightforward. The main problem of this paper is that the used datasets are limited, making the results not solid enough.
Decision: Accept-Findings