Learning Knowledge-Enhanced Contextual Language Representations for Domain Natural Language Understanding
Authors: Taolin Zhang
Ruyao Xu
Chengyu Wang
Zhongjie Duan
Cen Chen
Minghui Qiu
Dawei Cheng
Xiaofeng He
Weining Qian
Venue: EMNLP 2023 Main (2023)
Keywords: Closed-domain
Pre-trained Language Model
Knowledge Graph

Abstract: Knowledge-Enhanced Pre-trained Language Models (KEPLMs) improve the performance of various downstream NLP tasks by injecting knowledge facts from large-scale Knowledge Graphs (KGs). However, existing methods for pre-training KEPLMs with relational triples are difficult to be adapted to close domains due to the lack of sufficient domain graph semantics. In this paper, we propose a Knowledge-enhanced language representation learning framework for various closed domains (KANGAROO) via capturing the implicit graph structure among the entities. Specifically, since the entity coverage rates of closed-domain KGs can be relatively low and may exhibit the global sparsity phenomenon for knowledge injection, we consider not only the shallow relational representations of triples but also the hyperbolic embeddings of deep hierarchical entity-class structures for effective knowledge fusion. Moreover, as two closed-domain entities under the same entity-class often havel locally dense neighbor subgraphs counted by max point biconnected component, we further propose a data augmentation strategy based on contrastive learning over subgraphs to construct hard negative samples of higher quality. It makes the underlying KELPMs better distinguish the semantics of these neighboring entities to further complement the global semantic sparsity. In the experiments, we evaluate KANGAROO over various knowledge-aware and general NLP tasks in both full and few-shot learning settings, outperforming various KEPLM training paradigms performance in closed-domains significantly.

Learning Ghowledge-Enhandd Contextual Language Representations for Domain Natural Language Understanding
Taolin Zhang Ruyao Xu"; Chengyu Wang', Zhongjie Duan', Cen Chen',
Minghui Qiu?, Dawei Cheng Xiaofeng He Weining Qian'
East China Normal University, Shanghai, China
2 Alibaba Group, Hangzhou, China Tongji University, Shanghai, China wcy, zjyxuu ecnu, edu. en, {cenchen, weqian)@dase. ecnu. edu. cn hexfêcs ecnu. edu.un, cn, dcheng@tongji. edu. cn
ichangtaolin..tl,.cee
watpuuiaaaheeaabbaaiiiiiiaa com
Abstract
Closed Domain
Pre-training Corpus
Open Domain
The symptoms of COVID-19 in humans include respirato ry infections and fever.
Knowledge-Enhain Enhanced Pre-trained Language Models (KEPLMMS) improve the performance various downstream NLP tasks by injecting knowledge facts from large-scale Knowledge Graphs (KGs). However, existing methods for pre-training KEPLMS with relational triples are difficult be adapted close domains due to the lack of sufficient domain graph seman- tics. In this paper, we propose a Knowledge- enhanced IANGuAge Representation learning framework for various clOsed dOmains (KAN- GAROO) via capturing the implicit graph struc- ture among the entities. Specifically, since the entity coverage rates of closed-domain KGs can be relatively low and may exhibit the global sparsity phenomenon for knowledge injection, we consider not only the shallow relational representations of triples but also the hyper- bolic embeddings of deep hierarchical entity- class structures for effective knowledge fusion. Moreover, two closed-domain entities under the same entity-class often have locally dense neighbor subgraphs counted by max point bi- connected component, we further propose data augmentation strategy based on contrastive learning subgraphs construct hard nega- tive samples of higher quality. It makes the un- derlying KELPMs better distinguish the seman- ties of these neighboring entities to further com- plement the global semantic sparsity. In the ex- periments, we evaluate KANGAROO over vari- ous knowledge-awee both full and fow-shot lcarning settings, outper- forming various KEPLM training paradigms performance in closed-domains significantly.
MedKG
CN-DBpedia
48.43%
41.48%
25.37%
15.73%
Strongly connected and dense clusters
Scattered and sparse clusters
Enbiyy Enverage Ballo
Biconnected Comp.
Entities
CN-DBpedia
Figure Comparison of statistics between closed- domain and open-domain KGs (taking CN-DBPedia (Xu et al., 2017) and a medical KG as an example). Closed-domain KGs have lower entity coverage ratios over text corpora (global sparsity). Entities are more densely inter-connected in closed-domain KGs (local density) 2, (Best viewed in color.)
Introduction
The performance of downstream tasks (Wang al., 2020) can be further improved by KEPLMs (Zhang et al., 2019; Peters et al., 2019; Liu et al., 2020a; Zhang et al., 2021a, 2022a) which leverage rich knowledge triples from KGs to enhance language representations. In the literature, most knowl- edge injection approaches for KEPLMs can be roughly categorized into two types: knowledge em- bedding and joint learning. (1) Knowledge embed- ding-based approaches aggregate representations of knowledge triples learned by KG embedding mod- els with PLMs contextual representations (Zhang et al., 2019; Peters et al., 2019; Su et al., 2021; Wu et al., 2023). (2) Joint learning- baaeedg--aaddddee methods con- vert knowledge triples into pre-training sentences without introducing other parameters for knowl- edge encoders (Sun et al., 2020: Wang al., 2021). These works mainly focus on building KEPIMs for the open domain based on large-scale KGs (Vulic et al., 2020; Lai et al., 2021: 1; Zhang et al., 2022c).
and general NLP tasks in
"T. Zhang and R. Xu contributed equally to this work. "Co-corresponiigg authors.
'All the codes and model checkpoints have been released to public the EasyNLP framework (Wang et al., 2022).
URL: http://ttuuboo/alaaaaaa/aaaaaaea "The detailed analysis of entity coverage ratios and max point biconnected component is described in Sec. 2

---

Despite the success, these approaches for build- ing open-domain KEPLMs can hardly be migrated directly to closed domains because they lack the in-depth modeling of the characteristics of closed- domain KGs (Cheng et al., 2015; Kazemi and Poole, 2018; Vashishth et al., 2020). As in Fig- ure 1, the coverage ratio of KG entities w.r.t plain texts is significantly lower in closed domains than in open domains, showing that there exists global sparsity phenomenon for domain knowledge in- jection. This means injecting the retrieved few relevant triples directly to PLMs may not be suffi- cient for closed domains. We further notice that, in closed-domain KGs, the ratios of maximum-point biconnected components are much higher, which means that entities under the same entity-class in these KGs more densely interconnected and exhibit local density property. Hence, the seman- tics of these entities are highly similar, making the underlying KEPLMs difficult to capture the dif- ferences. Yet a few approaches employ continual pre-training over domain-specific corpora (Beltagy et al., 2019; Peng et al., 2019; Lee et al., 2020), or devise pre-training objectives over in-domain KGs to capture the unique domain semantics (which re- quires rich domain expertise) (Liu et al., 2020b; He al., 2020). Therefore, there is lack of simple but effective unified framework for learning KEPLMs for various closed domains.
Oord et al., 2018) to better capture fine- grained semantic differences of neighbor enti- ties under the same entity-class structure and thus further alleviate global sparsity. Specif- ically, we focus on constructing high-quality multi-level negative samples of knowledge triples based on the relation paths in closed- domain KGs around target entities. By us- ing the proposed approach, the difficulty of being classified of various negative samples is largely increased by searching within the max point biconnected components the KG subgraphs. This method enhances the robust- ness of domain representations and makes the model distinguish the subtle semantic differ- ences better.
In the experiments, compare KANGAROO against various mainstream knowledge injection paradigms for pre-training KEPLMS over two closed domains (i.e., medical and finance). The results show that we gain consistent improvement in both full and few-shot learning settings for vari- ous knowledge-intensive and general NLP tasks.
2 Analysis of Closed-Domain KGs
In this section, we analyze the data distributions of open and closed-domain KGs in detail. Specif- ically, we employ OpenKG 2 the data source to construct a medical KG, denoted as MedKG. In addition, a financial KG (denoted as FinKG) is constructed from the structured data source from an authoritative financial company in China', As for the open domain, CN-DBpedia' is employed for further data analysis, which is the largest open source Chinese KG constructed from various Chi- nese encyclopedia sources such Wikipedia.
To overcome the above-mest soote-mentioned issues, we devise the following two components in unified framework named KANGAROO. It aggregates the above implicit structural characteristics of closed- domain KGs into KEPLM pre-training:
Hyperbolic Knowledge-aware Aggregator: Due to the semantic deficiency caused by the global sparsity phenomenon, we utilize the Poincaré ball model (Nickel and Kiela, 2017) obtain the hyperbolic embeddings of entities based the entity-class hierarchies in closed-domain KGs supplement the se- mantic information of target entities recog- nized from the pre-training corpus. It not only captures richer semantic connections among triples but also implicit graph structural infor- mation of glosed-domain KGs to alleviate the sparsity of global semantics.
To illustrate the difference between open and closed-domain KGs, we give five basic indicators (Cheng et al., 2015), which are described in detail in Appendix A due to the space limitation. From the statistics in Table 1, we roughly draw the following two conclusions:
Global Sparsity. The small magnitude and the low coverage ratio lead to the global sparsity prob- lem for closed-domain KGs. Here, data magnitude refers to the sizes of Nodes and Edges, The low
Multi-Level Knowledge-aware Augmenter: As for the local density property of closed- domain KGs, we employ the contrastive learn- ing framework (Hadsell et al., 2006; van den
httpp://ppekk cu/. The medical data is taken from subset of OpenKG. See http://openkg.cn/daaaee wwwddataset//isssseeeeeee itttuute/diissse---oaatttoo information. http://www seek- data. com/

---

Closed
Domain MedKG
Open Domain
Statistics
FmKG
CN-DBpedia
#Nodes
e,4
e,4
e+7
#Edges
c+4
c.0
c+7
Disease
Drug
Symppoom
&Coverage Ratio
5.82%
15.75%
41.48%
Classes
%MMx PBC
46.86%
48.43%
25.37%
Orthopedics
Western
Head
Suhgraph Density c-1
c-5
1.1 e-7
Table 1: The statisties of open and closed-domain KGs.
Spardylonts
tabjettablet
Erythromycin
Headadie
Bicinass
Aathromycn
Hyperbolic Knowledge-aware Aggregator
In this section, we describe how to learn the hyper- bolic entity embedding and aggregate the positive triples' representations to alleviate the global spar- sity phenomenon in closed-domain KGs.
Figure 2: Illustration of the hierarchical structure of en- tities and classes in flosed-domall KGs (e.g. MedKG).
entity coverage ratio causes the lack of enough ex- ternal knowledge to be injected into the KEPLMs. Meanwhile, the perplexing domain terms are dif- ficult to be covered by the original vocabulary of open-domain PLMs, which hurts the ability of se- mantic understanding for closed-domain KEPLMs due to the out-of vocabulluy problem. From Fig- KGs nat-
3.1.1 Learning Hyperbolie Entity Embedding As discovered previously, the embedding algo- rithms in the Euclidean space such as (Bordes al., 2013) are difficult to model complex patterns due to the dimension of the embedding space. Inspired by the Poincaré ball model (Nickel and Kiela, 2017), the hyperbolic space has stronger representational capacity for hierarchical structure due to the recon- struction effectiivnesss To make up for the global semantic deficiency of closed domains, employ the Poincaré ball model learn structural and mantic representations simultaneously based on the hicrarchical entity-class structure. The distance between two entities (ex, eÿ) is:
ure 2, see that the urally contain the hierarchical structure of entities and classes. To tackle the insufficient semantic problem due to entity coverage, we inject the domain KGs tree structures rather than the embeddings of entities alone into the KEPLMs.
Local Density. It refers the locally strong con- nectivity and high density of closed-domant KGs, which are concluded by the statistics of maximum point biconnected components and subgraph den- sity. Compared to the number of the surround- ing entities and the multi-hop structures in the sub-graph of target entities in open-domain KGs, we find that target entities in -losedddommin KGs are particularly closely and densely connected to the surrounding related neighboring entities based on the statistics of Max PBC and Subgraph Den- sity. Hence, these entities share similar semantics, which the differences are difficult for the model to Iearn. We construct more robust, hard neg- ative samples for deep contrastive learning to learn the fine-grained semantic differences of target entities in closed-domain KGs to further alleviate the global sparsity problem.
d(es, eq)
Fn 1 21)((e)) - H(e)) (1H(()) [[A(e)))
(1)
where H(--) denotes the learned representation space of hyperbolic embeddings and Fn means the arcosh function. We define D = {r(e,, e,)) be the set of observed hyponymy relations between entities. Then we minimize the distance between re- lated objects to obtain the hyperbolic embeddings:
((A)
Σ log exp exp(-d(ei,e;)) r(ene))ED
(2)
Eejco exp (-d(es,e'))
exp
where Φ means Neg(e;) - (cgln(es, e/) & DJU {cz] and {e;] is the set of negative sampling for ej. The entity class embedding of token tfe can be formulated as hey H(C E CC) R2
3 KANGAROO Framework
In this section, introduce the various modules of the model in detail and the notations are described in Appendix B.5 due to the space limitation. The whole model architecture shown in Figure 3.
3.1.2 Domain Knowledge Encoder
This module is designed for encoding input tokens and entities as well as fusing their heterogeneous

---

Donzin Knonlelge Encoder
MM
Pre-training
Entity Infusion
Eniily Knnwledge Injecter
Masked Prediction
Pos, Emb
Model
knowledge
Trans
Trans. (*N)
Taken Emb
Trask
Xeg, Embb
training Input
Text Input: [CLS]
Symptoms
of
COVID-19 humness
include
respiratory infections
and
fever
SEP]
Pos. Neg. Samples:
X
X
I
Learning typerbalic Entity feteddingg
Pol sodint-bcooeerto Component Baseed
Pos. Neg. Triples Construction
Root
Data Process
Banas
Ong
KG
For: Sample
Sample
Kigg Somele
Emitly Emieedinn
Jave
Thans
Transformer Layers
target tokens
masked tokens
pos. samples
neg. samples
max agreement
keep distant
Figure 3: Model overview of KANGAROO. The Hyperbolic Entiry Class Embedding module mainly leverages the hierarchical entity-class structure to provide more sufficient semantic knowledge. Positive and Negative Triple Construction can obtain negative samples higher quality in multiple difficulty levels. (Best viewed in color.)
embeddings, containing two parts: Entity Space Infusion and Entity Knowledge Injector.
multi-headed self-attention layer denoted as Fm:
Entity Space Infusion. To integrate hyperbolic embeddings into contextual representations, inject the entity class embedding hey into the entity representation hp; by concatenation:
(he 1= (hi
Σ F (te
(he, P
(5)
where means the Vih layer. We inject entity em- bedding into context-awarr representation and re- capture them from the mixed representation:
hef ((he, H1hp,,W, by)
(3)
hez = (he Wi bi)
(4)
hr = =oWWeh + Welli +
(6)
where a is activation function GELU (Hendrycks and Gimpel, 2016) and means concatenation. hyg is entity representation (See Section 3.2.1). LA
Ky = oWahh + +b,) K/ = o + b.) (7)
where Wi E RRiids,, We E Redodds b Rd,, W, E Rddadd, by C Rd, We e RRduxds c Rds are parameters to be learned. h2 is the mixed fu- sion embedding. Rig and hig are regenerated entity and textual embeddings, respectively.
is the LayerNorm fuction (Ba et al., 2016). Wy e +dd)+d))xds, by E Rdds Wi E Rdaxds and by E Rdda are parameters to be trained.
Entity Knowledge Injector. It aims to fuse the het- erogeneous features of entity embedding (he,) p and textual token embedding (h,, H To match relevant entities from the domain KGs, we adopt the entities that the number of overlapped words is larger than threshold. We leverage the M-layer aggregators as knowledge injector be able to in- tegrate different levels of learned fusion results. In each aggregator, both embeddings are fed into a
3.2 Multi-Level Knowledge-aware Augmenter It enables the model to learn more fine-grained semantic gaps of injected knowledge triplets, lever- aging the locally dense characteristics further remedy the global sparsity problem. We focus on constructing positive and negative samples

---

higher quality with multiple difficulty levels via the point-biconnected components subgraph structure. this section, we focus on the sample construction process shown in Figure 4. The training task of this module is introduced in Sec. 3.3.
Path 4
Path
Path
Path
3.2.1 Positive Sample Construction
Distance (Level 1)
Hop Distance = (Level
We extract K neighbor triples of the target entity eo positive samples, which are closest to the target entity in the neighboring candidate subgraph struc- ture. The semantic information contained in these triples is beneficial to enhancing contextual knowl- edge. To better aggregate target entity and con- textual tokens representations, KC neighbor triples are concatenated together into sentence. We obtain the unified semantic representation via shared Text Encoder (e.g., BERT (Devlin et al., 2019)). Since the semantic discontinuity between the sampling of different triples from discrete en- tities and relations, we modify the position em- beddings such that tokens of the same triple share the same positional index, and vice versa. For ex- ample, the position of the input tokens in Fig. triple (Co, r(Co, e1), e)) is all 1. To unify the rep- resentation space, take the [CLS] (i.c., the first token of input format in the BERT) representation positive sample embedding to represent sample sequence information. We formulate hp, E IRdd as the positive embedding of an entity word ic. 'ec
Positive sample:
Yo.r
[SEP]
(faatior enbrdding
Negative sample 1):
(SEP)
Path
Nogative sample [Lovel 2)
CLS
Path
SEP
Path
(SEP)
Path
Figure 4: Examples of positive and negative sample construction. We add the [SEP] token between paths to differentiate triplets. Note that the subscripts are positional emhedding indexes.
Level-1 and Hop(.) n+1 for Level-n samples. We assume that the closer the hop distance is, the more difficult it is to distinguish the mantic knowledge contained between triples w.r.t. the starting node.
The constructed pattern of negative samples is similar to positive samples whose paths with the same distance merged into sentences. Note that we attempt to choose the shortest path (e.g., Path when nodes' pairs contain at least two disjoint paths (i.e., point- biconnected component). For each entity, we build negative samples of k levels.
STEP 3:
3.2.2 Point-biconnected Component-based Negative Sample Construction
In closed-domain KGs, nodes are densely con- nected to the neighbouring nodes owning to the locally dense property which is conducive to graph searching. Therefore, we search for a large amount nodes that are further away from target entities as negative samples. For example in Figure 4, we construct a negative sample by the following steps:
For the high proportions of Point-biconnected Component in closed-domain KGs, there are multi- ple disjoint paths between starting nodes and end nodes in most cases such as Path 4 and Path Figure 4. We expand the data augmentation strat- egy that prefers end node pairs with multiple paths and adds the paths to the same negative sample, enhancing sample quality with diverse information. The relationships among these node pairs contain richer and indistinguishable semantic information. Besides, our framework preferentially selects nodes in the same entity class of target entity to enhance the difficulty and quality of samples. Negative sam- ple embeddings are formulated (Nv, ) where In) dd and present various different level of negative samples. The specific algorithm descrip- tion of the negative sample construction process is shown in Appendix Algorithm 1.
STEP 1: Taking the starting node estart (i.e. co) as the center point and search- ing outward along the relations, we obtain end nodes Ennd with different hop distance Hop(P(G, Estart, lena)) where Hop(-) denotes the hop distance and P(g, es (j) denotes the shortest path between e; and ej in the graph G. in Path and Hop(P(() eo, en)) = 3 in Path 6.
For example, Hop(P({ eg: eo0))
We leverage the hop distance to con- struct negative samples with different struc- turally difficulty levels, where Hop(.) = for
STEP

---

Financial
Medical
Models +
Tasks +
NER
TC
QA
QM
NED
Average
NER
QNLI
QM
Average
RoBERTa
78.92
82.61
82.20
91.28
92.56
85.51
65.89
94.63
85.68
82.07
BERT
77.56
83.68
83.01
91.70
92.46
85. 35.68+0.22
70.24
94.60
84.82
83.22+a.15
Conpe
80.67
84.43
83.98
91.96
92.78
86.761..19
73.35
94.46
86.38
84.7310.22
ERNIE-Baidu
82.99
84.75
84.46
92.26
92.39
87.3710.14
75.60
95.24
86.02
85.6210
ERNIE-THU
87.19
85.35
83.83
92.51
92.86
88.35+0.17
80.36
95.85
86.42
87 47.5444.11
KnowBERT
86.89
85.13
82.74
92.09
92.33
87. 87.84+0.21
79.84
94.97
85.87
86.8 86.89+a.14
K BERT
86.28
85.41
84.56
92.41
92.52
88. 88.24+0.25
79.72
95.93
86.64
87 87.43+0.19
KGAP
85.43
84.91
83.76
91.95
92.04
87 87.62+0.15
77.25
94.88
86.34
86. 86.16+0.24
DKPLM
86.83
85.22
83,95
91.86
92.57
88.0810.17
80.11
94.97
85,88
86 86.9910.14
GreaseLM
85.62
84.82
83,94
91.85
92.26
87.70+0.23
78.73
95,46
85,92
86 86.70+0.22
KALM
87.06
85.38
82.96
91.87
92.64
87.9810.16
80.12
94.84
85.93
86. 86.9610 13
KANGAROO
87.41
85.22
84.02
92.46
92.96
48.41
80.57
95.32
86.19
87.36=0.22
KANGAROO*
88.16
86.58
84.92
93.26
93.54
89.29
81.19
96.15
87.42
88.25 t0.17
Table 2: The performance of fully-supervised learning in terms of F1 (%). A and I indicate that we pre-train our model on CN-DBpedia (i.e., open domain KG) and the corresponding close-domain KG, respectively. The results of knowledge-awer PLMs baselines are pre-trained in closed-domate KGs. Best performance is shown in bold.
3.3 Training Objectives
of NER task in financial and medical domains are related to knowledge entities and the rest are im- plicitly contained. The fine-tuning models use a similar structure compared to KANGAROO, which simply adds linear classifier at the top of the backbone. From the results, we can observe that: (1) Compared with PLMs trained on open-domain corpora, KEPLMs with domain corpora and KGs achieve better results, especially for NER. It ver- ifies that injecting the domain knowledge can im- prove the results greatly. (2) ERNIE-THU and K-BERT achieve the best results among baselines and ERNIE-THU performs better in NER. We con- jecture that it benefits from the ingenious knowl- edge injection paradigm of ERNIE-THE which makes the model learn rich semantic knowledge in triples. (3) KANGAROO greatly outperforms the strong baselines improves the performance con- sistently, especially in two NER datasets (+0.97%, + 0.83%) and TC (+ (+1.).%) confirms that our KGs
our framework, the training objectives mainly consist of two parts, including the masked language model loss LMLM (Devlin et al., 2019) and the con- trastive learning loss LCL, formulated as follows:
Liotal = ALLMLM + deCoL
(8)
where A and X2 are the hyperparameters. As for the multi-level knowledge-aware contrastive learn- ing loss, we have obtained the positive sample K'y and negative samples I for each entity target hi, (i.e the textual embedding of token in an entity word). We take the standard InfoNCE (van den Oord et al., 2018) as our loss function LCL:
canthey NugGit
LCL Elog cooth, Ne7 1/7
where T is temperature hyperparameter and cos is the cosine similarity function.
model effectively utilizes the to enhance structural and semantic information. Few-Shot Learning To construct few-shot data, we sample 32 data instances from each training set and employ the same dev and test sets. We also fine-tune all the baseline models and ours using the same approach as previously. From Table 3, we observe that: (1) The model performance has sharp decrease compared to the full data experi- ments. The model can be more difficult to fit test- ing samples by the limited size of training data. In general, our model performs best in all the base- line results. (2) Although ERNIE-THU gets the best score Question Answer, its performances
4 Experiments
In this section, we conduct extensive experiments to evaluate the effectiveness of the proposed frame- work. Due the space limitation, the details of datasets and model settings are shown in Appendix B and the baselines are described in Appendix C.
4.1 Results of Downstream Tasks
Fully-Supervised Learning We evaluate the model performance on downstream tasks which are shown in Table 2. Note that the input format

---

on other datasets are far below our model. The performance of KANGAROO, ERNIE-THU and K-BERT better than others. We attribute this to their direct injection external knowledge into textual representations.
Fastaer
4.2
Detailed Analysis of KANGAROO
4.2.1
Ablation Study
(a) TransE embedting
(b) Hyperbolic embedding
We conduct essential ablation studies on four im- portant components with Financial NER and Medi- cal QM tasks. The simple triplet method simplifies the negative sample construction process by ran- domly selecting triplets unrelated to target entities. The other three ablation methods respectively de- tach the entity-class embeddings, the contrastive loss and the masked language model (MLM) loss from the model and are re-trained in a consistent manner with KANGAROO. As shown in Table 4, we have the following observations: (1) Com-
Figure 5: Visualization TransE and hyperbolic em- beddings.
0.0200
Training dinection
0.0175
Aligriment
0.0150
0.0125
0.0100
pared to the simple triplet method, our model has significant improvement. It confirms that Point- biconnected Component Data Augmenter builds rich negative sample structures and helps models learn subtle structural semantic further compen- sate the global sparsity problem. (2) verifies that entity class embeddings and multi-level contrastive learning pre-training task effectively complement semantic information and make large contributions to the complete model. Nonetheless, without the modules, the model is still comparable to the best baselines ERNIE-THU and K-BERT.
0.0075
Drapout
Replace words
0.0050
Ours (Level-1)
Ours (Level-2)
0.0025
-3.996
-3.994
-3.992 -3.990
-1.988 -3.986
Uniformity
Figure 6: Results comparison of ours and other data augmentation methods of alignment and uniformity.
to the concept-level nodes such as drug, check and cure. It illustrates that hyperbolic embeddings fit the hierarchical data better and easily capture the differences between classes.
4.2.2 The Influence of Hyperbolic Embeddings
this section, we comprehensively analyze why hyperbolic embeddings are better than Euclidean embeddings for the closed-domain entity-class hi- erarchical structure.
Performance Comparison of Different Embed- dings. We replace entity-class embeddings with Euclidean embeddings to verify the improvement of the hyperbolic space. To obtain entity-class em- beddings in the Euclidean space, obtain embed- dings of closed-domain KGs by TransE (Bordes et al., 2013) and take them as a substitution of entity-class embeddings. As shown Table 5, we evaluate the Fuclidean model in four downstream tasks, including NER and TC task in the financial domain, together with NER and QM in the medical domain. The results show that the performance degradation is clear in all tasks with Euclidean entity-class embeddings. Overall, the experimental results confirm that the closed-domain data distri- bution fits the hyperbolic space better and helps learn better representations that capture semantic and structural information.
Visualization of Embedding Space. To compare the quality of features in Euclidean and hyperbolic spaces, we train KG representations by TransE (Bordes et al., 2013) and the Poincaré ball model (Nickel and Kiela, 2017) 3017), visualizing the embed- dings distribution using t-SNE dimensional reduc- tion (van der Maaten and Hinton, 2008) shown in Figure 5. They both reflect embeddings grouped by classes, which are marked by different colors. However, TransE embeddings are more chaotic, whose colors of points overlap and hardly have clear boundaries. In contrast, hyperbolic represen- tations reveal a clear hierarchical structure. The root node is approximately in the center and links

---

Financial
Medical
Models +
Tasks +
NER
TC
QA
QM
NED
Average
NER
QNLI
QM
Average
RoBERTa
69.31
70.95
71.23
82.44
83.39
75.64
59.87
61.48
59.82
60.3911
BERT
68.46
72.62
72.81
81.57
83.61
75.81=1.95
63.28
51.72
57.96
57 37.65
Conpe
71.62
70.95
77.67
80.24
85.83
17.26
63.87
64.04
60.34
63.4212202
ERNIE-Baidu
75.86
76.85
76.67
84.69
85.13
29.8
69.44
64.78
61.74
65.3212.22
ERNIE-THU
77.72
78. 12
79.36
83.26
82.89
80.27
71.45
66.65
66.18
68 41.09
KnowBERT
79.35
77.62
79.54
85.77
84.33
81.32=2.13
68.84
65.62
63,96
66, 1.772
K BERT
76.92
75.11
78.59
84.87
83.99
79.90
69.81
65.73
72.60
69.38 69.38+199
KGAP
78.51
76.78
77.72
83.04
84.15
80.04-1.65
70.52
67.05
70.64
69.40+2.18
DKPLM
77.40
76.34
78.48
85.16
83.62
80.20
70.62
68.58
68.93
69.38
GreaseLM
79.70
78.24
77.94
83.06
84.27
80.64-1.92
71.47
67.58
69.41
69.49
KALM
76.93
76.81
78.03
84.30
83.84
79.98
70.23
66.87
70.28
69. 69.1311.04
KANGAROO
79.25
78.41
78.29
83.62
85,45
81 +
68.50
68.29
71.23
69.34=1.42
KANGAROO*
81.61
80.73
79.98
87.92
86.12
83.27 # 1.00
73.42
70.34
75.32
73.03 11.72
Table The overall results of few-shot learning in terms of F1 (%).
Models J. Tasks +
Fin. NER
Med. QM
Neg.
Pos.
L1 //12/L3
KANGAROO
88,16
87.42
w/o Simple Triplets
87.81
86.46
KANGAROOf
0.0911
0.0061 0.0044 0.0018
w/o Entity Class
87.89
86.
Dropout 0.0991
-0.0177
w/o Contrastive Loss
87.86
86.61
Word Rep.
0.0992
-0.0144
w/o MLM
87.92
86.78
Table 6: The averaged cosine similarities of posi- tive/negative samples. "L.I" means the Level-1 samples.
Table The performance models for ablation study in terms of F1 (%).
ples help to improve the quality of data augmen- tation. We visualize the alignment and uniformity metrics ( (Wang and Isola, 2020) of models during training. To make this more intuitive, we use the cosine distance to calculate the similarity between representations. The lower alignment shows sim- ilarity between positive pairs features and lower uniformity reveals presentation preserves more in- formation diversity. As shown in Figure 6, our models greatly improve uniformity and alignment stcadily to the best point.
Tasks + Models
Financial
Medical
NER
TC
NER
QM
Euclidean
87. 29.99
86. 16
80.34
86.30
Hyperholic
86.16
86.58
81.19
87.42
Table 5: The model results when Euclidean and hyper- bolic embeddings are employed in terms of Fl (%).
4.2.3 The Influence of Point Biconnected Component-based Data Augmentation To further confirm that our data augmentation tech- nique for contrastive learning is effective, we ana- lyze the correlation between positive and negative samples w.r.t. target entities. We choose two strate- gies (i.e., dropout (Gao al., 2021) and word replacement (Wei and Zou, 2019) for positive sam- ples) as baselines. The negative samples are ran- domly selected from other entities. As shown in Table 6, we calculate the averaged cosine similar- ity between samples and target entities. In posi- tive samples, the cosine similarity of our model is lower than in baselines. illustrating the diversity between positive samples and target entities. As for negative samples, we design the multi-level sam- pling strategy in our model, in which Level-1 is the most difficult followed by Level-2 and Level-3. The diversity and difficulty of the negative sam-
Related Work
Open-domain KEPLMs. We summarize previ- ous KEPLMs grouped into four types: (1) Knowl- edge enhancement by entity embeddings (Zhang et al., 2019; Su et al., 2021; Wu et al., 2023). (2) Knowledge-enhancemet (Wang et al., 2021; Zhang et al., 2021a). (3) Knowledge-enhanceme
by text descriptions
by converted triplet's texts (Liu et al., 2020a; Sun et al., 2020). (4)
Knowledge-enhancemmnt by retrieve the external text and token embedding databases (Guu et al., 2020; Borgeaud et al., 2022).
Closed-domain KEPLMs. Due to the lack of in-domain data and the unique distributions domain-specific KGs (Cheng et al., 2015: Savnik

---

al., 2021), previous works of closed-domain KE- PLMs focus on three domain-specific pre-training paradigms. (1) Pre-training from Scratch. For ex- ample, PubMedBERT (Gu et al., 2022) derives the domain vocabulary and conducts pre-training us- ing solely in-domain texts, alleviating the problem of out-of- (2) Continue Pre-traines Pre-training. These works (Beltagy al., 2019; Lee et al., 2020) have shown that using in-domain texts can provide additional gains over plain PLMs. (3) Momed-DDmain Pre-training (Liu et al., 2020b; Zhang et al., 2021b). this approach, out-domain texts are still helpful and typically ini- tialize domain-specific pre-training with general- domain language model and inherit its vocabulary. Although these works inject knowledge triples into PLMs, they pay little attention to the in-depth char- acteristics of closed-domain KGs (Cheng al.. 2015; Kazemi and Poole, 2018; Vashishth al., 2020), which the major focus our work.
have some negative impacts, such as gender and social bias. Our work would unavoidably suffer from these issues. We suggest that users should carefully address potential risks when the KAN- GAROO models are deployed online.
Acknowledgements
and perplexing domain terms.
This work supported by the National Natural Science Foundation of China under grant number 62202170 and Alibaba Group through the Alibaba Innovation Research Program.
References
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR, abs/1607.064500
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In EMNLP, pages 3613-3618.
Antoine Bordes, Nicolas Usunier, Alberto Gareía- Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. In NIPS, pages 2787-2795.
6 Conclusion
In this paper, we propose unified closed- domain framework named KANGAROO to learn knowledge-aware representations via implicit KGs structure. We utilize entity enrichment with hy- perbolic embeddings aggregator to supplement the semantic information of target entities and tackle the semantic deficiency caused by global sparsity. Additionally, we construct high-quality negative samples of knowledge triples by data augmentation via local dense graph connections to better capture the subtle differences among similar triples.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann. Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy. Jacob Menick, Roman Ring. Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving. Oriol Vinyals. Simon Osindero, Karen Si- monyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2022. Improving language models by retrieving from trillions of tokens. In ICML, pages 2206 22400
Limitations
Wenliang Cheng, Chengyu Wang. Bing Xiao, Weining Qian, and Aoying Zhou. 2015. On statistical char- acteristics of real-life knowledge graphs. In BPOE, pages 37-49.
KANGAROO only captures the global sparsity structure in closed-domain KG with two knowledge graph embedding methods, including euclidean (e.g. transE (Bordes et al., 2013)) and hyperbolic embedding. Besides, our model explores two rep- resentative closed domains (i.e. medical and finan- cial), and hence we might omit other niche domains with unique data distribution.
Jacob Devlin, Ming-Was Wing-Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre- pre-training of deep bidirectional transformers for language under- standing. In NAACL, pages 4171-4186.
Shangbin Feng. Zilong Chen, Qingyao Li, and Min- nan Luo. 2021. Knowledge graph augmented po- litical perspective detection in news media. CoRR. abs/2108.0386/
Ethical Considerations
Shangbin Feng, Zhaoxuan Tan, Wenqian Zhang, Zhenyu Lei, and Yulia Tsvetkov. 2022. KALM: knowledge- aware integration of local, document, and global con- texts for long document understanding. In ACL.
Our contribution in this work is fully method- ological, namely new pre-training framework of closed-domain KEPLMs, achieving the perfor- mance improvement of downstream tasks. Hence, there is no explicit negative social influences in this work. However, Transformer-bawee models may
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. In EMNLP, pages 6894-6910.

---

Yu Gu, Robert Tinn, Hao Cheng. Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2022. Domain-specific lan- guage model pretraining for biomedical natural lan- guage processing. ACM, 3(1):211-2225
Matthew Peters, Mark Neumann, Robert L. Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contex- tual word representations. EMNLP, pages 43-54,
Iztok Savnik, Kiyoshi Nitta, Riste Skrekovski, and Nikolaus Augsten. 2021. Statistics of knowledge graphs based on the conceptual schema. CoRR, 1bs/2109.09391.
Kelvin Guu, Kenton Lee, Zora Tung. Panupong Pasu- pat, and Ming-Was Wei Chang. 2020. REALM: retrieval- augmented language model pre-training. CoRR, abs/2002.08909.
Yusheng Su, Xu Han, Zhengyan Zhang, Yankai Lin, Peng Li. Zhiyuan Liu, Jie Zhou, and Maosong Sun. 2021. Cokebert:
Raia Hadsell, Sumit Chopra, and Yann LeCun, 2006, Dimensionality reduction by learning an invariant mapping. In CVPR, pages 1735-1742.
Contextual knowledge selection and embedding towards enhanced pre-trained language models. Al Open, 1:1271134.
Yun He, Ziwei Zhu, Yin Zhang, Qin Chen, and James Caverlee. 2020. Infusing disease knowledge into BERT for health question answering, medical in- ference and disease name recognition. In EMNLP. pages 4604-4614.
Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. 2020. Colake: Contextualized language and knowledge em- bedding. In COLING, pages 3660-3670.
Dan Hendrycks and Kevin Gimpel. 2016. Bridging non- lincarities and stochastic regularizers with gaussian error linear units. CoRR, abs/ 100.00845.
Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. ERNIE: enhanced representation through knowledge integration. CoRR, abs/1904.09223.
Seyed Mchran Kazemi and David Poole. 2018. Simple embedding for link prediction in knowledge graphs. In NeurlPS, pages 4289-4300.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. CoRR, tbs//807.00774.
Yuxuan Lai. Yijia Liu, Yansong Feng, Songfang Huang. and Dongyan Zhao. 2021. Lattice-bert: Leverag- ing multi-gramularity representations in chinese pre- trained language models. In NAACL, pages 1716- 1731.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. pages 2579-2605.
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin. Nilesh Agrawal, and Partha P. Talukdar. 2020. Inter- acte: Improving convolution-baset knowledge graph embeddings by increasing feature interactions. In AAAI, pages 3009-3016.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinform., 36(4): 1234-1240.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998-6008.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang. Qi Ju. Haotang Deng, and Ping Wang. 2020a. K-BERT: enabling language representation with knowledge graph. AAAI. pages 2901-2906
Ivan Vulic, Edoardo Maria Ponti, Robert Litschko, Goran Glavas, and Anna Korhonen. 2020. Prohing pretrained language models for lexical semantics. In EMNLP, pages 7222-7240.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Chengyu Wang, Minghui Qiu, Jun Huang, and Xiaofeng He. 2020. Meta fine-tuning neural language models for multi-domain text mining. EMNLP, pages 3094-3104.
Zhuang Liu, Degen Huang. Kaiyu Huang. Zhuang Li, and Jun Zhao. 2020b. Finbert: A pre-trained finan- cial language representation model for financial text mining. In IJCAL, pages 4519
Chengyu Wang. Minghui Qiu, Taolin Zhang, Tingting Liu, Lei Li, Jianing Wang, Ming Wang. Jun Huang, and Wei Lin. 2022. Easynlp: A comprehensive and easy-to-use toolkit for natural language processing. In EMNLP, pages 22-29.
Maximilian Nickel and Douwe Kiela. 2017. Poincaré embeddings for learning hierarchical representations. NeurlPS, pages 6338-66 6347.
Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Trans- fer learning biomedical natural language process- ing: An evaluation of BERT and elmo on ten bench- marking datasets. In BioNLP, pages 58-65.
Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. ICML, volume 119, pages 9929-9939. PMLR.

---

Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang. Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021. KEPLERR A unified model for knowledge embed- ding and pre-trained language representation. Trans. Assoc. Comput. Linguistics, 9:176-194.
A Indicators of Closed-domain KGs
The following explanations are the different indi- cators to analyze the closed-domain KGs.
#Nodes and #Edges are the numbers of nodes and edges in the corresponding KG.
Jason W. Wei and Kai Zou. 2019. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In EMNLP, pages 6381- 6387.
Coverage Ratio is the entity coverage rate of the KG in its corresponding text corpus. We calculate it by the percentage of entity tokens matched in the KG by the number of the full- text tokens, formulated as CR f. where te is the number of entity tokens and & is the number of all textual tokens. Texts for closed domains the same the pre-training cor- pora to be described in the experiments. The corpora for the open domain is taken from the CLUE benchmark
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. Medklip: Medical knowl- edge enhanced language-iman pre-training. CoRR. abs/2301.02225
Bo Xu, Yong Xu, Jiaqing Liang, Chenhao Xie, Bin Liang, Wanyun Cui, and Yanghua Xiao. 2017. Cn- dbpedia: A never-ending chinese knowledge extrac- tion system. IEA/AIE, volume 10351, pages 428- 438.
Taolin Zhang, Zerui Cai, Chengyu Wang, Peng Li, Yang Li, Minghui Qiu, Chengguang Tang, Xiaofeng He, and Jun Huang. 2021a. HORNET: enriching pre- trained language representations with heterogeneous knowledge sources. In CIKM, pages 2608-2617. ACM.
%Max Point Biconnected Component (i.... %Max PBC) is the number nodes in the biggest point biconnected component divided by the number of all nodes. Note that point biconnected component is graph such that if any node is removed, the connectivity is not changed.
Taolin Zhang, 7erui Cai, Chengyu Wang, Minghui Qiu, Bite Yang, and Xiaofeng He. 2021b. Smedbert: A knowledge-enhanced pre-trained language model with structured semantics for medical text mining. In ACL, pages 5882-5893.
Subgraph Density is the average density of 100 random subgraphs where each of them contains 10% of total nodes of the KG. Note that the density of graph G (V, E) for- mulated as [V[[V]])) 1)) where E is the number of edges and V is the number nodes.
Taolin Zhang, Chengyu Wang, Nan Hu, Minghui Qiu, Chengguang Tang. Xiaofeng He, and Jun Huang. 2021c. DKPLM: decomposable knowledge- enhanced pre-trained language model for natural lan- guage understanding. CoRR, abs/21 12.01047.
B Data and Model Settings
Taolin Zhang, Chengyu Wang, Nan Hu, Minghui Qiu, Chengguang Tang, Xiaofeng He, and Jun Huang. 2022a. DKPLM: decomposable knowledge- enhanced pre-trained language model for natural lan- guage understanding. In AAAI, pages 11703-11711.
B.1 Pre-training Corpus
Our experiments are conducted in two representa- tive closed domains, including finance and medical. The pre-training corpus of the financial domain is crawled from several leading financial news web- sites in China, including SOHU Financial?, Cai Lian Press and Sina Finance", etc. The corpus of the medical domain is from Chinese network community of professional doctors called DXY Bulletin Board System After pre-processing, the
Xikun Zhang. Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D Manning, and Jure Leskovee. 2022b. Greaselm: Graph reason- ing enhanced language models. In ICLR.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: enhanced language representation with informative entities. In ACL, pages
*ee use the Chinese Wikipedia corpus "wiki2019zh" and the corpus "news2016zh" the pre-training cor- pora. http:://gghub..combbrrrnmmmme wwwbbriigimmarrrtnnnniiiio "https://en, wi wikipedia. org/ wwwikk//DDnesssuurrpppp
Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang. 2022c. Sg-net: Syn- tax guided transformer for language representation. IEEE Trans. Pattern Anal. Mach. Intell., 44(6):3285- 3299.
hatus://businessssohh som.
" hhups://www.cls.cc//
" "hups://finance sina com

---

Dataset
#Train
#Dev
#Test
Algorithm 1 Negative Samples Construction
1: Input: Knowledge graph 9 = (E, R.), entity 6anns level
NER
23,675
2,910
3,066
D. sample length L
TC
37,094
4,122
4,580
2: Output: Sequence of negative sample S
Fin.
QA
612,466
76,781
76,357
3: S + emppy sequence
4: while len( < L do
QM
81,000
9,000
10,000
5: (Cann) &- {e E]Hop(P(GG Cutri e)) D 11
NED
4,049
450
500
6: Na(Cam) < {e [e (CCaun)]llass(smmn class(e))
7:
if (.(an)) Ø then
NER 34,209
8,553
8,553
8:
fend + random_ seaninsseeeeee/eeeeeeeeeee/eeere (oaccctt/Veesee (esur))
Med.
QM 16,067
1,789
1,935
9:
else
10:
Fend random_scleett (a end if
QNLI 80,950
9,066
9,969
11:
12:
concatenate(S, P(G, Cuuat Gend))
Table 7: The number of samples of the datasets in finan- cial and medical domain, respectively.
13:
E' - P(G, Cauni Goad) (Crur Gold)
14:
R' (r(es.e2)leI,ee. E
15:
g' (E',
16:
Cunn and Can connected G' then
S concatenate(S, P(G', Esuns Gend))
financial corpus contains 12,321,760 text segments and that number segments in the medical corpus is 9,504,007.
end
while
20:
return
B.2 Knowledge Graph
B.4 Model Hyperparameters
As for KGs, FinKG contains five types of classes including companies, industries, products, people and positions with 9,413 entities and 18,175 triples. The MedKG is discase-related, containing diseases, drugs, symptoms, cures and pharmaceutical fac- tories. It consists of 43,972 entities and 296,625 triples.
We adopt the parameters of BERT released by Google to initialize the Transformer blocks for text encoding. For optimization, we set the learning rate as 5e 5 the max sequence length as 128, and the batch size as 512. The dimension of embed- ding d1, d2, d3 and d4 are set as 768, 100, 100 and 768, respectively. The number of the Text Encoder layers N is 5 and the number of the Knowledge Encoder layers M is 6. The temperature hyperpa- rameter is set to I and the coefficients λι and A2 are both 0.5. The number of negative samples lev- els & is 3. Pre-training KANGAROO takes about 48 hours per epoch on NVIDIA A100 GPUs (80GB memory per card). Results are presented in average with 5 random runs with different random seeds and the same hyperparameters.
B.3 Downstream Tasks
We use five financial datasets and three medical datasets evaluate model in full and few-shot learning settings. Financial task data is obtained from public competitions and previous works, in- cluding Named Entity Recognition" (NER), Text Classification' 12 (TC), Question Answering (QA), Question Matching 14 (QM) and Negative Entity Discrimination' (NED). Medical data is taken from lue tasks and DXY Company!7. Medical datasets contain NER, Question Matching and Question Natural Language Inference (QNLI). For most competitions where only train and dev sets are available for us, we randomly slice the datasets, making the proportion of train/dev/test ratio close to 9/1/1. The train/dev/test data distribu- tion and data sources are shown in Table 7.
B.5 Model Notations
We denote the input token sequence as [t1, fe: ...ppp input sequence. to means one of the tokens of an entity word that belongs to a specific entity class C, such as the disease class in the medical domain. We obtain hidden feature of tokens [h,, [h,))_ E IR by the Text Encoder which is composed by N Transformer layers (Vaswani et al., 2017) where dj is the PLM's output dimension. d2 is the dimension of entity class embedding. Furthermore, the knowledge graph G (E,R) consists of the entities set & and the relations set R. The
where is the length of
http::///mmbddddigggihbb... hhtps:/www..hhinnttaaay//ommtttr
data/
14 htttss//fiitubbboom//aooovveeeee 15 10 hhtp:////gggg.ooooottbbbbeeeeeen
13
17 "https///utt.dxyy

---

Music
Social
Average
et al., 2021c) specifically focuses on long-tail enti- ties, decomposing the knowledge injection process of three PLMs' stages including pre-training. fine- tuning and inference. GreaseLM (Zhang et al., 2022b) fuses encoded representations from PLMs and graph neural networks over multiple layers of modality interaction operations. KALM (Feng et al., 2022) jointly leverages knowledge in local, document-level, and global contexts for long docu- ment understanding.
Models 4 Tasks
D1
D2
RoBERTa
57.34
70.86
63.29
63.83 18
BERT
54.26
67.13
62.58
61.32++.22
Compt
59.43
72.04
63.97
65.1510..
ERNIE-Baidu
59.43
73.60
65.88
66.3010.14
ERNIE-THU
58.62
72.92
65,10
65.55+0.17
KnowBERT
99.91
74,84
66.36
67. 67.04+0.24
K K-BERT
58.74
73.71
67.01
66.49 66.49+0.19
KGAP
99.62
74.35
66.82
66.93+0.22
DKPLM
57.33
73.64
67.43
66.13
GreaseLM
60.18
74,95
68.14
67.76=0.20
KALM
58.94
73.55
67.85
66.7820.14
KANGAROO
61.51
75.89
69.27
68.8910.18
D Model Performance
Table Results of other domains in terms of Acc (%).
D.1 Other Domain Results
In the table 8, we supplement data from two do- mains including music domain and social domain to further demonstrate the effectiveness of the pro- posed KANGAROO model. We conduct exper- iments in the largest Chinese knowledge graph database (i.e., openKG), Both the music and so- cial data are crawled from the largest open-source knowledge graph database in Chinese and Baidu BaiKe. We evaluate the baselines and our kanga- roo model in full data fine-tuning settings and the results shown are follows. The downstream datasets of music D1, music D2 and social DI are text classification tasks, which 'D1' means Dataset Specifically, Music DI and D2 are the mu- sic emotion text classification tasks that download song comment data on the music app. Social DI is classifying the relationships between public figures such as history and entertainment. The evaluation metric is accuracy (i.c., ACC).
triplet set is St {(e; ((e, E, e,r(e,,)) RJ where is the head entity with relation r(e;, eÿ) to the tail entity ej.
ej E
C Baselines
In our experiments, we compare our model with general PLMs and KEPLMs in base level parame- ters with knowledge embeddings injected: Gen- eral PLMs: BERT (Devlin et al., 2019) is pre-trained Transformer layers initialized by pub- lie weights. RoBERTa (Liu et al., 2019) is the RoBERTa model pre-trained with a Chinese cor- pus, which improves dynamic masking and the training strategy of BERT. Conp. is the continual pre-trained BERT domain pre-training data. It further helps to improve the original BERT model performance in closed domains.
KEPLMs: For the fairness of the results, all the KEPLMS are reproduced via the closed domain KG and our pre-training corpus using official open source code. ERNIF-BBadu is the KEPLM (Sun et al., 2019) that adds external knowledge through entity and phrase masking. ERNIE-THH THU (Zhang al., 2019) encodes the graph structure of KGs with knowledge embedding algorithms and injects it into contextual representations. KnowBERT (Pe- ters et al., 2019) proposes the entity linkers and self- supervised language modeling objective are jointly trained end-to-end in multitask setting that com- bines small amount of entity linking supervision with large amount of raw text. K-BERT (Liu et al., 2020a) KEPLM that converts the triples into the sentences domain knowledge. KGAP (Feng al., 2021) adopts relational graph neural net- works and conduct political perspective detection as leaph classification tasks. DKPLM (Zhang
D.2 K-hop Thresholds' Results
In the table 9, we consider three different situations to discuss the k-hop thresholds selection including (1) k- h-hop triple path as positive and k+1, k+2, k+3 hop triple path as negative (2) k, k+1, k+2 triple path as negative and k+3 triple path positive (3) k hop triple path viewed as both positive and negative samples and k+1, k+2 as negative samples. Specifically, we seleet the original closed domain pre- tre-iraining and KG data to perform full data fine- tuning tasks. For the above three situations, in order to prevent overlapping triple path's 's conflicts between positive and negative during the sampling process, mask the sampled triple path data in the iterative sampling process.
The "S1" means "Situations 1". Fin and Med means Financial and Medical respectively. From the above table, we can observe that (1) The closer

---

Situations Models
Fin NER
Fin TC
Fin QA
Fin QM
Fin NED
Med NER
Med QNLI
Med QM
S1
k=1
88.16
86.58
84.92
93.26
93.54
81.19
95.15
87.42
S1
k=1
87.43
85.73
84.06
92.50
92.78
80.31
95.98
86.48
S1
k=3
85.33
83.27
81.97
91.02
91,42
76.64
93.75
84.81
S2
k=1
82.08
82.94
81.74
90.27
90.98
70.38
93.28
84.06
S2
k=2
80.96
82.28
80.85
89.85
88.33
68.85
90.44
82.15
S2
k=3
79.45
79.96
78.49
87.46
87.56
64.72
87.30
81.57
S3
k=1
87.82
86.26
84.63
92.98
93.18
80.90
95.03
87.20
S3
k=2
85.29
83.55
82.50
91.54
91.82
78.75
93.27
85.32
S3
k=3
81.99
81.30
80.41
90.66
88.46
75.41
90.73
81.25
Table 9: Results w.rt. k-hop thresholds.
the positive sample hop path is to the target entity, the better the model performance (See S1 results). (2) The closer the negative samples are sampled to the target entity or even closer than positive sam- ples, the performance of the model will sharply de- crease (See S2 and S3 results). Hence, should construct positive samples closer the target entity, while negative samples should not be too far away simultaneously in graph path to avoid introducing too much knowledge noise.


Reviews:
[{"review_id": "twN0SkEHlg", "reviewer": "Reviewer_m3BN", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Section 3.1 may have further details about the component and how it interacts with the remaining ones instead of barely introducing its submodules, resulting in redundancy with the previous few lines of Section 3.\nThe Hyperbolic Knowledge-aware Aggregator module in section 3.1 mainly integrates the representation of domain knowledge in hyperbolic space with the context-aware representation generated by pre-trained language models. Then the fused knowledge-enhanced token representations are transferred to pre-training tasks, performing the underlying MLM task and our proposed section 3.2 Multi-Level Knowledge-aware Contrastive Learning Augmenter. The knowledge-enhanced token representations further benefit the positive and negative samples' representations due to the point-biconnected components subgraph implicit structure.\n\n# Evaluating the proposed system on further domains would completely enlighten the advantages and issues that KANGAROO has in its application, revealing additional insights that may depend on the execution domain.\nDue to the limited time available for the rebuttal stage, we first supplement data from two domains including music domain and social domain to further demonstrate the effectiveness of the proposed KANGAROO model. We conduct experiments in the largest Chinese knowledge graph database (i.e., openKG). Both the music and social data are crawled from the largest open-source knowledge graph database in Chinese and Baidu BaiKe. We evaluate the baselines and our kangaroo model in full data fine-tuning settings and the results are shown are follows. The downstream datasets of music D1, music D2 and social D1 are text classification tasks, which 'D1' means Dataset 1. Specifically, Music D1 and D2 are the music emotion text classification tasks that download song comment data on the music app. Social D1 is classifying the relationships between public figures such as history and entertainment. The evaluation metric is accuracy (i.e., ACC). We will supplement the results in the appendix after the paper is accepted. \n| Model |  Music D1 | Music D2 | Social D1| Average| \n| :--------: | :--------: | :--------: | :--------: | :--------: | \n| RoBERTa | 57.34 | 70.86 | 63.29 | 63.83$_\\{\\pm 0.18}$ | \n| BERT | 54.26 | 67.13 | 62.58 | 61.32$_\\{\\pm 0.21}$ | \n| Con_pt | 59.43 | 72.04 | 63.97 | 65.15$_\\{\\pm 0.20}$ | \n| ERNIE-Baidu | 59.43 | 73.60 | 65.88 |  66.30$_\\{\\pm 0.14}$ | \n| ERNIE-THU | 58.62 | 72.92 | 65.10 | 65.55$_\\{\\pm 0.17}$  | \n| KnowBERT | 59.91 | 74.84 | 66.36 |  67.04$_\\{\\pm 0.24}$ | \n| K-BERT | 58.74 | 73.71 | 67.01 | 66.49$_\\{\\pm 0.19}$ | \n| KGAP | 59.62 | 74.35 | 66.82 | 66.93$_\\{\\pm 0.22}$ | \n| DKPLM |57.33| 73.64 | 67.43 | 66.13$_\\{\\pm 0.15}$ | \n| GreaseLM | 60.18| 74.95 | 68.14 | 67.76$_\\{\\pm 0.20}$ | \n| KALM | 58.94 | 73.55 | 67.85 | 66.78$_\\{\\pm 0.15}$ | \n| KANGAROO | 61.51 |  75.89 | 69.27 | 68.89$_\\{\\pm 0.18}$ |\n\nWe sincerely thank for your time and efforts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors implement an exciting framework to advance the NER state-of-the-art regarding two closed domains. However, the designed solutions hold a general approach to the problem that can migrate to other closed domains.\n\nThe proposal presents a strong structure, a good formalisation of the discussed issue and an in-depth description of the implemented framework. Also, the authors made available their code to reproduce the entire experiment.\n\nThe idea is intriguing and produces evident advances in the state-of-the-art. Also, it may lead to further and stimulating discussion about the characteristics of Knowledge we are currently employing in specific scenarios. The EMNLP audience will receive benefits in discussing such a proposal.", "reasons_to_reject": "The related work needs improvements since it is just a list of works without effectively positioning the proposal in the current literature. Its similarities and novelty are here to be shown.\n\nAlso, the limitations could be more actively written. It misses some limitations, especially technical, that the framework has in leveraging close-domain tasks.\n\nSection 3.1 may have further details about the component and how it interacts with the remaining ones instead of barely introducing its submodules, resulting in redundancy with the previous few lines of Section 3.\n\nEvaluating the proposed system on further domains would completely enlighten the advantages and issues that KANGAROO has in its application, revealing additional insights that may depend on the execution domain.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a Knowledge enhanced language representation learning framework for closed-domain tasks by capturing implicit graph structures among the entities. To achieve such a goal, the KANGAROO system exploits hyperbolic embeddings of deep hierarchical entity-class structures, avoiding entities' global sparsity and locally dense neighbour subgraphs through a data augmentation (for negative triples) strategy based on contrastive learning.\n\nIn detail, the framework contains an Entity Encoder, divided into an entity space infusion and entity knowledge injector, to encode textual input by including knowledge, semantic and structural information. While the model learns the last two data, the former is injected in the embeddings holding the semantic and structural information of the related domain. A Multi-level knowledge-aware augmenter intervenes to learn the semantic gaps of injected triples by augmenting locally existing data with negative samples. Then, the model dedicated to solving a Named Entity Recognition (NER) task learns to solve the specific goals by exploiting the principle of the masked language model and the contrastive learning paradigm.\n\nThe experiments show superior performances compared to the baseline, besides providing an in-depth ablation study to show the various components' influences.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kCcplV6cZ2", "reviewer": "Reviewer_Z6FT", "summary": "", "questions": "1. In Equation 5, how the multi-headed self-attention can have the multi-inputs and multi-outputs.\n\n2. Why set only one-hop node in the knowledge graph as positive samples? And why set all one-hop nodes in the knowledge graph as positive samples?\n\n3. Why one-hop nodes in the knowledge graph can not be negative samples?\n\n4. Give more details in Section 4.2.3.\n\n5. Give more details in Equation 5.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Some report results are close to baselines. The authors should do a significate test to prove the performance and improvements are significant\nIn all experimental settings, results are presented in average with 5 random runs with different random seeds and the same hyperparameters (see Appendix B.4) to ensure the reliability and significance of the results. Meanwhile, these domain tasks are relatively difficult due to the domain text term understanding. Both our model and the previous baselines models have not made much progress, failing to achieve the significant performance improvement that was expected. To further ensure the significance of the results, we conduct a t-tests experiment demonstrate the improvements of models are statistically significant with p < 0.05 level using the original output results.\n\n# The setting of positive and negative samples in Section 3.2.1 is quite strange.\n(1) For the positive and negative samples construction, these NLP data augmentation methods in contrastive learning are generally heuristic construction based on task scenarios such as dropout, delete word and back-translation. In KEPLMs, the retrieved knowledge triples viewed as positive or negative are defined based on the distance on the graph. The closer the distance in graph, the smaller the semantic gap of these triples. Meanwhile, considering that if the negative samples are too far away in the KG, the model finds it too easy to distinguish between positive and negative samples through contrastive learning, which is not conducive to the robustness of the model.  \n\n(2) For the negative sample include the positive samples problem, we select the negative triple graph path exclude the positive samples.  There is usually more than one triple around the target entity.  \n\n(3) For the positive and negative samples' settings changed problems,  we discuss the changes in different negative samples and the corresponding changes in section 4.3.2 using two important metrics in contrastive learning (i.e., Alignment and Uniformity). Our learning speed is faster and more stable and we perform the specific result as follows. We evaluate our model with different negative sample methods in three medical and financial tasks with full data fine-tuning settings during the rebuttal stage.\n| Model |  Medical QM | Medical QNLI | Financial TC |\n| :--------: | :--------: | :--------: | :--------: |\n| Replace Word | 87.05 | 95.73 | 86.04 | \n| Dropout | 87.17 | 95.84 | 86.29 | \n| Our Level-1 | 87.94 | 96.11 | 86.35 | \n| Our Level-2 | 88.19 | 96.08 | 86.49 | \n\n# Give more details in Equation 5\nIn Equation 5, we generally formalize the input and output of each layer of the Transformer into a function $\\mathcal{F}_m^v$. In our implementation, our model not only includes multi-head self attention but also includes FFN layers, which means its inputs are the same as the Transformer inputs of each layer. Hence, our model input in Equation 5 can be viewed as Transformer input.\n\nWe sincerely thank for your time and efforts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper proposes one method with two components for enhancing the training process based on the structure of the close-domain Knowledge domain of Pre-trained Language Models.\n\n* The authors evaluate their approach on two open datasets, providing a comprehensive evaluation of the proposed method. Based on the reported results, the proposed method may be effective.", "reasons_to_reject": "The primary concerns of this paper are,\n\n* Some report results are close to baselines. The authors should do a significate test to prove the performance and improvements are significant.\n\n* The setting of positive and negative samples in Section 3.2.1 is quite strange. First, the motivation for utilizing all first-hop nodes in the knowledge graph as the positive sample is unclear. Second, the motivation for utilizing other hops' nodes in the knowledge graph as negative samples is also unclear. Third, the experiments didn't show if the positive and negative samples' settings changed, how the result will be changed. Fourth, for negative samples, the authors choose the path from the centre node. It must include the positive samples. \n\nOther minor concerns include that in Section 4.2.3, the motivation and setting (such as baselines) are unclear. Specifically, I can not understand the baseline from the current description. \n\nThen, in Equation 5, how the multi-headed self-attention can have the multi-inputs and multi-outputs.", "questions_to_authors": "1. In Equation 5, how the multi-headed self-attention can have the multi-inputs and multi-outputs.\n\n2. Why set only one-hop node in the knowledge graph as positive samples? And why set all one-hop nodes in the knowledge graph as positive samples?\n\n3. Why one-hop nodes in the knowledge graph can not be negative samples?\n\n4. Give more details in Section 4.2.3.\n\n5. Give more details in Equation 5.", "paper_topic_and_main_contributions": "This paper focuses on Knowledge-Enhanced Pre-trained Language Models (KEPLMs) and proposes a Knowledge-enhanced lANGuAge Representation learning framework for various clOsed dOmains (KANGAROO).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jPwAq8cTAQ", "reviewer": "Reviewer_fcFt", "summary": "", "questions": "What is the running time of the negative sampling methods?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Lack of experiments in general domain KG injection into pre-trained language models\n（1）In this domain knowledge enhanced pre-trained language scenarios, our main experiments are evaluated our model in domain data setting including pre-training data and knowledge graph data. In order to verify our model in general domain, we also supplement the experimental results in Table 2 KANGAROO$^\\Delta$ settings. From the experimental results, it can be seen that our kangaroo algorithm is more effective for domain models than general data enhanced model.  \n（2）Due to the limited time available for rebuttal stage and the large amount of data in the general domain, we will further supplement the relevant data results after paper acceptance. At present, we can only achieve these data results during the rebuttal stage, selecting three recent SOTA KEPLMs (i.e., GreaseLM, KALM and DKPLM). All the Chinese open domain pre-training data is constructed by authoritative institution WUDAO and the open domain KG data is the same as KANGAROO$^\\Delta$ downloaded from the CN-DBpedia. The experimental results are full data fine-tuning settings.\n\n| Model |  Fin NER | Fin TC | Fin QA | Fin QM| Fin NED| Med NER | Med QNLI| Med QM|\n| :--------: | :--------: | :--------: | :--------: | :--------: |  :--------: | :--------: | :--------: | :--------: | \n| DKPLM     | 83.25 | 81.64 | 79.83 | 90.34 | 92.59 | 68.71 | 93.22 | 84.90| \n| GreaseLM | 81.86 | 83.95 | 82.86 | 90.31 | 92.62 | 74.52 | 93.80 | 84.98 | \n| KALM.       | 82.73 | 82.93 | 82.16 | 91.01 | 90.37 | 76.82 | 94.08 | 85.47 | \n| KANGAROO     | 84.99 |  83.81| 83.63 | 91.48 | 92.58 | 78.11 | 94.13 | 86.32 | \n\nFrom the above table, we can draw a conclusion roughly: (1) The results of knowledge augmentation method trained using large-scale pre-trained corpus in the open domain is generally better than the underlying model (i.e., BERT, ROBERTA) in the closed domain. We conjecture that the size of the pre-trained corpus in open and closed domain resulting in the KG data can not be used effectively. (2) Our model can still demonstrate superior performance in open domain large-scale pre-trained corpus.\n\n# What is the running time of the negative sampling methods?\nFor the running time of the negative sampling methods, it will vary depending on the dense structure of the KG data in different domains and computing power of the machine. According to our domain KG and machine, we need about 1~2 seconds for choosing negative samples for a target entities, whereas the speed of positive triple sampling is very fast about 0.1 seconds.\n\nWe sincerely thank for your time and efforts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written. The proposed model is well-justified and clearly explained.\n2. The proposed model achieves state-of-the-art results.", "reasons_to_reject": "1. Lack of experiments in general domain KG injection into pre-trained language models.", "questions_to_authors": "What is the running time of the negative sampling methods?", "paper_topic_and_main_contributions": "The paper proposed a model for injecting domain knowledge on a pretrain language model. The main contribution includes the use of hyperbolic embeddings and a robust negative sampling to address local density problem.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new representation learning framework for closed-domain tasks, wherein representations are trained to capture implicit graph structure within the modelled data. The framework is shown to lead to improved performance on named entity modelling on two benchmarks. 

Beyond simply demonstrating state of the art results, the paper is also interesting as the results may lead to further discussion of what knowledge structures entity embeddings encode. The paper is well-written and clear, and as such provides a good starting point for such a discussion.
Decision: Accept-Main