Enhancing Cross-Domain Term Extraction with Neural
Topic-based Models
Yuan Meng

Xiaoqiu Lu∗

Deqiang Yin

Southeast University
NanJing, China
yuan_meng@seu.edu.cn

Southeast University
NanJing, China
luxq823@163.com

Southeast University
NanJing, China
gmrqiang@gmail.com

Guilin Qi†

Wei Song

Southeast University
NanJing, China
gqi@seu.edu.cn

Research Center for Intelligent
Robotics, Zhejiang Lab
HangZhou, China
weisong@zhejianglab.com

ABSTRACT
Automated terminology extraction (ATE) aims to identify domainspecific specialized terms in textual data, providing crucial support
for constructing knowledge graphs and facilitating information
retrieval[1]. With the proliferation of text in emerging domains
and the maturity of domain-specific term extraction frameworks,
there is a growing interest in improving cross-domain model performance through transfer learning. However, due to the lack ofhigh-quality labelled data in emerging domains and disparities indata distribution and domain-specific information, traditional ATE
faces challenges in achieving generalization and domain adaptation when transitioning from source domains to target domains. To
address this challenge, we introduce a Cross-Domain Neural Topic
Model (dubbed CNTM) that employs a domain training-adaptationinference paradigm. Specifically, this paradigm enables CNTM toadapt to specific features in the target domain while leveraging
source domain knowledge to enhance performance. The approach
involves training the model on data-rich source domain data, finetuning it with limited target data, and extracting domain-specificterminology from low-resource target text. Additionally, we introduce the Neural Topic Model (NTM) to enhance the model’s
capacity to identify commonalities across domains, thereby improving cross-domain generalization capabilities. Furthermore, CNTM
leverages the robust representation capabilities of pre-trained language models and the discriminative optimization of the contrastive
learning module to achieve precise term extraction, especially in a
few resource scenarios. Extensive experiments on four real-world∗ Co-first author.
† Corresponding author.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
IJCKG’24, December 8 to 9, 2023, Miraikan, Tokyo, Japan© 2023 Association for Computing Machinery.
ACM ISBN 978-1-4503-XXXX-X/18/06. . . $15.00
https://doi.org/XXXXXXX.XXXXXXX

datasets across various domains with different sizes and term description sources validate the efficacy and resilience of CNTM interms of overall performance.

CCS CONCEPTS• Computing methodologies → Information extraction.

KEYWORDS
Term Extraction, Neural Topic Model, Cross-Domain
ACM Reference Format:
Yuan Meng, Xiaoqiu Lu, Deqiang Yin, Guilin Qi, and Wei Song. 2023. Enhancing Cross-Domain Term Extraction with Neural Topic-based Models.
In Proceedings of the International Joint Conference on Knowledge Graphs(IJCKG’24). ACM, New York, NY, USA, 9 pages. https://doi.org/XXXXXXX.
XXXXXXX

1

INTRODUCTION

Term Extraction (TE) plays a crucial role in deepening our understanding of specialized domain knowledge, providing invaluable input for various Natural Language Processing (NLP) applications[2–5], such as information retrieval, thesaurus construction, question
answering, and machine translation. Faced with the frequent emergence of new terms in rapidly evolving domains, traditional term
extraction models typically rely on domain-specific corpora andrules, making it challenging for them to generalize to new domains
or cross-domain applications. Meanwhile, emerging domains often
lack sufficient quantity and quality of annotated data, making it
more difficult to build high-performance TE models.
To address this challenge, a large number of previous studies havefocused on the task of transfer learning[6–8], where the knowledge
and features learned by the model in one domain are transferred
and adapted to another domain. This shows that even in the caseof low resource scenarios, existing domain knowledge can be effectively leveraged to enhance term extraction performance in new domains. For example, Zhang[7] explores the potential of fine-tuning
pre-trained BERT models for Automatic Term Extraction acrossdiverse domains and languages. Hanh[9] conducts a comprehensive study on Transformers-based pre-trained language models for
multi-language, cross-domain automatic term extraction, demonstrating the superiority of monolingual models and the benefits of

IJCKG’24, December 8 to 9, 2023, Miraikan, Tokyo, Japan

ensemble strategies. These approaches leverage the generalization
capabilities of PLM across multiple languages and domains, thereby
enhancing term extraction performance in emerging domains.
Traditional PLM are typically trained on large-scale generic corpora, excelling at capturing general semantic information. However, this can lead to a decrease in performance when dealing withdomain-specific terms, especially in emerging domains with limited annotated data, as they lack domain-specific knowledge andstruggle to identify unique information across different domains.
Terms from different domains may possess distinct meanings andcontexts, making it challenging for pre-trained models to accurately
capture and classify these terms. For example, "heart attack" has aclinical meaning related to cardiovascular health in medicine, whilein the automotive field, it refers to an engine suddenly stopping.
Traditional pre-trained language models may struggle to distinguish between these meanings due to their lack of domain-specificknowledge and context, leading to potential misinterpretations.
To address these challenges, we propose a comprehensive approach, CNTM for terminology extraction, which comprises threemain stages: domain training, domain adaptation, and domain inference. During the domain training phase, we leverage a richlylabeled source dataset to provide a pre-trained architecture for text
in the target domain. This architecture includes an input module,
an encoding module, a topic-enhanced encoding module, and adecoding module. These modules collaborate to capture the intricate relationships between terms and their context, ensuring acomprehensive representation of terminology within the domain.
The whole process provides the basis for the subsequent stages ofdomain adaptation and domain reasoning.
In the domain adaptation stage, we employ limited annotateddata from the target domain to fine-tune CNTM, thereby enhancing its adaptation to the distinctive characteristics of the target
domain. This approach helps mitigate the challenge of transferring
information between different domains, as the model can undergo
parameter adjustments to align more effectively with the data distribution of the target domain. The adaptation also involves the useof an early stopping strategy to control gradient propagation andprevent overfitting. Additionally, a topic enhancement process is
employed to capture semantic information from the target domain’s
documents. The domain inference phase utilizes the neural networkmodel trained in the domain training and adaptation phases to perform terminology inference on unlabeled data in the target domain.
This involves calculating the distance between domain-specific descriptors and individual words in input documents to assess domain
relevance, ultimately achieving cross-domain term extraction tasks.
In summary, CNTM provides a comprehensive solution for crossdomain terminology extraction, offering valuable support in addressing the challenge of weak information transferability. By leveraging pre-trained models, contrastive learning, and a neural topic
module, these integrated techniques significantly enhance the accuracy of term extraction, particularly in scenarios characterized byemerging domains and limited annotated data. A series of comprehensive experiments conducted on four real-world datasets spanning various domains provides strong evidence for the robustness
and effectiveness of CNTM.

Meng et al.

2 RELATED WORK
2.1 Neural topic model
Recently, the Neural Topic Model has emerged as a prominent
research area, attracting significant attention in the fields of text
mining and natural language processing. While traditional topic
models like Latent Dirichlet Allocation (LDA) [10] have made commendable progress in text topic modeling, they often face challenges
in capturing intricate semantics and implicit information embedded
within textual content. The Neural Topic Model (NTM) harnesses
the integration of neural networks into topic modeling, enabling
more effective learning of semantic representations and underlying
topics from extensive textual datasets. NTM’s potential to unveil
intricate topic structures and decode latent semantic structures has
attracted researchers exploring its applicability across diverse domains [7, 11]. By combining neural networks with topic modeling,
NTM provides a pathway to extract deeper insights from textual
data, contributing to advancements in various natural language
understanding tasks. Zhao [12] proposed a method for learning
document topic distribution representations by directly minimizing
the optimal distance between document-to-document word distributions. Wang [13] introduced the Layer-Assisted Neural Topic
Model (LANTM), which enhances the encoding capability of topic
representations by connecting text content with auxiliary networks.
Yang [14] presented TopNet, a method that utilizes recent advancesin neural topic modeling to generate high-quality backbone words,
addressing the limitations of short inputs. Gupta [15] proposed aneural topic modeling framework that utilizes a multi-view embedding space, incorporating pre-trained topic embeddings andword embeddings, which not only better handles polysemy but
also enhances the quality of obtained topics. While continuous
advancements in neural topic modeling have been made, existing
work has not fully harnessed the rich information conveyed by
input documents. We introduce a topic-enhancement module toamplify the abundant thematic semantic information corresponding to the original input documents. Simultaneously, we leveragea pre-trained language model based on contrastive learning andprompt learning to acquire comprehensive label information forterm labels, thereby effectively enhancing the domain adaptation
capability of term extraction tasks.

2.2

Automatic term extraction

In recent years, the field of term extraction algorithms has undergone a significant transformation, transitioning from manual extraction to automated methods. Manual extraction heavily relies on
domain experts and their specialized knowledge, which can be laborintensive and inefficient. In the realm of automated term extraction,
the initial focus was primarily on the intrinsic characteristics ofterms and shallow linguistic analysis based on their frequency intarget corpora.Methods in this early stage of development can bebroadly categorized into three main approaches: linguistics-based,statistics-based, and hybrid approaches that combine linguistic andstatistical elements.Linguistics-based approaches in term extraction
are primarily centered around leveraging manually constructed linguistic rules to facilitate the automatic extraction of terms. For
instance, researchers like Bourigault[16] employ part-of-speechtagging to annotate documents within a given corpus. Based on a

Enhancing Cross-Domain Term Extraction with Neural Topic-based Models

IJCKG’24, December 8 to 9, 2023, Miraikan, Tokyo, Japan

Pre-trained
Topic vectors

Many-shot dataset

Source domain

Domain-training

Few-shot dataset
Domain-adaptation

Domain-inference

Target domain

Target domain
sampleFigure 1: The overall framework of CNTM, which consists of three main components: domain training, domain adaptation,and domain inference. Domain training uses labeled data from the source domain, while domain adaptation and inference
utilize limited labeled data from the target domain for fine-tuning and term inference.
pre-existing set of standard terms, they utilize finite-state annotate(FSM) to automatically deduce a collection of rules implicit within
the documents. Frantzi[17]were among the pioneers to amalgamate
linguistic and statistical methods in term extraction. They introduced the C-value method, which initially selects candidate terms
based on certain rules and subsequently employs statistical criteria
to filter these candidate terms.
With the continuous advancement of machine learning and deeplearning, an increasing number of cutting-edge technologies havebeen applied to term extraction tasks. Automatic terminology extraction is mostly based on methods including statistical analysis,semantic relationships, machine learning, and rule formulation.
For expamele, vivaldi[18] introduced an external knowledge base,Wikipedia, into the terminology extraction task to enrich semantic
information. Yang[19] introduced an adaptive unsupervised clustering algorithm that enhances the algorithm’s robustness and stability
by utilizing noisy data to construct pseudo-terms for training andincorporating fault tolerance mechanisms. CoAKT[20] uses deeplearning to reduce the dependence of traditional artificial featureengineering methods. AutoPhrase[21] employs distant supervision
techniques and phrase segmentation guided by part-of-speech tagging, effectively avoiding the need for additional manual labeling
efforts and enhancing the effectiveness of terminology extraction.
Hazem[22] proposed a BERT-based approach for terminology extraction, achieving favorable experimental results on multilingual
datasets. Tran[9] and Hazem[23] employed pre-trained models toautomatically extract terms across domains from low-resource data.
They achieved cross-lingual and cross-domain transfer learning by
pre-training on the source language and fine-tuning on the target
language.
While pre-trained language models trained on extensive corpora
have demonstrated remarkable accuracy and recall in term extraction, they encounter challenges in cross-domain term extraction
tasks, particularly in nascent domains with limited labeled data.
Prominent challenges involve the limited capacity to effectivelytransfer information across domains and the intricacy of identifying distinctive elements amid diverse domains. To address theseissues, the subsequent term extraction framework introduces the
topic information derived from the neural topic model into thepre-trained language model, thereby increasing the information
extraction ability of the model.

3

PROBLEM DEFINITION

We adhere to the established named entity recognition BIO annotation format (Lample et al., 2016), where ’B’ indicates the beginningof a term, “I" represents an intermediate term, and “O" signifies nonterm entities. Each token 𝑥𝑖 in a specific input text 𝑋 = {𝑥 1, ..., 𝑥𝑡 } isassigned a label 𝑦𝑖 ∈ 𝐶, where 𝐶 denotes a predefined set of domainspecific term labels. Our datasets consist of a source domain training
set with abundant labeled data (𝐻 ) and a low-resource target domain dataset (𝑇 ), which is a union of sparsely labeled samples (𝑆)and an unlabeled target domain test set (𝐿).

4

METHODThe overall framework of the neural topic model-based terminology extraction method is depicted in Figure 1. Firstly, the domain
training phase involves training the neural network on a substantial
amount of labeled domain-specific data, thereby enabling the model
to learn a representation that adapts to the distinctive features ofthe domain’s data. Secondly, the purpose of domain adaptation is
to fine-tune model parameters using a limited amount of samples,
aiming to enhance the model’s adaptation to specific domain datacharacteristics. Finally, domain inference is performed on domainspecific datasets with limited annotations, applying the learned
model to extract term information from unlabeled data. Throughout
these three phases, we leverage pre-trained topic vectors to enhancesemantic information within the datasets, improving cross-domain
term recognition. Then we will provide a detailed elaboration ofthese three stages.

4.1

Domain training phase

During the domain training phase, the model is trained on a source
dataset with rich labels, thus providing a pre-trained architecture
for texts in the target domain. The specific model architecture, as
illustrated in Figure 2, mainly comprises input Module Based on
Prompt Learning, encoding Module based on contrastive learning,a topic-enhanced encoding module, and a decoding module basedon BiLSTM and CRF.
4.1.1 Knowlegable context construction. The prompt-based input
module aims to fully exploit the rich semantic information inherent
in texts to represent domain-specific term labels. For instance, givena specfic input sentence “[BOS] A Taxonomy of Cyber Attacks on对应文档所蕴含的丰富语义信息，从而弥补源数据集和目标数据集术语标签语义信息匮
乏和目标数据集标注数据少的问题，进而提升模型跨领域术语识别的性能表现。

领域训练阶段

在领域训练阶段，需要在一个富标签的源数据集上进行训练，从而为跨领域的术语
抽取模型提供预训练模型。领域训练阶段的模型架构如图所示，主要包含了基于提IJCKG’24, December 8 to示学习的输入改造模块，基于对比学习的语言模型编码模块，基于主题增强的编码模块
9, 2023, Miraikan, Tokyo, Japan

Meng et al.

以及基于 %L/670 和 &5) 的解码模块。

图  领域训练阶段的模型结构

Figure 2: Model Structure in Domain training phase

基于提示学习的输入改造模块

3G Networks [EOS]". We基于提示学习（SURPSW）的输入改造模块旨在充分利用自然语言中包含的丰富语义
can utilize predefined prompt template
irrelevant domain-specific term descriptions. This strategy is de“Technology none", “Technology
none" indicates a technology 举例来说，
domainsigned to effectively capture
the$relationship
信息来代表特定领域的术语标签。
对于原始的输入语句“>%26@
7D[RQRP\between words andterm, and “none" signifies
non-terms.
Alternatively,"none" >(26@”can also ，可以在其后接入这个语句对应的领域提示信
domain labels, ensuring that the model can precisely differentiate
RI &\EHU$WWDFNVRQ * 1HWZRUNV息，如“7HFKQRORJ\
QRQH”，其中“7HFKQRORJ\”代表相应语句对应科技术语领域的术
be interpreted as an
O-class word in the original
BIO
annotation,
terms across various domains.
语标签表述，而“QRQH”则代表对应语句中不是科技领域术语的单词，也可以认为是原
representing non-term
entities. This approach associates input text
Specifically, each positive pair is defined as (𝑥𝑝 , 𝑣 𝑝 ), where 𝑣 𝑝
始 %,2 标注对应的 2 类单词。由此可见，为了能够充分的利用提示信息，需要首先针
with domain informationand helps label non-term words using
denotes the corresponding domain term description. On the other对特定的领域数据集构建任务对应的提示，定义一个用于将术语标签和领域描述相映射
prompt templates. To optimally leverage the prompt information,
hand, each negative pair is formed by combining 𝑥𝑝 with unrelated
the construction of task-specific prompts tailored to specific domain  domain term description from the prompt information. Thus, thedatasets is necessary.
contrastive loss for word 𝑥𝑝 is given by Equation 2.
Hence, we denote the set of prompts as 𝑀, which associates term  labels with domain descriptions. Each word 𝑣𝑖 used for describing
′ /𝜏
exp
−𝑑
h
,
h
𝑝
𝑝

domain information can be obtained through the mapping 𝑀 and  ℓ 𝑥𝑝 = − log Í(2)2′
the corresponding domain label 𝑦𝑖 ∈ 𝐶, where 𝐶 represents the
𝑞=1 exp −𝑑 h𝑝 , h𝑞 /𝜏
predefined set of term domain labels. This study employs a simple
Where 𝜏 represents the temperature hyperparameter, we employ
yet effective approach to customize such a mapping set 𝑀. For
the Euclidean distance to measure similarity of vectors, as shown
instance, for a specific domain term label like "Tech," intuitively
in Equation 3.
using "Technology" as the domain description prompt is chosen,i.e., 𝑀 (Tech) = T echnology. The design of prompts is usuallycustomized for a specific term or domain. This type of prompts inherently contains general semantic information and context related
to the corresponding term domain and helps mitigate the bias oflimited labeled data.
For the term extraction task, the term domain labels are initially mapped to domain description information. Subsequently,
the generated prompt information is concatenated with the input document 𝑋 , resulting in an extended input sequence 𝑋 ′ ={𝑥 1, 𝑥 2, ..., 𝑥𝑡 , 𝑣 1, 𝑣 2 }. Here, 𝑡 represents the length of the original
input document, 𝑣 1 signifies the domain-specific label, and 𝑣 2 represents “none”, denoting non-domain terms. The input text 𝑋 ′ is fed
through a pre-trained language model 𝑔(.) to obtain corresponding
latent vector representations. The final hidden layer outputs serveas the representations of each word, as shown in Equation 1:
𝐻 = [ℎ 1, . . . , ℎ𝑡 , ℎ ′1, ℎ ′2 ] = 𝑔([𝑥 1, . . . , 𝑥𝑡 , 𝑣 1, 𝑣 2 ])(1)

4.1.2 Domain Training through Contrastive Learning. In order toenhance the model’s cross-domain learning capabilities and improve its adaptation to diverse semantic information across various
domains, we introduce the optimization concept of contrastive
learning. The goal of this module is to minimize the distance between the input text 𝑋 and its corresponding domain-specific term
description, while simultaneously maximizing the distance between2𝑑 h𝑝 , h𝑞′ = h𝑝 − h𝑞′2(3)

The loss function based on contrastive learning can be defined asin Equation 4.
Lcon =1 ∑︁
ℓ (𝑥𝑖 )
|X|(4)𝑥𝑖 ∈ X

Where Xrepresents the collection of documents.
4.1.3 Topic Enhancement Module. The key distinction between TE
tasks and NER lies in the fact that the latter can identify multiple categories of fine-grained instance, providing richer and more diverse
label semantics. Conversely, the descriptions in TE task is generally
consistent and unique. Compared to the finer-grained label information in named entity recognition, term extraction tasks typicallyinvolve coarser-grained domain labels. To harness the semantic
richness in domain-specific dataset documents and mitigate label
scarcity in term extraction tasks, we introduce a topic-enhanced
encoding module to enhance semantic space encoding.
Specifically, we employ a neural topic model based on mutual
information[24] to obtain high-quality latent representations foreach input document. These topic representations effectively capture the semantic information of the original input documents andcan be utilized for text reconstruction. For a given input document

Enhancing Cross-Domain Term Extraction with Neural Topic-based Models

IJCKG’24, December 8 to 9, 2023, Miraikan, Tokyo, Japan

𝑋 = {𝑥 1, ..., 𝑥𝑡 }, its latent vector representation after being processed by a pre-trained language model is denoted as 𝐸 = {𝑒 1, ..., 𝑒𝑡 },and the corresponding topic vector representation is denoted as
𝑍 = {𝑧 1, ..𝑧𝑖 }, where 𝑖 represents the predefined number of topics.
P =𝑍 ⊕𝐸(5)

Concatenating the elements from E and Z yields P, which provides
the decoding module with highly representative and reconstructive
feature vectors.
4.1.4 Decoding Module and Training Objectives. The decoding module primarily decodes the comprehensive encoding vector 𝑃 to obtain the predicted labels. Subsequently, the predicted label is derived
by calculating the cross-entropy loss between the input document
and the output document. Specifically, we employ Bi-LSTM and
CRF as the decoding module, with 𝑃 serving as input to the decoder to produce an output vector 𝑄. The dimension of 𝑄 is 𝑡 × 𝑚,
where 𝑡 signifies the number of words in the input document, and𝑚 represents the count of labels. In this context, 𝑄𝑖 𝑗 represents theprobability associated with the 𝑗th label for the 𝑖th word in the
input document. For a predicted sequence 𝑦 = {𝑦1, 𝑦2, ..., 𝑦𝑡 }, its
probability is formulated as depicted in Equation 6:
𝑓 (𝑋, 𝑦) =𝑡∑︁𝐴 𝑦𝑖 𝑦𝑖+1 +

𝑖=0

𝑡
∑︁

𝑄𝑖,𝑦𝑖

(6)

𝑖=1

The matrix 𝐴 represents the transition matrix, with 𝐴𝑖 𝑗 denoting
the probability of transitioning from label 𝑖 to label 𝑗. Here, 𝑦0 and𝑦𝑡 correspond to the starting and ending tags for the predictedsequence, respectively. The probability of generating the label sequence 𝑦 based on the original input sentence 𝑋 is mathematically
represented by Equation 7:
e 𝑓 (𝑋 ,𝑦)(7)˜𝑦˜ ∈𝑌𝑋 𝑓 (𝑋, 𝑦)
Where 𝑦˜ represents the true label. We employ the cross-entropy
loss function to maximize the likelihood probability of the accurate
label sequence, as illustrated by Equation 8:
𝑝 (𝑦 | 𝑋 ) = Í© ∑︁ 𝑓 (𝑋 ,𝑦)¯ ª
Lentropy = − log(𝑝 (𝑦 | 𝑋 )) = 𝑓 (𝑋, 𝑦) − log ­e® (8)«𝑦˜ ∈𝑌𝑋¬
Where 𝑌𝑋 represents all possible label sequences corresponding toan input sequence 𝑋 . Finally, the objective function can be viewed
as a fusion of the contrastive learning loss function and the crossentropy loss function, as depicted in Equation 9:
L = 𝜔 ∗ Lcon + (1 − 𝜔) ∗ Lentropy(9)

It can be observed that the domain traing phase encompasses aninput module based on prompt learning, Domain Training throughcontrastive learning, a topic enhancement module, and decoder
modules.

4.2

Domain Adaptation

The domain adaptation phase aims to fine-tune the model trainedon the source domain using a small amount of labeled data fromthe target domain, enabling cross-domain term inference tasks. The
model structure of the domain adaptation phase is similar to thatof the domain adaptation phase, with most modules resembling
those from the domain training phase. Furthermore, to enhance
the model’s domain adaptation capability for different label spaces,
we have improved the topic-enhanced module during the domain
adaptation phase. This enhancement aims to fully leverage therich semantic information present in the unlabeled target domain
dataset.
In the term extraction task, the primary distinction between
the target domain and the source domain lies in the variation ofterm labels. Given that each dataset corresponds to a specific document within the domain, the domain-specific descriptions anddomain labels used in hint learning vary. To enhance the model’sdomain adaptation capability, we fine-tune a pre-trained neuralnetwork model that has been trained in the source domain usinga small amount of annotated data from the target domain. In thisprocess, we use an early stopping strategy[25] to control the degreeof gradient propagation, which works by monitoring the model’s
performance on the validation data set during training and stopping
the training process when performance starts to decline. This indicates that the model has started to overfit the training data, thereby
preventing overfitting and improving the model’s generalization.
4.2.1 Topic Enhancement based on ElasticSearch. When dealing
with text data associated with a specific domain, the textual information related to the target domain usually contains diverse
semantic details that can effectively capture the essence of domainspecific terms. Additionally, these meticulously structured andconstrained vocabularies often demonstrate a notable extent ofword overlap. Motivated by the insights from reference[26], we
employ an approach in which the provided text functions as aquery, aiming to retrieve the most analogous samples from thetarget domain’s database. In this study, we leveraged the existing Elasticsearch[27]retrieval engine and employed the efficient
BM25[28] retrieval method. We constructed an Elasticsearch index using the built-in standard analyzer, specifically tailored forlarge-scale unlabeled domain corpora. This index enables real-time
retrieval of the top K samples based on the BM25 scores computedfrom the input text. Subsequently, the top K samples with the closest
similarity are extracted for further topic enhancement. Specifically,
we perform a weighted average of the pre-trained topic vectors
corresponding to these documents,resulting in an average topic vector that corresponds to the original input document. This processserves to enhance the semantic information encapsulated within
the original input document.

4.3

Domain Inference Phase

The purpose of domain inference is to utilize the neural networkmodel trained during the domain training and domain adaptation
phases. This aims to perform terminology inference on a large
volume of unlabeled data in the target domain, determining whether
the input document contains specific terms from the domain. The
model structure is illustrated in Figure 3.
We calculate the distance between domain-specific descriptors
and each individual word to assess the domain relevance of words.
For the input documents in the test set 𝐿test , they are fed into thepre-trained language model to obtain their corresponding latentvector representations, as shown in Equation 10:领域适应。在实现过程中，本文采用 >@ 提到的提前停止标准来实现目标领域的梯度传
播，从而对神经网络的参数进行更新，避免过度适应的问题。

领域推断阶段领域推断的目的在于利用领域训练和领域适应阶段训练好的神经网络模型，针对目IJCKG’24, December 8 to 9,标领域上的大量无标注数据进行术语推断，2023, Miraikan, Tokyo, Japan从而判断输入文档中是否含有特定领域内的

Meng et al.

术语，模型结构如图所示。在领域推断阶段，提示学习中的领域描述信息将被当作

图  领域推断阶段的模型结构

Figure 3: Model Structure in the Domain Inference Phase

是度量指标来计算与每个单词之间的距离。具体而言，对于测试集中的输入文档 Ltest ，h1, . . . , h𝑡 , h1′ , h2′

𝐿test =
(10)
For each word 𝑥𝑖 in the input document, we find the closest anchor
embedding and assign its corresponding domain label information
𝑐 𝑗 to the respective word token, as shown in Equation 11.
𝑦𝑖test = argmin h𝑖 − h′𝑗
𝑐𝑗

2
2

(11)

Where ℎ𝑖 , ℎ ′𝑗 represents the latent representation of word,domainlabel, respecially. In the process of term inference, the Viterbi algorithm is still used to predict the output sequence with the largest
score, so as to obtain the BIO prediction sequence representation
corresponding to the input document, as shown in Equation 12:˜𝑦 ∗ = arg𝑦˜ ∈𝑌𝑋 max 𝑓 (𝑋, 𝑦)(12)
The final term inference results can be obtained through the
integration of distance-based contrastive learning predictions andprobability predictions based on Viterbi decoding, thereby achieving cross-domain term extraction tasks.

5 EXPERIMENT
5.1 Experiment Set
Dataset. To provide a comprehensive evaluation of the performance of the TNE for cross-domain term extraction tasks, we conducts experiments on four real-world datasets. Table 1 provides
statistics for each dataset, including the number of documents,
average document length, and the count of standard terms.
ACLVer2: This dataset comprises abstracts from 300 articles
in the field of computer science, sourced from the Association for
Computational Linguistics (ACL) index publications.
GENIA: The GENIA dataset is designed for semantic annotationin biomedical text mining, consisting of 2,000 abstracts coveringvarious domains in biomedicine, such as human biology, blood cells,
and transcription factors.
TTCM: Comprising 37 articles focused on mobile technology,
this dataset was constructed by web scraping and includes a manually filtered "standard terminology list," making it a valuable resource for research and analysis in the field.
TTCW: This collection of 103 articles, centered around windenergy, offers invaluable data concerning wind turbine performanceand environmental conditions.

Table 1: Terminology Extraction Statistics for the Four
Datasets
Dataset#TermsAverage Document Length

ACL
GENIA
TTCM
TTCW

3059
33396
255
288

736
1498
55652
49737

5.2

#Documents
300
2000
37
103

Baseline Models

To evaluate the practical effectiveness of our framework, we compare our model with the most prominent methods:• ComboBasic[29] is a terminology extraction algorithm,
places its primary focus on evaluating the frequency of candidate terms in documents and analyzing their contextual
relationships. Additionally, it utilizes domain-specific vocabularies and incorporates multi-positional features to effectively filter out irrelevant terms.
• TermExtractor[30] is a graph-based terminology extraction
algorithm that builds a co-occurrence matrix and a corresponding weighted graph through the fusion of textual data.
The edges within the weighted graph signify co-occurrence
associations between candidate terms, enabling the extraction of domain-specific terms through graph analysis.
• CoAKT[20] is an automatic key term extraction method
based on deep learning that operates without explicit features. Its goal is to decrease reliance on traditional feature engineering methods through deep learning techniques, leveraging a large-scale corpus for training. By employing unsupervised learning, it enhances the representational capacity of text features, ultimately reducing the time and effort
needed for manual feature construction.
• TTE[31] is a multi-language terminology extraction model
based on Transformers. It employs multiple Transformer layers to extract latent text features and utilizes self-attention
mechanisms to capture relationships among words. Furthermore, the model incorporates an innovative filtering mechanism for efficient candidate term selection.
• BERT-biLSTM-CRF[23] is a cross-domain terminology extraction model. Based on BERT, it captures shared contextualEnhancing Cross-Domain Term Extraction with Neural Topic-based Models

IJCKG’24, December 8 to 9, 2023, Miraikan, Tokyo, Japan

Table 2: Performance of Domain Terminology Extraction
MethodComboBasic
TermExtractor
CoAKT
TTE
BERT-biLSTM-CRF
TNE

ACL
Pre Rec
0.15 0.11
0.23 0.15
0.38 0.19
0.42 0.40
0.51 0.43
0.53 0.50

information for terms across domains and languages. Leveraging BERT, this model conducts cross-language and crossdomain transfer learning, resulting in enhanced extractionof both single-word and multi-word terms. This approachsignificantly improves the efficiency of term extraction.

5.3

Evaluation Metrics

Term extraction models typically extract candidate terms fromdomain-specific corpora as output. This study employs the complete
term list obtained from the corpus as the ground truth. Evaluationof model-generated candidate terms is performed using Precision,
Recall, and F1-score.
• Precision(Pre) is calculated by determining the proportionof correctly extracted terms among all candidate terms predicted by the model, as Equation 13.
True Positives
Precision =(13)
True Positives + False Positives• Recall(Rec) is calculated by determining the proportion ofcorrectly extracted terms among all standard terms within
the domain corpus, as Equation 14.
True Positives
Recall =(14)
True Positives + False Negatives• F1-score(F1) is obtained by computing the harmonic meanof precision and recall, providing a comprehensive assessment of the term extraction model’s performance, as Equation 15.
2 · Precision · Recall(15)
F1-score =Precision + Recall

5.4

Implementation Details

Hyperparameters for ComboBasic, TermExtractor, CoAKT, TTE,
and BERT-biLSTM-CRF were configured based on the optimal settings outlined in their respective works. The ratios of domain training, domain adaptation, and domain inference in the same domain
dataset are set to 8:1:1, respectively. During model training, a learning rate of 10 −4 was employed in conjunction with the Adam optimizer across all datasets. The 𝜔 parameter in the overarching loss
function described in Equation 4.8 was empirically set to 𝜔 = 0.4.
All experiments consisted of 100 training epochs and employeda batch size of 128. In the content of cross-domain term extraction, this paper employed 1-shot and 5-shot settings during thedomain adaptation phase to assess the performance of neural topic
model-based term extraction methods and other baseline models
in cross-domain term extraction tasks.

GENIA
Pre Rec
0.52 0.09
0.41 0.10
0.57 0.15
0.60 0.45
0.63 0.41
0.65 0.52

TTCM
Pre Rec
0.27 0.34
0.23 0.17
0.43 0.38
0.53 0.51
0.52 0.42
0.58 0.45

TTCW
Pre Rec
0.36 0.64
0.20 0.32
0.47 0.58
0.61 0.52
0.63 0.54
0.67 0.56

6 EXPERIMENT ANALYSIS
6.1 Experimental results in a domain
The overall performance of compared methods and the proposed
framework is indicated in Table 2.
Firstly, when we evaluate the traditional term extraction frameworks, including ComboBasic, TermExtractor, and CoAKT, we observe that their precision (Pre) and recall (Rec) values are relatively
modest across all datasets. Simultaneously, it is evident that the two
pre-trained language models, TTE and BERT-biLSTM-CRF, outperform the conventional term extraction frameworks, ComboBasic,
TermExtractor, and CoAKt, underscoring the substantial capabilities of pre-trained language models in terms of domain-specificsemantic information and their ability to harness domain knowledge effectively. We observe that the TNE yields optimal resultsin terms of precision and recall on the ACL, GENIA, TTCM, and
TTCW datasets. This suggests that the fusion of the neural topic
model and pre-trained language models facilitates the acquisitionof domain-specific thematic information, effectively enhancing therepresentation capabilities of the topic-enhancement module within
the term extraction model.

6.2

Cross-domain results on Low Resource Data

To validate the domain training, domain adaptation, and domaininference paradigms mentioned in CNTM, we focus on assessing
the performance of the model specifically in cross-domain terminology extraction tasks. Specifically, in this study, we employ the
ACL dataset as the source domain dataset. We then evaluate theexperimental results when using GENIA, TTCM, and TTCW astarget domain datasets. During the domain adaptation phase, we
conduct experiments with small-sample adaptations set at 1-shotand 5-shot scenarios. The cross-domain term extraction F1 scoresare presented in Table 3
We can find our model consistently achieves the highest F1scores. This indicates that our model exhibits superior domaintransferability compared to other baseline models, ultimately enhancing the model’s scalability. In contrast to BERT-biLSTM-CRF,
TNE performs better in the few data scenario. This improvement
can be attributed to domain adaptation phase, which incorporates
an ElasticSearch-based topic enhancement module. This module
effectively leverages unsupervised topic information to identify
domain-specific thematic information, leading to enhanced term
extraction performance.

IJCKG’24, December 8 to 9, 2023, Miraikan, Tokyo, Japan

Meng et al.

Table 3: Performance of cross-domain terminology extraction
MethodComboBasic
TermExtractor
CoAKT
TTE
BERT-biLSTM-CRF
Ours

6.3

GENIA
0.08
0.10
0.15
0.31
0.33
0.43

1 shot
TTCW
0.04
0.09
0.12
0.25
0.27
0.41

TTCM
0.03
0.08
0.16
0.24
0.26
0.39

GENIA
0.09
0.12
0.17
0.33
0.35
0.45

5 shot
TTCW
0.04
0.10
0.13
0.25
0.28
0.42

TTCM
0.02
0.11
0.16
0.26
0.29
0.43

Ablation experiment

In this section, we conducted ablation experiments to analyze theimpact of major components of TNE framework: 1) CNTM-T:wehave removed the Topic Enhancement Module from the TNE framework.; (2)we have excluded the Contrastive Learning from TNE
to evaluate whether it imposes limitations on the extraction of
domain-specific terms;
Table 4: Performance with Different Adaptation Shots
Method

GENIA

TTCW

TTCM

CNTM-T
CNTM-C
CNTM

0.25
0.31
0.43

0.21
0.28
0.41

0.19
0.29
0.39

As Table 4 shows, when we remove the Topic Enhancement
Module in "CNTM-T," or Contrastive learning Module in "TNE-C",
there are noticeable drop in performance. Specially, TNE integrating
both modules, attained the highest performance with F1-scores of
0.43, 0.41, and 0.39. These results emphasize the importance of both
modules and their synergy for effective terminology extraction.

6.4

Template Prompt Comparison Experiment

To explore the influence of different templates on the cross-domain
term extraction task, we attempted to create distinct template
prompts for various datasets. For the four datasets, AC, GENIA,
TTCM, and TTCW, the corresponding template are shown in Table 5 as follows: Initial templates included a single domain keyword,while expanded templates enriched domain information from theinitial templates. The resulting term extraction F1 scores are illustrated in Figure 4
Figure 4 clearly shows that the extended template outperforms
the initial template in cross-domain experiments. This improvement
Table 5: Term Extraction Statistics for the Four Datasets
DatasetInitial template

ACL
GENIA
TTCM
TTCW

Linguistics
Biomedical
Technology
EnergyExtended template
Computational Linguistics
Biomedical Science
Mobile Technology
Wind Energy

Figure 4: Experimental performance based on different templates
can be attributed to the enhanced semantic content provided by
the extended template for term labels.

7

CONCLUSION

In this study, we proposed a cross-domain neural topic model(dubbed
CNTM) to address low-resource term extraction challenges. CNTM
showed substantial advancements in recognizing domain-specificterms, adapting to diverse domains, and has the potential to enhance
applications like knowledge graph construction and information
retrieval. The incorporation of topic-based enhancements and transfer learning techniques enhances the capability to capture semantic
nuances, making it more adaptable to different domains, especiallyresource-poor emerging domains. This paves the way for future
research to explore the framework’s further enhancements andapplications.

ACKNOWLEDGEMENT
This work is partially supported by National Nature Science Foundation of China under No. U21A20488. We thank the Big Data
Computing Center of Southeast University for providing the facility support on the numerical calculations in this paper.

REFERENCES[1] T. T. H. Hanh, M. Martinc, J. Caporusso, A. Doucet, S. Pollak, The recent advancesin automatic term extraction: A survey, CoRR abs/2301.06767 (2023). arXiv:
2301.06767, doi:10.48550/arXiv.2301.06767.
URL https://doi.org/10.48550/arXiv.2301.06767[2] M. T. Pazienza, M. Pennacchiotti, F. M. Zanzotto, Terminology extraction: an analysis of linguistic and statistical approaches, in: Knowledge mining: Proceedingsof the NEMIS 2004 final conference, Springer, 2005, pp. 255–279.
[3] A. Peñas, F. Verdejo, J. Gonzalo, et al., Corpus-based terminology extraction
applied to information access, in: Proceedings of corpus linguistics, Vol. 2001,
2001, p. 458.

Enhancing Cross-Domain Term Extraction with Neural Topic-based Models[4] H. Du, Z. Le, H. Wang, Y. Chen, J. Yu, Cokg-qa: Multi-hop question answeringover covid-19 knowledge graphs, Data Intelligence 4 (3) (2022) 471–492.
[5] H. Chen, N. Hu, G. Qi, H. Wang, Z. Bi, J. Li, F. Yang, Openkg chain: A blockchain
infrastructure for open knowledge graphs, Data Intelligence 3 (2) (2021) 205–227.
[6] F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, Q. He, A comprehensive
survey on transfer learning, Proceedings of the IEEE 109 (1) (2020) 43–76.
[7] M. Peng, Q. Zhang, Y. Jiang, X. Huang, Cross-domain sentiment classification withtarget domain specific information, in: I. Gurevych, Y. Miyao (Eds.), Proceedingsof the 56th Annual Meeting of the Association for Computational Linguistics, ACL
2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, Association
for Computational Linguistics, 2018, pp. 2505–2513.
[8] Y. A. Winatmoko, A. A. Septiandri, A. P. Sutiono, Aspect and opinion term
extraction for hotel reviews using transfer learning and auxiliary labels, arXiv
preprint arXiv:1909.11879 (2019).
[9] H. T. H. Tran, M. Martinc, A. Pelicon, A. Doucet, S. Pollak, Ensembling transformers for cross-domain automatic term extraction, in: International Conference on
Asian Digital Libraries, 2022, pp. 90–100.
[10] D. M. Blei, A. Y. Ng, M. I. Jordan, Latent dirichlet allocation, Journal of machine
Learning research 3 (Jan) (2003) 993–1022.
[11] Y. Miao, L. Yu, P. Blunsom, Neural variational inference for text processing,in: M. Balcan, K. Q. Weinberger (Eds.), Proceedings of the 33nd International
Conference on Machine Learning, ICML 2016, New York City, NY, USA, June
19-24, 2016, Vol. 48 of JMLR Workshop and Conference Proceedings, JMLR.org,
2016, pp. 1727–1736.
[12] H. Zhao, D. Phung, V. Huynh, T. Le, W. Buntine, Neural topic model via optimaltransport, arXiv preprint arXiv:2008.13537 (2020).
[13] Y. Wang, X. Li, J. Ouyang, Layer-assisted neural topic modeling over document
networks., in: IJCAI, 2021, pp. 3148–3154.
[14] Y. Yang, B. Pan, D. Cai, H. Sun, Topnet: Learning from neural topic model togenerate long stories, in: Proceedings of the 27th ACM SIGKDD Conference on
Knowledge Discovery & Data Mining, 2021, pp. 1997–2005.
[15] P. Gupta, Y. Chaudhary, H. Schütze, Multi-source neural topic modeling in multiview embedding spaces, arXiv preprint arXiv:2104.08551 (2021).
[16] K.-h. Chen, H.-H. Chen, Extracting noun phrases from large-scale texts: A hybrid
approach and its automatic evaluation, arXiv preprint cmp-lg/9405034 (1994).
[17] K. Frantzi, S. Ananiadou, H. Mima, Automatic recognition of multi-word terms:.
the c-value/nc-value method, International journal on digital libraries 3 (2000)
115–130.
[18] J. Vivaldi, L. A. Cabrera-Diego, G. Sierra, M. Pozzi, et al., Using wikipedia tovalidate the terminology found in a corpus of basic textbooks., in: LREC, Citeseer,2012, pp. 3820–3827.
[19] Y. Yang, H. Yu, Y. Meng, Y. Lu, Y. Xia, Fault-tolerant learning for term extraction,
in: Proceedings of the 24th Pacific Asia Conference on Language, Information
and Computation, 2010, pp. 321–330.
[20] K. Khosla, R. Jones, N. Bowman, Featureless deep learning methods for automated
key-term extraction (2019).
[21] J. Shang, J. Liu, M. Jiang, X. Ren, C. R. Voss, J. Han, Automated phrase mining frommassive text corpora, IEEE Transactions on Knowledge and Data Engineering
30 (10) (2018) 1825–1837.
[22] C. Lang, L. Wachowiak, B. Heinisch, D. Gromann, Transforming term extraction:
Transformer-based approaches to multilingual term extraction across domains,
in: Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021,
Online Event, August 1-6, 2021, Vol. ACL/IJCNLP 2021 of Findings of ACL, Association for Computational Linguistics, 2021, pp. 3607–3620.
[23] A. Hazem, M. Bouhandi, F. Boudin, B. Daille, Cross-lingual and cross-domaintransfer learning for automatic term extraction from low resource data, in: Proceedings of the Thirteenth Language Resources and Evaluation Conference, LREC
, Marseille, France, 20-25 June, European Language Resources Association, 2022,
pp. 648–662.
[24] K. Xu, X. Lu, Y. fang Li, T. Wu, G. Qi, N. Ye, D. Wang, Z. Zhou, Neural topic
modeling with deep mutual information estimation, Big Data Research 30 (2022)
100344.
[25] Y. Huang, K. He, Y. Wang, X. Zhang, T. Gong, R. Mao, C. Li, Copner: Contrastive
learning with prompt guiding for few-shot named entity recognition, in: Proceedings of the 29th International conference on computational linguistics, 2022,
pp. 2515–2527.
[26] X. Zhang, Y. Jiang, X. Wang, X. Hu, Y. Sun, P. Xie, M. Zhang, Domain-specificner via retrieving correlated samples, arXiv preprint arXiv:2208.12995 (2022).
[27] B. Elasticsearch, Elasticsearch: The official distributed search & analytics engine(2020).
[28] S. E. Robertson, S. Walker, Some simple effective approximations to the 2-poissonmodel for probabilistic weighted retrieval, in: SIGIR’94, Springer, 1994, pp. 232–
241.
[29] N. A. Astrakhantsev, D. G. Fedorenko, D. Y. Turdakov, Methods for automatic
term recognition in domain-specific text collections: A survey, Programming andComputer Software 41 (2015) 336–349.
[30] F. Sclano, P. Velardi, Termextractor: a web application to learn the shared terminology of emergent web communities, in: Enterprise Interoperability II: NewIJCKG’24, December 8 to 9, 2023, Miraikan, Tokyo, Japan

Challenges and Approaches, Springer, 2007, pp. 287–290.
[31] C. Lang, L. Wachowiak, B. Heinisch, D. Gromann, Transforming term extraction:transformer-based approaches to multilingual term extraction across domains,
in: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021,
2021, pp. 3607–3620.