Analyzing Contextualized Knowledge Base Aware Representation Learning models for WSD
AAAI 2023, Mozhgan Saeidi

The Problem

The Architecture of C-KASE Embedding

3-Sense Embedding:In this last component, we build the final representation of each mention. From theprevious step, we took the representation of mention, R(m), and the representationof each one of its senses. Our unique representations combine the mention representation with sense representation, concatenating the two vector representations ofR(m) and R(si).

Natural Language Processing (NLP) includes many tasks which, most of which areengaged with representation learning (RL). Text representation learning has shown its
essential impact on the final results of NLP tasks, including Word Sense Disambiguation (WSD), Information Retrieval (IR), and Question Answering. After the development of deep neural networks, different approaches have been widely used to solve NLP
tasks. Some of these deep neural networks are convolutional neural networks (CNNs),
recurrent neural networks (RNNs), graph-based neural networks (GNNs), and attention mechanisms. Representation learning is also one of the tasks that use the powerof deep learning to alleviate feature engineering difficulties. RL models usually use
low-dimensional and dense vectors to represent the syntactic or semantic features ofthe language implicitly.

Sim(m, si) = Cosine(R(m, si), R(P D, P W )), for i = 1, . . . , k(3)Experiments
We use the English WSD test set framework, which is constructed by five standard
evaluation benchmark datasets. For WSD modeling, we employed a 1-nearest neighbor approach as previous methods in the literature to test our representations on
the WSD task. For each target word m in the test set, we computed its contextual
embedding using BERT and compared it against the embeddings of C-KASE associated with the senses of m. Hence, we took as a prediction for the target word thesense corresponding to its nearest neighbor.

On the large corpus, the pre-trained models can learn language representations andthen be used to solve downstream tasks. Between different RL approaches, the Skipgram and GloVe are such models that are very shallow for computational efficiencies.
While by emerging the deep models, including transformers, the RL architecture istransferred from shallow to deep. The pre-trained embeddings capture the semanticsof the words they represent, but they suffer from the context in their representations.
The importance of context in word representation is vital in some NLP tasks, likeWSD.

Fig. 2: Demonstration of the C-KASE representation and its three components. Component 1) Collecting all Wikipedia pages for
ambiguous words, Component 2) Using hypernymy and hyponymy relations to extract all synsets for ambiguous words from Babelnet in

Fig. 3: The accuracy performance of WSD evaluation framework on the test sets nominal instances of the unified

set E, Component 3) Concatenating (word, sense) representation for all senses in E from the second component with (Document’sdataset.

paragraph, Wikipedia’s paragraph) representation from the first component as context

Fig. 1: A depict of vector representation learning

The biomedical domain is one of these domains that needs specific pre-trained language models; this need is because of the volume of the biomedical text, which isgrowing at a good speed and needs analysis for different problems. On average, morethan 3000 new articles are published every day in peer-reviewed journals, excluding
pre-prints and technical reports such as clinical trial reports in various archives. Consequently, there is increasingly more demand for accurate biomedical text mining tools
to extract text information.
Word representations may improve the effectiveness of the disambiguation models
if they carry helpful information from the context and the knowledge base. In thiswork, first, we provide an in-depth quantitative and qualitative analysis of existing transformer-based language models to understand their capabilities and potential limitations in encoding and recovering word senses. Second, we present a novel
contextual-knowledge base aware sense representation learning method. The name ofthis new embedding approach is C-KASE stands for Contextualized-Knowledge base
Aware Sense Embedding. The novelty in our representation is the integration of theknowledge base and the context. This representation lies in a space comparable tocontextualized word vectors, thus allowing a word occurrence to be easily linked to its
meaning by applying a simple nearest-neighbor approach. Finally, we compare ourapproach with state-of-the-art embedding methods for WSD.

C-KASE Representation Learning Model
Our proposed algorithm, C-BERT, is a pre-trained language representation model. C-BERT iscreated by combining semantic and textual information from the first paragraph of each sense’s
Wikipedia page and the paragraph of the input document text, which includes the senses. CKASE is based on three components; Context Retrieval, Word Embedding, and Sense Embedding.

Fig. 4: The performance of the 1-NN WSD of each embedding on All dataset split by parts of speech

1-Context Retrieval:
For each synset s, we collect all the connected concepts to s from Wikipedia and BabelNet. We
show this set of related synsets to s by Rs, which isFig. 5: Error rate analysis of the 1-NN WSD evaluation framework with ARES, BERT, and C-KASE representations

Rs = {s′|(s, s′) ∈ E}

(1)on the All dataset, separated by type.

2-Word Embedding:In the second component, we use BERT to extract the given ambiguous word from the inputtext.
W O(p1, p2) = (Xw∈O1
p1p2 )(rw + rw

|O|X1
i=12i)−1(2)Fig. 6: Confusion-Error table for Noun type by each model. This table shows how models are confused by the type ofword at the time of disambiguation