Learning Contextualized Box Embeddings with Prototypical Networks (2024)

Abstract: This paper proposes ProtoBox, a novel method to learn contextualized box embeddings. Unlike an ordinary word embedding, which represents a word as a single vector, a box embedding represents the meaning of a word as a box in a high-dimensional space: that is suitable for representing semantic relations between words. In addition, our method aims to obtain a “contextualized” box embedding, which is an abstract representation of a word in a specific context. ProtoBox is based on Prototypical Networks, which is a robust method for classification problems, especially focusing on learning the hypernym–hyponym relation between senses. ProtoBox is evaluated on three tasks: Word Sense Disambiguation (WSD), New Sense Classification (NSC), and Hypernym Identification (HI). Experimental results show that ProtoBox outperforms baselines for the HI task and is comparable for the WSD and NSC tasks.1

heading: 1 Introduction
text: Word embedding is an abstract representation of a word, usually as a vector in a high dimensional space. Nowadays, word embeddings are widely used in models based on deep learning. Word embedding can represent the meaning not only of a word itself (Mikolov et al., 2013) but also of a word in a context. For example, BERT (Devlin et al., 2019) is often used to obtain vector representations of words in a given sentence. In this paper, we call such word embeddings “contextualized word embeddings.” In addition, box embedding (Dasgupta et al., 2020) is a kind of word embedding, which represents a word not with a point but with an area in vector space. While an ordinary word embedding is primary used to measure the similarity between two words, box embeddings can be used to capture other semantic relations between
1Our code is available at: https://github.com/iehok/ ProtoBox.
words such as that between a hypernym and a hyponym. However, past studies of box embeddings did not well consider the context of the word, that is, the box embedding was not contextualized.
Contextualized word embeddings can be regarded as “sense embeddings,” since a word may have two or more senses and convey one of those possible senses in a specific context. Word Sense Disambiguation (WSD) (Navigli, 2009) is a task that aims to identify the meaning of a word in a context. Most previous work on WSD has focused only on the senses in a predefined inventory and has ignored new (not predefined) senses. However, senses of words change day by day and new senses are constantly created (Yu and Xu, 2023). It is preferable that a WSD system can handle a new sense by classifying a word even if it is being used in a new sense, and, if possible, explaining the meaning of the new sense.
In this paper, we propose ProtoBox, a method to produce a contextualized box embedding of a word in a given context. In general, box embeddings can represent hypernym–hyponym relations between
1
words, as illustrated in Figure 1 (a). If a box of word wa subsumes the box of another word wb, wa can be regarded as a hypernym of wb. Such relations between words can be represented as taxonomy (Figure 1 (b)). Our ProtoBox can produce contextualized box embeddings. For example, a box embedding of x in the sentence “The x hopped across the grass.” can be obtained as shown in Figure 1 (a). Contextualized box embeddings enables us to judge that x has a new sense when the box embedding of x does not overlap any other box embeddings of fine-grained senses such as “cat” and “dog”. In addition, “animal” can be identified as a hypernym of x, since the box embedding of x is subsumed by that of “animal”. Identification of a hypernym can provide a rough explanation of a new sense, i.e., x is a kind of an animal. Furthermore, ProtoBox can expand the existing taxonomy by adding a new node x to the structure as shown in Figure 1 (b).
We evaluate ProtoBox with three tasks: WSD, New Sense Classification (NSC), and Hypernym Identification (HI). Three datasets of different domains, one is large and two are small, are used to thoroughly evaluate our proposed method. Experimental results show that ProtoBox is better than or comparable to the baselines for WSD and NSC, and always outperforms the baselines for HI.
The contributions of this paper are summarized as follows:
• We propose ProtoBox, a new method to learn contextualized box embeddings based on Prototypical Networks (Snell et al., 2017).
• We propose a method to construct an minibatch to learn hypernym–hyponym relations between senses in the contextualized box embeddings.
• We empirically evaluate the effectiveness of ProtoBox for three down-streaming tasks: WSD, NSC, and HI.

heading: 2 Related Work

heading: 2.1 WSD
text: Many recent WSD methods use glosses (sense definitions) and lexical relations (e.g., hypernym– hyponym relations) to improve their performance (Huang et al., 2019; Kumar et al., 2019; Bevilacqua et al., 2020; Bevilacqua and Navigli, 2020; Blevins and Zettlemoyer, 2020; Scarlini et al., 2020; Barba
et al., 2021). However, the accuracy of WSD for infrequent senses tended to be lower than that for the whole of the test data (Maru et al., 2022).
To address this problem, Chen et al. (2021) proposed MetricWSD, a method to learn contextualized embeddings using Prototypical Networks (Snell et al., 2017). Prototypical Networks is a meta learning method that works better on imbalanced data. MetricWSD achieved a state-of-the-art WSD performance without additional lexical information such as glosses or lexical relations.
Generationary (Bevilacqua et al., 2020) is another approach for WSD. First, a definition of a sense for a given word is generated by BART (Lewis et al., 2020). Then, the similarity score between the generated definition and each definition of the target word in WordNet (Miller, 1995) is calculated by Sentence-BERT (Reimers and Gurevych, 2019) and the most similar sense chosen to be the predicted sense. Generationary aims not only to improve the performance at WSD but also explain a new sense. This paper also tries to explain the meaning of a new sense using trained contextualized box embeddings. Instead of generating a definition of a new sense, a hypernym of a new sense is identified as a coarse meaning of it.

heading: 2.2 Taxonomy Expansion
text: Taxonomy Expansion is the task to infer a hypernym of a new concept (Bordea et al., 2016). It has been actively studied. Recent methods improved the performance by using graph neural networks (Shen et al., 2020) and learning the shortest path between a target concept and the root concept (Yu et al., 2020). Some methods (Aly et al., 2019; Ma et al., 2021) used Hyperbolic space (Nickel and Kiela, 2017) learn hypernym–hyponym relations. This paper also presents a method of Taxonomy Expansion, but a hypernym of a new concept is guessed by contextualized box embeddings.

heading: 2.3 Contextualized Box Embeddings
text: There have been a few studies that have applied contextualized box embeddings to some tasks. The Entity Typing task is a multi-label classification problem to predict appropriate types such as “event” and “person”, for a target in a context (Choi et al., 2018). Onoe et al. (2021) represented target entities by contextualized box embeddings, and also represented types of entity by dedicated box embeddings. The model was trained so that the contextualized
box embedding of the target entity was enclosed by the box embeddings of its type of entity.
Jiang et al. (2023) proposed a method for Taxonomy Expansion by learning the box embeddings of concepts. The box embeddings of entities in the existing taxonomy were derived from their definition sentences. The model, which converts a sentence to a contextualized box embedding was trained by hypernym–hyponym pairs in the taxonomy so that the box embedding of a hypernym enclosed that of a hyponym. Although the above studies presented methods to learn contextualized box embeddings, we adapt another approach. Specifically, our framework follows that of Prototypical Networks, which can work well for imbalanced training data. We expect that our method can learn appropriate box embeddings for infrequent senses.

heading: 3 Proposed Method
text: This section describes the details of ProtoBox, our proposed method to train contextualized box embeddings. We first explain box embeddings, as background, in subsection 3.1, then explain ProtoBox in the succeeding subsections.

heading: 3.1 Box Embeddings
text: Single vectors represent items as points, while box embeddings represent items as boxes. Box embeddings can naturally represent asymmetric relations like hypernym–hyponym relations by the overlap of two boxes. In this work, a box embedding b is constructed from two vectors c, the center of the box, and o, an offset from c. c is the center of a box and o is the offset from c. Note that the dimensions of c and o are equal. The area of the ith dimension of the box embedding is defined as the range [ci − oi, ci + oi].
Given two boxes bi and bj , the probability that bi encloses bj can be defined as
P (bj |bi) = Vol(bi ∩ bj)
Vol(bi) , (1)
where bi ∩ bj is the are of the overlap of bi and bj , and Vol(b) is the function that calculates the volume of b.
The hard definition of the probability in Equation (1) often leads a serious problem for training box embeddings. When two boxes have no overlap, P (bj |bi) is zero, causing the training to halt due to the vanishing of the gradient. Therefore, in general, a soft definition is often used. Following previous
work (Onoe et al., 2021; Jiang et al., 2023), we use Gumbel Box (Dasgupta et al., 2020), one of the box embeddings that calculates the above probability with a soft definition. Specifically, the probability that bi encloses bj is calculated with the Gumbel distribution.

heading: 3.2 MetricWSD
text: Since ProtoBox is an extension of MetricWSD (Chen et al., 2021), we first briefly introduce the latter. The left side of Figure 2 shows an overview of MetricWSD. It is a model for WSD, based on Prototypical Networks. The training data is a collection of sentences including a target word (e.g., ‘dog’) labeled with its gold sense (e.g., dog.1). It is divided into two sets: a support set and a query set. Each sentence in the support set is converted to a contextualized word embedding (or sense embedding) by a model fθ. In MetricWSD, BERT (Devlin et al., 2019) is used as fθ. The prototype vector of each sense (e.g., dog.1, dog.2) is defined as the average of the contextualized word embeddings of that sense. Next, the sentences in the query set are converted to contextualized word embeddings by the same model fθ, and then the loss is calculated by the distances between the query vector and the prototype vectors. Finally, the parameters of the model, θ, are updated so that the loss becomes minimized. At the inference, a test sentence is converted to an embedding by fθ, and then the similarities between it and the prototype sense vectors are calculated, and the most similar sense is chosen.

heading: 3.3 Learning Contextualized Box Embeddings
text: The right side in Figure 2 shows an overview of ProtoBox. In our method, MetricWSD is modified in three ways. First, instead of a single vector, a sentence is converted to a box embedding by the model. Following previous work (Onoe et al., 2021), we add a Fully Connected Layer (FCL) after BERT. For a sentence x where the zth word is the target word, its contextualized box embedding is obtained as follows:
b = fθ(x, z) = FCL(BERT(x)[z]). (2)
The input of FCL is BERT(x)[z], the contextualized word embedding of the zth word when x is entered to BERT. The output of FCL forms the box embedding b, which is equally divided into two vectors c and o by b = [c,o].
Second, the episodes are constructed differently. In Prototypical Networks, a mini-batch used to train a model is called an “episode.” On the one hand, an episode is a set of instances with different senses of the same target word in MetricWSD. On the other hand, an episode is a set of instances with different senses of multiple target words (e.g. dog.1, animal.1, and tree.1) in ProtoBox. The details of the construction of an episode will be explained in subsection 3.4.
Third, the loss is calculated between a query representation (contextualized box embedding) bq and each prototype representation (sense box embedding) bp. Given a query set EQ = {x1, x2, ..., xNQ}, box embedding of each sample xi is computed by
bqi = fθ(xi, z). (3)
Let us suppose C = {s1, s2, ..., sNC} is the set of the senses (of different words) in the entire support set and EPj = {xj1, xj2, ..., xjNP } is the support set of the jth sense sj . The prototype representation of sj , b p j , is defined as the mean of the box embeddings of the samples in the support set:
bpj = 1
NP
NP∑
i=1
fθ(xji, z). (4)
In the above two equations, z stands for the position of the target word in the sentence.
Following Onoe et al. (2021), we use the binary cross-entropy loss between the prototype sense sj in the support set and the sample xi in the query set:
l(bpj ,b q i ) =− δ·logP (b p j |b q i )
− (1− δ)·log (1− P (bpj |b q i )).
(5)
Here, δ is 1 if the prototype sense sj is equal to the sense of xi or sj is a hypernym of xi, otherwise 0. Finally, the total loss L is defined as follows:
L = 1 NQNC
NQ∑
i=1
NC∑
j=1
1 2 (l(bpj ,b q i ) + l(b q i ,b p j )).
Intuitively, the model fθ is trained so that contextualized box embeddings of the same sense overlap each other and a contextualized box embedding of a hypernym encloses that of a hyponym.

heading: 3.4 Episode Construction
text: The training data of ProtoBox is a collection of “sense instances.” A sense instance is an example sentence including a certain sense of a target word. To train the model, the training data is divided into episodes. Note that each episode is a pair of support and query sets, (ES , EQ). The following sets are made: (1) W , a set of small number of randomly chosen target words, (2) PS , a set of senses of the target words inW , and (3) PH, a set of direct hypernym senses of the senses inPS . Then, several senses in PS and PH are chosen as the prototype senses, thus the support set is formed by sense instances of those prototype senses. The query set is made up of the sense instances of the senses in PS that are mutually exclusive with the support set. We limit the number of the target words in each episode to NW , the maximum number of the prototype senses to NC , the maximum number of sentences for each prototype sense to NP (the maximum number of sentences in the entire support set is NC ×NP ), and the maximum number of the sentences in the query set to NQ.
Algorithm 1 shows how the episodes are constructed. First, NW words are randomly chosen
Algorithm 1 Construction of Episodes
1: Dtrain: the training data 2: V: all words in the training data 3: E ← ∅ 4: while V ≠ ∅ do 5: W ← RANDOM(V, NW ) 6: V ← V \W 7: PS ← ⋃ w∈W SENSEOFWORD(w)
8: PH ← ⋃
s∈S DIRECTHYPERNYMS(s) 9: if |PS |+ |PH| > NC then
10: /* ensure |PS |+ |PH| = NC */ 11: PH ← RANDOM(PH, NC − |PS |) 12: ES ← ∅; EQ ← ∅ 13: /* Dtrains : sense instances of s in Dtrain */ 14: for s ∈ PS do 15: ẼS ← RANDOM(Dtrains , NP ) 16: ES ← ES ∪ ẼS 17: EQ ← EQ ∪ (Dtrains \ ẼS) 18: for s ∈ PH do 19: ES ← ES ∪ RANDOM(Dtrains , NP ) 20: EQ ← RANDOM(EQ, NQ) 21: E ← E ∪ {(ES , EQ)} 22: return E
(line 5). Second, all senses of the randomly chosen target words are kept as PS (line 7), and all direct hypernym of those senses are kept as PH (line 8). The senses in PS and PH are used as the prototype senses. More precisely, all the senses in PS are kept as prototype senses, while the rest are randomly chosen from PH so that the total number of prototype senses becomes NC (lines 9–11). Then, the randomly chosen NP instances for each prototype sense are kept as the support set ES (lines 15–16, 19), while the NQ instances of the senses in PS , which were not selected in the support set, are chosen as the query set EQ (lines 17, 20). Note that the function RANDOM(S, n) randomly chooses n samples at most from the set S; all samples are chosen when |S| < n. The above procedure is repeated until all words in the training data have been used to make episodes.
Since instances of hypernym senses as well as all the senses of a target word are included in the support set, ProtoBox can consider not only the sense discrimination, as does MetricWSD, but also the hypernym–hyponym relations in the training of the model that produces contextualized box embeddings.

heading: 4 Applications of ProtoBox
text: This section describes how ProtoBox is applied to three tasks: Word Sense Disambiguation, New Sense Classification, and Hypernym Identification.

heading: 4.1 Word Sense Disambiguation
text: Task Definition The goal of Word Sense Disambiguation (WSD) is to select the most appropriate sense of the target word w in a given context x from a predefined inventory Sw of senses.
Method First, we get the contextualized box embedding bq of w in x. Second, we create the sense embedding bpi for each sense si in Sw from the training data. Finally, we calculate the similarity score between bpi and b
q using Equation (6), which measures by how much two box embeddings overlap, then the most similar sense is chosen to be the predicted sense.
sim(bpi ,b q) = 2× P (b
p i |bq)P (bq|b p i )
P (bpi |bq) + P (bq|b p i )
(6)

heading: 4.2 New Sense Classification
text: Task Definition The goal of New Sense Classification (NSC) is to classify the target word w in a given context x, whether it has a new sense or not. In this study, new senses are defined as senses that do not appear in the training data.
Method First, we get bq and bpi in the same way that WSD does. For all senses si in Sw, if sim(bpi ,b
q) is smaller than a threshold αsi , w in x is predicted to be a new sense, otherwise not.
Then αsi is determined for each sense using the training and development data. Let Ddevsi be a set of sense instances of si in the development data. The threshold is set to be
αsi = 1
|Ddevsi |
|Ddevsi |∑
j=1
sim(bpi ,b q j), (7)
where bpi is the box embedding of the prototype sense si and b q j is the box embedding of the jth instance in Ddevsi . That is, αsi is determined as the average similarity between the sense instance of si in the development data and the prototype sense si in the training data. When there is no sense instance of si in the development data, the threshold is set to the average of αsi for all senses.

heading: 4.3 Hypernym Identification
text: Task Definition Hypernym Identification (HI) is the task of predicting a hypernym of a new sense. Specifically, for a given new sense of a target word w in a context x, we choose and rank the top ten senses that are most likely to be a hypernym of it.
Method First, we get the contextualized box embedding bq of w in x and the box embeddings of the prototype senses bpi as in the WSD task. Then, the setH of candidates of hypernym senses is created: H = {si | P (bpi |bq) > β}, (8) where β is a pre-defined threshold. Next, we choose the sense where the difference of the volume of bpi and b
q is the smallest as the best hypernym sense u.
u = arg min si∈H |Vol(bpi )−Vol(bq)| (9)
The motivation to consider the difference of the volumes is that when the volume of the box embedding is large, the sense may be an abstract concept and not likely to be a direct hypernym of an input new sense. Finally, all other senses are ranked by their similarity with u (using Equation (6)) and the top nine senses are chosen to make the final ranked list of the hypernyms.

heading: 5 Experiments

heading: 5.1 Dataset
text: Following the WSD framework proposed by Raganato et al. (2017), we use SemCor 3.0 (Miller et al., 1994) as the training data, SemEval-2007 (Pradhan et al., 2007) as the development data, and Senseval-2 (Edmonds and Cotton, 2001), Senseval3 (Snyder and Palmer, 2004), SemEval-2013 (Navigli et al., 2013), SemEval-2015 (Moro and Navigli, 2015) as the test data. All datasets are corpora annotated with sense labels defined by WordNet (Miller, 1995).
In this work, the only the senses of nouns in the datasets are used. In WordNet, senses of nouns connected by hypernym–hyponym relations form a Directed Acyclic Graph of which the root is the synset “entity.n.01”. We create three datasets: Dliving_thing, Dartifact, and Dentity. These datasets consist of instances of hyponyms of “living_thing.n.01”, “artifact.n.01”, and “entity.n.01” in WordNet, respectively. Here, Dentity is a large dataset that includes all nouns, whileDliving_thing andDartifact are smaller ones including a restricted number of nouns.
Training data The statistics of the training data Dtrain are presented in Table 1. The sizes of Dtrainliving_thing and Dtrainartifact are almost the same, while Dtrainentity is much larger than they are.
Development and test data The development and the test data for the WSD task are constructed from instances including a target word that has multiple senses and its gold sense appears in the training data. Statistics are shown in Table 2. It is found that a considerable number of test instances have infrequent senses.
The development and the test data for the NSC task are constructed from instances including a target word that appear in the training data. The instances are labeled as “new sense” if its gold sense does not appear in the training data, otherwise as “not new sense”. The statistics are shown in Table 3.
The development and the test data for the HI task are constructed from instances whose gold senses do not appear in the training data. The statistics are shown in Table 4. The gold hypernym is determined by WordNet.

heading: 5.2 Settings
text: Baselines We prepare two baselines: vanilla BERT (BERT-NN) and MetricWSD (Chen et al., 2021). These models output a contextualized embedding (single vector) r for a given sense instance. In BERT-NN, the embedding of a prototype sense are obtained by the average of the vectors of sense instances derived from the pre-trained BERT. The similarity between two vectors ri and rj is defined as the dot product sim(ri, rj) = ri · rj .
The baselines perform WSD and NSC in the same way as our method, except that the similarity between two instances is measured by two single vectors. In HI, the baseline chooses the ten most similar senses to make up a ranked list of hypernym senses.
Parameters For all models in BERT-NN, MetricWSD, and ProtoBox, we use bert-base-uncased as the BERT model. We set the number of dimensions of the output layer of FCL to 256 (i.e. the size of c and o is 128), NW is 32, NC is 128, NP is 5, and NQ is 64. As for the hyperparameters for the fine-tuning of BERT, the learning rate is set to 1e-5. The number of epochs is optimized, that is, it is varied from 1 to 200 and the best value is chosen using the development data.

heading: 5.3 Results and Analysis
text: Word Sense Disambiguation Table 5 shows the accuracy on the WSD task. As can be seen from the column “ALL”, our ProtoBox outperformed the two baselines for Dliving_thing, but was comparable for Dartifact and Dentity. We guess that the poor performance on Dentity was caused by the scale, that is, our method failed to obtain appropriate contextualized box embeddings when it was applied to many sense instances. The reason why ProtoBox was worse than MetricWSD on Dartifact may not be a scale issue, but the semantic domain of the target noun, since the sizes of Dliving_thing and Dartifact were almost the same.
A similar tendency for the disambiguation of infrequent senses can be seen in the column “≤ 10”. Surprisingly, BERT-NN achieved the best accuracy
on Dentity, although the pretrained BERT model was just applied without fine-tuning. MetricWSD and ProtoBox still suffered from the data sparseness when they were applied to the large dataset.
New Sense Classification The results on the New Sense Classification task are shown in Table 6. Comparing the F1-score, ProtoBox was comparable to MetricWSD onDliving_thing andDentity, and significantly worse on Dartifact. The poor performance for NSC for the senses of artifacts was coincident with the results of the WSD task, where ProtoBox was worse than MetricWSD on Dartifact. The F1-score of BERT-NN on Dliving_thing was notable as it was better than that of MetricWSD and ProtoBox. ProtoBox is designed to learn hypernym– hyponym relations between senses, but such knowledge may not be indispensable for New Sense Classification. This might be the reason why ProtoBox could not outperform MetricWSD.
Hypernym Identification Following previous work on Taxonomy Expansion (Shen et al., 2020; Yu et al., 2020; Jiang et al., 2023), we evaluate the baselines and ProtoBox in term of three metrics: accuracy (ACC), Mean Reciprocal Rank (MRR), WuPalmer similarity (W&P) (Wu and Palmer, 1994). Accuracy measures the agreement ratio between the gold hypernym and the highest ranked hypernym, while Wu–Palmer similarity measures how closely these two hypernyms are located in WordNet. The parameter β described in subsection 4.3 is set to 0.5, 0.7, or 0.9.2
The results on the HI task are shown in Table 7. ProtoBox outperformed the baselines in all three evaluation metrics on the three datasets. In particular, the difference between ProtoBox and the baselines was significant onDliving_thing. The many gold hypernyms in the test data ofDliving_thing were “person.n.01”, which were correctly predicted by ProtoBox. On the other hand, on Dartifact and Dentity, the differences in terms of ACC and MRR between ProtoBox and the baselines were small. However, a significant difference of W&P was confirmed, indicating that ProtoBox could predict hypernyms closer to the correct ones. Finally, the performance of ProtoBox was sensitive to the parameter β, especially in terms of ACC and MRR. Investigating how to optimize β would be the important future work.
2β was not optimized due to the insufficiency of the development data.

heading: 5.4 Optimization of Number of Dimensions
text: We analyzed how the performance of WSD was influenced by the number of the dimensions of the box embeddings c and o. In this experiment, the number of dimensions of the box embeddings was set to {32, 64, 128, 192, 256}. Table 8 shows the accuracy of WSD on the development data of Dentity. It was found that the best performance for both “ALL” and “≤ 10” was obtained when the number of dimensions was set to 128. Therefore, as described in subsection 5.2, the number of dimensions was set to 256 (128 + 128). For NSC and HI tasks, we did not optimize this since the development data was small, but set it to be the same number as for the WSD task.

heading: 6 Conclusion
text: This paper proposed ProtoBox, an expansion of MetricWSD to learn contextualized box embeddings. The representations of words in a context were changed from single vectors in MetricWSD to box embeddings in our ProtoBox, since box embeddings are suitable to represent semantic relations between senses such as the hypernym–hyponym relation. Additionally, we proposed a method to construct episodes to train the model to produce the contextualized box embeddings. We evaluated ProtoBox on three tasks: Word Sense Disambiguation (WSD), New Sense Classification (NSC), and Hypernym Identification (HI). ProtoBox outperformed the baselines in terms of all evaluation metrics in the HI task. This was reasonable, since ProtoBox was designed to take the hypernym–hyponym relation into account when training the contextualized box embeddings. In addition, ProtoBox achieved a performance comparable with the baselines for the other sense related tasks, WSD and NSC.
In the future, the scalability of ProtoBox should be improved. As reported in subsection 5.3, the performance of ProtoBox was degraded when the number of sense instances was large. A more ef-
ficient and precise method to learn contextualized box embeddings should be investigated. In addition, the definition of the prototype representation (the box embedding of a sense) can be reconsidered. Currently, the prototype representation is an average of box embeddings of the elements in the support set. However, it can be a box that includes all the elements in the support set. It is worth to explore better ways to obtain the prototype representation.
Limitations
In the experiments, ProtoBox was only applied to nouns. Additional experiments are required to investigate how ProtoBox can work well for other parts of speech, such as verbs.
The parameter β in the HI task was not optimized due to the insufficiency of the development data. It is worth investigating how to find an appropriate threshold in the future.
We did not compare ProtoBox with other methods of contextualized box embeddings such as Onoe et al. (2021) and Jiang et al. (2023) in the experiments, since the target tasks were not completely the same as the three tasks in this paper. However, empirical comparison is necessary to clarify the contribution of our method.
Ethics Statement
Since ProtoBox was developed using the established datasets for WSD that have been widely used in the community and contain no private information, there is no concern for data and privacy.

heading: Future Work
text: In the future, the scalability of ProtoBox should be improved. As reported in subsection 5. 3, the performance of ProtoBox was degraded when the number of sense instances was large. A more ef-
ficient and precise method to learn contextualized box embeddings should be investigated. In addition, the definition of the prototype representation (the box embedding of a sense) can be reconsidered. Currently, the prototype representation is an average of box embeddings of the elements in the support set. However, it can be a box that includes all the elements in the support set. It is worth to explore better ways to obtain the prototype representation. Limitations
In the experiments, ProtoBox was only applied to nouns. Additional experiments are required to investigate how ProtoBox can work well for other parts of speech, such as verbs. The parameter β in the HI task was not optimized due to the insufficiency of the development data. It is worth investigating how to find an appropriate threshold in the future. We did not compare ProtoBox with other methods of contextualized box embeddings such as Onoe et al. (2021) and Jiang et al. (2023) in the experiments, since the target tasks were not completely the same as the three tasks in this paper. However, empirical comparison is necessary to clarify the contribution of our method