FUSE: Measure-Theoretic Compact Fuzzy Set Representation for Taxonomy Expansion (2024)

Abstract: Taxonomy Expansion, which models complex concepts and their relations, can be formulated as a set representation learning task. The generalization of set, fuzzy set, incorporates uncertainty and measures the information within a semantic concept, making it suitable for concept modeling. Existing works usually model sets as vectors or geometric objects such as boxes, which are not closed under set operations. In this work, we propose a sound and efficient formulation of set representation learning based on its volume approximation as a fuzzy set. The resulting embedding framework, Fuzzy Set Embedding (FUSE), satisfies all set operations and compactly approximates the underlying fuzzy set, hence preserving information while being efficient to learn, relying on minimum neural architecture. We empirically demonstrate the power of FUSE on the task of taxonomy expansion, where FUSE achieves remarkable improvements up to 23% compared with existing baselines. Our work marks the first attempt to understand and efficiently compute the embeddings of fuzzy sets.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 2707–2720 August 11-16, 2024 ©2024 Association for Computational Linguistics
Taxonomy Expansion, which models complex concepts and their relations, can be formulated as a set representation learning task. The generalization of set, fuzzy set, incorporates uncertainty and measures the information within a semantic concept, making it suitable for concept modeling. Existing works usually model sets as vectors or geometric objects such as boxes, which are not closed under set operations. In this work, we propose a sound and efficient formulation of set representation learning based on its volume approximation as a fuzzy set. The resulting embedding framework, Fuzzy Set Embedding (FUSE), satisfies all set operations and compactly approximates the underlying fuzzy set, hence preserving information while being efficient to learn, relying on minimum neural architecture. We empirically demonstrate the power of FUSE on the task of taxonomy expansion, where FUSE achieves remarkable improvements up to 23% compared with existing baselines. Our work marks the first attempt to understand and efficiently compute the embeddings of fuzzy sets.

heading: 1 Introduction
text: Taxonomy is a crucial data structure for modeling semantic concepts, hence of great importance for NLP (Lu et al., 2023; Xu et al., 2023; Yu et al., 2023). Concepts in a taxonomy can often be viewed as sets, the most fundamental object in mathematics, whose operations directly link to First Order Logic (FOL). For example, in a science taxonomy, “Biology” and “Computer Science” are semantic concepts, whose intersection results in a new concept “Bio-informatics”, and “Diffusion Model” and “GAN” belong to a coarser-grained concept, “Generative Model”. Usually, sets are seen as a fixed collection of objects. For example, the set N consists numbers {0, 1, · · · } by definition. However, in the context of semantic concepts, their
meanings can change overtime and incorporate ambiguity. For example, “beauty" is a concept that has become broader overtime, and “deep learning models" can expand to have more elements with more discoveries made by the community. This underlying uncertainty and ambiguity are instead captured by a fuzzy set (Zadeh, 1999, 1978), an extension of classical sets.
A wide range of works have been developed for set representation learning. Early efforts are made to construct simple vector embeddings (Mikolov et al., 2013; Pennington et al., 2014; Devlin et al., 2019; Vaswani et al., 2023; Radford et al., 2018) based on similarity measures. To better model complex relationships such as asymmetrical relationships between concepts, geometric embeddings (Jiang et al., 2023; Hamilton et al., 2019; Ren et al., 2020; Ren and Leskovec, 2020) have been developed, which leverages the inherent geometric properties to model hierarchical relationships. However, these methods cannot address all the set operations including intersection, union, and complement. For example, box embedding (Jiang et al., 2023; Ren et al., 2020; Huang et al., 2023) doesn’t define union and complement of boxes. Worse yet, existing geometric objects are not closed under set operations: the union of two boxes is no necessarily a box, which can compromise the consistency of reasoning in the embedding space.
In this paper, we directly tackle the challenge of fuzzy set representation learning for concept modeling. Our objective is to use their volume to quantify their information and their associated uncertainty. However, learning powerful representations for fuzzy sets is challenging. First, although extensive efforts have been made to incorporate deep learning techniques into fuzzy set modeling (Chen et al., 2022; Dasgupta et al., 2022b; Zhu et al., 2022), their training procedure could be expensive when the universe of discourse is large. Second, compared with geometric embeddings, which have
2707
clear definitions of volume, it is unclear how to model the volumes of fuzzy sets due to the introduction of uncertainty and their abstract nature.
To tackle the previous challenges, we propose a principled and learnable model named Fuzzy Set Embedding (FUSE) for fuzzy set representation learning. The core of FUSE is to introduce a compact approximation of fuzzy sets and then prove that FUSE can arbitrarily approximate the original fuzzy sets under reasonable regularity conditions. FUSE avoids the computational burden of accounting for all the elements in the space of discourse at once, while enjoying the properties of fuzzy logic, hence satisfying all set operations. We further introduce a rank-based loss and asymmetric relations to enhance set representation learning. To validate the effectiveness of our proposed FUSE, we evaluate on taxonomy expansion task and show that FUSE can achieve the performance improvement up to 23% compared with state-of-the-art baselines, and we explore the effectiveness of our theoretical formulation through various ablation studies.
Our main contributions can be summarized as follows: (a) We propose an embedding framework to model fuzzy sets and show that the embeddings satisfy all set operations and are closed under set operations. (b) We systematically construct this embedding as a proper approximation of fuzzy sets. (c) We demonstrate the effectiveness of this embedding framework on taxonomy expansion by comparing it against previous vector and geometrybased embedding methods.

heading: 2 Related Work

heading: 2.1 Taxonomy Expansion
text: Taxonomy organizes concepts as a hierarchical graph, where nodes are concepts and edges denote “is-a” relationships between parent and child nodes. As new knowledge is emerging, taxonomy expansion seeks to expand existing taxonomy with new nodes, which is a fundamental task for many realworld applications such as information filtering and recommendation. Existing works have focused on using a lexical vector representation in the spirit of language modeling and word embedding (Chang et al., 2018; Snow et al., 2004; Mikolov et al., 2013; Pennington et al., 2014). More recently, geometric embeddings such as box embedding has been used to better model the asymmetric relationship between parent and child nodes (Jiang et al., 2023). Compared to vector-based representations, they im-
proved both the predictive performance and interpretability of the learned embeddings.

heading: 2.2 Set Representation Learning
text: Set representation learning seeks to learn lowdimensional representations of data with a notion of volume and coverage. It is desirable when the representations can capture the rich semantic information and the complex relationships of data (Rossi et al., 2020; Wang et al., 2021; Zhang et al., 2022; Zhong et al., 2023). For example, language modeling (Devlin et al., 2019; Vaswani et al., 2023; Radford et al., 2018) has aimed to learn vectors to represent combinatorically intractable combinations of human languages. In this context, semantic concepts can be viewed as sets. Recently, geometry-based approaches (Ren et al., 2020; Dasgupta et al., 2022b; Ren and Leskovec, 2020; Chen et al., 2021) have further improved the efficiency of the representations by enabling set operation such as intersection, but they fail to cover all operations and are not closed under them. Fuzzy set theory has explicitly formulated a way to represent the ambiguity of sets such as concepts in taxonomy construction, while automatically satisfying all desired properties of sets (Chen et al., 2022; Zhu et al., 2022). It is an extension to classical set theory with extensive applications (van Krieken et al., 2022; Wagner and d’Avila Garcez, 2022; Liang et al., 2023; Yu et al., 2022; Xu et al., 2022). For example, Michael Boratko and McCallum (2022) and Dasgupta et al. (2022b) have explored the connection between fuzzy sets and box embeddings to model words. However, existing fuzzy set representations lack a principled approach on what the low-dimensional representation stands for, and can be inefficient when the number of sets to model increases. We propose a novel solution by identifying the central characterization of a fuzzy set as its volume and approximate it using a compact representation, while yielding superior performance on the set representation learning task of taxonomy expansion.

heading: 3 Preliminary

heading: 3.1 Fuzzy Sets
text: In contrast to classical set theory, which assigns a Boolean value to whether an element belongs to a set, a fuzzy set (Zadeh, 1978) assigns a value between 0 and 1 to denote a degree of membership. For a universe of discourse U , a fuzzy set is
mathematically defined as a tuple A = (U,mA), where A ⊆ U and mA : U → [0, 1] is its membership function. For any element x ∈ U , mA(x) represents the degree of membership of element x in A. Fuzzy set models the uncertainty of membership by encoding imprecision and ambiguity in concepts. As an example, it can be used to describe the compatibility between two concepts, such as “is-a" relationship. For example, for a concept “Kobe Bryant" and a set of entities {Basketball Player, Team Owner, Entrepreneur}, a fuzzy membership function can be represented as the set {1.0, 0.1, 0.9}, each signifying “Kobe Bryant"’s compatibility with each of the concepts in the set.
Similar to standard sets, intersection, union, and complement between fuzzy sets are defined. Fuzzy set is related to fuzzy logic, which defines logical operations over soft truth values and follows Gödel, product, or Łukasiewicz systems. For a detailed discussion of fuzzy logic systems, see Chen et al. (2022). For language modeling, fuzzy sets can be used to model the ambiguity of the semantic meanings of words (Dasgupta et al., 2022a). In taxonomy, fuzzy sets can be used to model concepts.

heading: 3.2 Possibility Theory
text: The membership function mA associated with a fuzzy set A is constructed based on the theory of possibility in Zadeh (1999, 1978). In the formulation of Zadeh (1978), to reason about linguistic concepts such as “likely”, a fuzzy set can be endowed with a probability-possibility distribution: Definition 1 (Possibility-Probability Distribution). Let U be the universe of discourse, and (U,F , P ) be a probability space, where F is the sigma-algebra and P is the probability measure. Let X be a fuzzy variable that can take any values x ∈ U , and let F be a fuzzy subset of U with membership function mF , then the possibility of probability of X with respect to F is:
πP,X =
∫
U πXdP =
∫
U mFdP.
where πX is the possibility distribution associated with X and ∫ U mFdP is the Lebesgue integral of the membership function w.r.t to the probability measure P .
Here the Lebesgue integral ∫ U mFdP , in the sense of fuzzy set theory, represents the amount of information contained by the fuzzy set F . This construct can be seen as the measurement of information and uncertainty in the fuzzy variable X ,
making it a desirable quantity to approximate when learning a low-dimensional embedding of a fuzzy set. In our Fuzzy Set Embedding, we generalize definition 1 in definition 4.

heading: 4 Proposed Framework: FUSE
text: We now present Fuzzy Set Embedding (FUSE) for learning set representations in a principled way.

heading: 4.1 Fuzzy Set Embedding
text: To construct a proper embedding for fuzzy sets, we assume that the universe of discourse U admits a finite partition, {Ui}di=1, such that U = ⋃d i=1 Ui, and Ui, Uj are disjoint if i ̸= j. For a formal description of this assumption, see Appendix B. In particular, this indicates that fuzzy set membership function mA has an associated simple function:
Definition 2 (Simple Fuzzy Set). Let (U,F , ξ) be a measure space and Let U = ⋃d i=1 Ui be a finite partition of the universe U , and let A ∈ F and mA its membership function, then the Simple Fuzzy Set associated with A is the tuple (U, µA), where:
µA(x) := d∑
i=1
1{x∈Ui}µ (i) A (x), ∀x ∈ U (1)
is the Simple Membership Function of A, where 1 is the indicator function and ∀x ∈ U,∀i ∈ {1, · · · , d}:
µ (i) A (x) := sup
x∈Ui mA(x). (2)
µA can be summarized in d values µ (1) A , · · · , µ (d) A , each determined by the supremum of mA in the corresponding partition. To facilitate the standard application in deep learning, we formulate an alternative representation in vector form to distinguish it from the functional representation denoted in Eqn. 1.
Definition 3 (Fuzzy Set Embedding). Let A = (U, µA) be a simple fuzzy set defined on the measure space (U,F , ξ), where U = ⋃di=1 Ui, then its corresponding Fuzzy Set Embedding (FUSE) is the d-dimensional vector:
UA := [µ(1)A , · · · , µ (d) A ], (3)
Since we are reducing the reasoning space from [0, 1]|U | to [0, 1]d, we need to examine the loss incurred by this reduction. To reason about it in detail, we provide the following definition inspired
by Zadeh (1978); Nahmias (1978) to quantify the amount of information covered by the fuzzy sets across the entire universe U .
Definition 4 (Simple Fuzzy Measure). Let U be a compact universe of discourse and let A = (U,mA) be a fuzzy subset of U . Let (U,F , ξ) be a measure space defined on U , then the fuzzy measure of the fuzzy set (U,mA) is:
P(A) := ∫
U mAdξ.
Then a Simple Fuzzy Measure of a simple fuzzy set A = (U, µA) is defined as:
Pµ(A) := ∫
U µAdξ.
Given a finite partition, furthermore:
Pµ(A) = d∑
i=1
∫
Ui
µ (i) A dξ =
d∑
i=1
µ (i) A ξ(Ui), (4)
where ξ(Ui) corresponds to the measure of partition set Ui. If ξ is a probability measure, then Definition 4 corresponds to Definition 1. In practice, we examine choices of different measures empirically in Section 5. In short, a simple fuzzy set A = (U, µA) approximates the fuzzy measure of the underlying fuzzy set (U,mA). We state this observation formally in theorem 1 and illustrates it in Figure 2(a).
Theorem 1. Let U be a compact universe of discourse and (U,F , ξ) a measure space. Let A be a fuzzy subset of U and mA its membership function that’s measurable. Moreover, let µA be its simple membership function, then ∀ϵ > 0, ∃δ > 0, d > 0 such that if dδ = ξ(U) and ||Ui|| := mini ξ(Ui) < δ, we have:
0 < Pµ(A)− P(A) < ϵ.
The fuzzy measure of a simple fuzzy set is an upperbound for the fuzzy measure of its underlying fuzzy set and converges to the possibility of its underlying fuzzy set when the partition is sufficiently fine-grained. With suitable assumption on the function mA, we can further establish the rate of convergence:
Corollary 1. Following the condition in definition 4, if in addition mA is Lipschitz-continuous or of bounded variation, then the convergence rate in 1 is O(1/n), where n is the number of partitions.

heading: 4.2 Embedding-based Fuzzy Set Operators
text: FUSE combines set theory and measure theory and provides a theoretically sound embedding. For an entity x ∈ U , we treat it as a concept and associate with it a fuzzy subset (U,mA), representing its compatibility with other concepts. We can treat every entity as a fuzzy set embedding, define set operations in the language of set theory, and compute them using vector operations. Suppose we have two entities x, y ∈ U , and UA,UB ∈ [0, 1]d
are the two fuzzy set embeddings associated with them, we can define following operations:
• Fuzzy Mapping: Every entity/element is a singleton set, and M maps an entity x ∈ U to its associated fuzzy set embedding U{x} ∈ [0, 1]d. In the case of taxonomy expansion, the input is the word vector x ∈ Re obtained from a pretrained language model like BERT (Devlin et al., 2019). To construct a map between the word vector and its associated fuzzy set embedding, we use a neural networks f : Re → [0, 1]d:
UA = M(A; θ) = σ(f(x; θ)) ∈ [0, 1]d, (5)
where σ is a normalization constraint to make the embedding space compact, such as sigmoid, 0-1 clamping, or Layernorm (Ba et al., 2016).
We can futher define set operations by product logic. Other fuzzy systems, such as Gödel logic, is illustrated in Fig. 1.
• Intersection: The intersection between the two fuzzy sets A ∩ B can be computed by elementwise product t-norm (Klement et al., 2013):
UA∩B = UA ⊙ UB. (6)
where ⊙ is element-wise multiplication.
• Union: The union of two fuzzy sets A ∪B can be computed by element-wise product t-conorm:
UA∪B = UA + UB − UA ⊙ UB. (7)
• Complement: The complement of a fuzzy set A denoted as Ac can be computed as:
UAc = 1− UA. (8)

heading: 4.3 Taxonomy Expansion with FUSE
text: In this part, we use taxonomy expansion task to showcase the advantages of representing concepts with fuzzy set embeddings.
Membership Prediction with FUSE. After representing a concept with fuzzy sets, the core task of taxonomy expansion is to determine whether an element y belongs to a setA, and this is often used as a score function in pair-based relationship in taxonomy expansion task (Jiang et al., 2023; Shen et al., 2020; Yu et al., 2020). Using our framework, for some element y ∈ U , we can apply the entity mapping function M to find its fuzzy set embedding U{y}. Then we can simply measure the degree of membership of element y in some other fuzzy set A by considering the fuzzy measure of the fuzzy set embedding UA∩{y} = UA ⊙ U{y}, which we denote as Pµ(A ∩ {y}) and compute it as:
Pµ(A ∩ {y}) = d∑
i=1
( U (i)A U (i) {y} ) ξ(Ui). (9)
In training, we model the volume using global trainable weights w = {w1, · · · , wd} with a normalization transform to restrict the type of measure ξ. Therefore, we approximate PA∩{y} and define the standard score function:
ψ(y,A) = d∑
i=1
µ (i) A∩{y}wi = (UA ⊙ U{y}) Tw.
(10)
and the corresponding ranking-based loss:
L(y,A) =− log σ (ψ(y,A)− γp)
− 1 k
k∑
i=1
log σ ( γn − ψ(y′, A) ) (11)
where (y,A) are positive pairs and (y′, A) negative pairs, and γp, γn are margins for positive and negative predictions. We use different margins since by Theorem 1, fuzzy measure of the fuzzy set embedding is an upperbound for the underlying fuzzy set, so the result we obtain is overestimating the actual fuzzy measure. We provide an ablation study regarding choice of margin in section 5.
Incorporating Asymmetric Relation. As signified in Jiang et al. (2023), membership prediction in a taxonomy expansion task usually involves asymmetric relations. For example, parent nodes in a taxonomy usually strictly incorporate the concept in the child nodes. Since the score function, which uses the intersection between two sets, is symmetric, here we propose another score function to signify this asymmetry. Suppose we have two entities ep, ec ⊆ U , such that ec is a child of ep. Let Ep, Ec be their associated fuzzy sets, then we can model this relationship by:
P (ep|ec) = Pµ(Ep ∩ Ec)
Pµ(Ec) = (UEp ⊙ UEc)Tw UEc ,
(12)
where we use the simple fuzzy measure for each set as its volume and use the ratio between the volume of Ep ∩Ec and the volume of Ec as the result. For a positive child-parent pair (ec, ep), the loss is:
L+p = (P (ep|ec)− 1)2, (13)
whereas for a negative child-parent pair (ec, e′p):
L−p = (P (e ′ p|ec)− 0)2. (14)
The main difference between Eqn. 10 and the case of box embedding in Jiang et al. (2023) is that the volume of a fuzzy set embedding spans the entire universe U . We combine the pair-based ranking
loss and the asymmetric child-parent pair loss:
Ltaxo = L(ec, ep) + λ(L + p + L − p ), (15)
where λ is a hyper-parameter to control the strength of each loss.

heading: 5 Experiments
text: Dataset: We use two public datasets (Environment, Science) from SemEval-16 taxonomy construction tasks. Following the training setup in Jiang et al. (2023), we sample 20% of the leaf nodes as test set and use the rest as training data. The performance
of taxonomy expansion task can be found in table 1, where the results are averaged over 5 runs to reduce variance.
Metrics: We use three metrics, Accuracy (ACC), Mean Reciprocal Rank (MRR), and Wu & Palmer similarity (Wu&P) to measure the performance for the baseline models and FUSE.
Baselines: The baseline comparison models are vector-based models like TAXI (Panchenko et al., 2016), HypeNet (Shwartz et al., 2016), BERT+MLP (Yu et al., 2020), TaxoExpan(Shen et al., 2020), STEAM (Yu et al., 2020), and geometry-based model like BoxTaxo (Jiang et al., 2023).

heading: 5.1 Does FUSE Benefit from More Partitions?
text: Here we examine the impact of choice of number of partitions, since according to theorem 1, as d increases, we should expect better approximation to the underlying fuzzy set. In this experiment, we vary the number of partitions from 100 to 550, with increment of 50, and measure the performance of taxonomy expansion on both Science and Environment datasets, averaged over 5 runs. The resulting trend on both datasets can be seen in Figure 4.
In both cases, we can see a general upward trend on model’s average performance when the number of partitions goes up. This provides empirical support for the theoretical result in theorem 1. The variance of model performance tends to increase when the number of partitions goes up, which suggests that smaller learning rates and other normalization for optimization may be considered.

heading: 5.2 Does the Choice of Measure Affect Model Performance?
text: As for the proposed new score function based on the volume of the fuzzy set in Eqn. 10, we study the impact of different choices of measure ξ. This corresponds to different choices of normalization applied to the volume of each partition. If we follow definition 1, then partition volumes follows a probability distribution, indicating a softmax normalization on the global weights. Otherwise, we can choose to use sigmoid or 0-1 clamping. Results over 5 runs for different choices of measures can be found in table 2. We observe that the softmax normalization, which enforces the volume weights to follow a probability distribution, doesn’t work well overall. This suggest that the construction we proposed in definition 4 is more suitable than the classical definition in 1. We also observe that using a sigmoid normalization on the volume weights can improve results.

heading: 5.3 Does Asymmetry Loss Help Taxonomy Expansion Task?
text: In this ablation study, we examine the impact of applying asymmetry losses, since set intersection is a symmetric operation, whereas the membership relation is asymmetric. The value of λ in Eqn. 12 controls how much should the asymmetry-based loss affect the training (the greater the value of λ, the greater the impact). The results, averaged over 5 runs is in figure 5. We can see a trend of performance improvement on Science dataset and an increase in performance for λ > 0 for the Environment dataset. This result validates the importance of modeling asymmetric relations.

heading: 5.4 Does Wider Margin Affect Model Performance?
text: In this ablation study, we examine the impact of different margins on the learning performance, as presented in Eqn. 11, since theoretically, our construction of FUSE over-estimates the volume under the fuzzy set. In figure 6, which presents the average result over 5 runs. From the result, we can see that having wider margin (in this case γp = 0.6, γn = 0.4) does benefit the model performance, supporting our hypothesis that FUSE over-estimates the volume of a fuzzy set.

heading: 5.5 Additional Experiment: Is there Synergy Between Different Modeling Choices?
text: In this part, we examine the synergy between configurations across all the ablation studies performed. In this case we study the case where all the best configurations (λ = 0.5,∆γ = 0.2, sigmoid-normalization) are used. We call the resulting model FUSE-comb(ine) and report the performance averaged over 5 runs compared to other models in table 3. The result doesn’t immediate suggest that combining multiple best case scenarios would result in optimal performance. The strongest performance so far comes from setting λ = 0.5.

heading: 5.6 Infer about Union and Complement Using Trained Embeddings
text: In our taxonomy expansion experiment, the model is trained using only the volume-based intersection and asymmetry losses, without observing pairs of sets that are related by union or complements. In this case study we examine whether our embedding can generalize to these two operations.
Infer about Set Union: For a parent entity ep in the taxonomy and its m child entities ec1 , · · · , ecm ,
we examine the similarity of union of fuzzy set embeddings of child entities with the fuzzy set embedding of the parent entity. That is, between U⋃m i=1 Eci and UEp . To this end, we use the trained fuzzy set embedding from the FUSE (λ = 1) model and apply union operation in Eqn. 7 among all the child fuzzy set embeddings, then we rank the Euclidean distance between the obtained fuzzy set embedding (union of all child embeddings) against all the existing parent embeddings in the dataset. From 4, we observe that fuzzy set embedding captures union patterns. As an example from the Science dataset, for child entities [“calculus of variations", “analysis", “integral calculus"], with parent entity “calculus", the top-3 closest simple fuzzy set embedding corresponds to entity “calculus", “analysis", “geophysics", and the model’s prediction is closest to the correct parent.
Infer about Set Complement: In this case we examine complement by the set operation A \B = A ∩Bc. In particular, the fuzzy set for parent (denote it A) minus a child fuzzy set (denote it B) should be similar to the union of the remaining children fuzzy set. We follow the union, complement, and intersection operation to compute the fuzzy set embedding in this case, and again we use the embedding from FUSE (λ = 1) model. We compute the Euclidean distance between A ∩ Bc and all the existing child embeddings in the dataset. Here we present MRR and accuracy result in table 4. In contrast to union, it seems that complement doesn’t achieve a reasonable performance. This may be due to the fact that the complement of a fuzzy set is taken over the entire universe of discourse, rather than simply in the scope of all the children entities.

heading: 6 Conclusion
text: For taxonomy expansion, We propose a novel and theoretically sound Fuzzy Set Embedding (FUSE) to model concepts and relationship between concepts that incorporate set operations (intersection, union, complement). We show theoretically that FUSE preserves the information of the fuzzy set
with sufficiently fine-grained partitions and demonstrate empirically that it can outperform existing vector-based and geometry-based embedding methods on taxonomy expansion. For future works, we believe that expanding the taxonomy dataset with more complicated combination of set operations, such as First Order Logic (FOL), can further improve the model performance.

heading: 7 Limitations
text: This work is the first attempt to use fuzzy set to model concepts in taxonomy expansion. To examine only the effectiveness of fuzzy set representation, we only use simple neural architectures and use only the child-parent pairs. The full capacity of FUSE should be further examined using datasets that contain First Order Logic (FOL) statements, since by construction, fuzzy sets should satisfy all the fuzzy logic axioms (Chen et al., 2022). This suggests future directions to expand taxonomy datasets with more complicated queries, and to handle more graph-structured data in social analysis and text mining (Ren and Leskovec, 2020; Ren et al., 2020; Chen et al., 2022; Zhu et al., 2022; Ju et al., 2023). Moreover, we can explore more explicit form of fuzzy membership function, such as a mixture of Gumbel boxes, to make the learning more concrete.

heading: A List of Symbols and Notations
text: In table 5, we list all the symbols and notations used in the paper, then we provide the theoretical proofs for the main results in the paper.

heading: B Formal Statement of the Compactness Assumption
text: Here we state the assumption regarding the universe of discourse formally:
Assumption 1. The universe of discourse U is topologically compact and has an open cover.
Assumption 2. The universe of discourseU is measurable and is associated with a measure space (U,F , ξ), where F is the σ-algebra and ξ its associated σ-finite measure. Moreover, ∀A ∈ F , the fuzzy membership function mA : U → [0, 1] is ξ-measurable.

heading: C Proof of Main Results
text: In this part we provide the proof sketches of the main results:
Lemma 1: Let U be the universe of discourse and A a fuzzy subset of U with continuous membership function mA, and let ( µ (t) A ) be a sequence of simple membership functions of mA in Definition 2, such that {Ui}nti=1 is a refinement of {Ui} nt−1 i=1 (nt > nt−1), when t goes to infinity, then µ (t) A converges to mA in the point-wise sense.
Proof. The proof takes 2 steps:
Claim 1: By construction, the sequence ( µ (t) A ) is a sequence of monotonic non-increasing function and it is bounded below by the fuzzy membership function mA.
Proof. The case where U is finite or countably infinite is straightforward. For the case where cardinality of U is uncountable but U is compact, we have the following: By definition 2, we have that for i ∈ {1, · · · , d} and n ∈ N, ∀u ∈ Ui, µ (t) A (u) = supu∈Ui mA(u) ≥ mA(u). Hence on the entire domain, µ(t)A (u) ≥ mA(u). So ∀t ∈ N+, m
(t) A is an upperbound for mA. Since nt > nt−1 indicates {Ui}nti=1 is a finer-grained partition than {Ui}nt−1i=1 , and the fact that supremum of a function
Symbol Description
A,B,C, · · · mathematical sets. ec, ep, · · · entities in a taxonomy Ec, Ep, · · · fuzzy sets associated with entities U The universe of discourse, the set of all concepts. N+ The set of all positive integers. (U,F , ξ) A measure space with universe U , sigma-algebra F and a measure ξ. Ac The complement of a set. A ∩B Intersection of two sets. A ∪B Union of two sets. (A,mA) a fuzzy set, where A ⊂ U, mA : U → [0, 1]. mA The fuzzy membership function associated with a fuzzy set. (A,µA) a simple fuzzy set, where A ⊂ U, µA : U → [0, 1]. µA The simple membership function
associated with a simple fuzzy set (A,µA).
(µA,n) a sequence of simple membership functions with monotonically finer-grained partitions. UA The fuzzy set embedding of (A,mA). P(A) The fuzzy measure (volume)
of a fuzzy set under some measure space (U,F , ξ).
Pµ(A) The simple fuzzy measure (volume) of a fuzzy set embedding under some measure space (U,F , ξ). M Fuzzy mapping, which maps an input element into its associated Fuzzy Set Embedding. P A probability measure defined on the space of concepts.
Table 5: Table for all the symbols used in this paper
over finer grained partition is not greater than supremum over coarse-grained partition (e.g., supremum is monotonic w.r.t partitions), we have a nonincreasing sequence of functions.
Claim 2: The sequence of functions ( µ (t) A ) con-
verges to mA.
Proof. Since the space U is compact, we can define the space of function to be a compact Banach space of functions f : U → [0, 1] and so mA is in the space. By the monotone convergence theorem (Folland, 1999) and by the fact that supremum over a singleton set {x} is simply supu∈{x}mA(u) = mA(x), we can conclude that convergence holds in the point-wise sense.
Theorem 1: Let U be a compact universe of discourse and (U,F , ξ) a measure space. Let A be a fuzzy subset of U and mA its membership function that’s measurable. Moreover, let µA be its fuzzy set embedding membership function, then ∀ϵ > 0, ∃δ > 0, d > 0 such that if dδ = ξ(U) and ||Ui|| := mini ξ(Ui) < δ, we have:
0 < Pµ(A)− P(A) < ϵ.
Proof. By Lemma 1 and assumption that mA and hence all {µ(t)A } are ξ-measurable, we have a monotonic non-decreasing sequence of non-negative simple functions that converge point-wise to mA. Then by the Monotone Convergence Theorem of simple functions (Folland, 1999), we have that∫ U mAdξ = limt→∞ ∫ µ (t) A dξ. Moreover, since µ (t) A ≥ mA,∀n ∈ N+, we have that ∀ϵ > 0, ∃N ∈ N, such that ∀t > N , Pµ(A) > P(A) < ϵ. This is equivalent to say (by virtue of construction of simple functions in definition 2) that ∃δ > 0, d > 0, such that dδ = ξ(U) and ||Ui|| := mini ξ(Ui) < δ (or equivalently, the partition is sufficiently finegrained), the conclusion holds.
Here we also provide a proof sketch for the Euclidean case, where the universe U is mapped into a compact subspace U ⊂ Rd, and the fuzzy measure is defined as a Riemann integral in Rd (the Euclidean volume), which is often the case for the embedding space. In this case, the following formulation of the theorem holds:
Theorem 1 (Euclidean Case) Let Ω ⊂ Rd be compact and let (A,mA) be a fuzzy set with membership function mA : U → [0, 1], and let Ω =
⋃d i=1 Ui be a partition and (A,µA) its associated partition-level fuzzy set. Then if mA is Riemannintegrable on Ω, then ∀ϵ > 0, ∃δ > 0, d > 0 such that if dδ = Vol(Ω) and ||Ui|| := mini Vol(Ui) < δ, we have that:
0 < Pµ(A)− P(A) < ϵ
that is, the possibility of a partition-level fuzzy set is an upperbound for its underlying fuzzy set and it converges to the possibility of its underlying fuzzy set when the partition is sufficiently fine-grained.
Proof. The proof takes 3 main steps: Since Ω and [0, 1] are both compact, we need to show that (a) under a rectangular partitions in Rd, as partition granularity increases, the possibility Pµ(A) monotonically decreases and (b) the possibility Pµ(A) is an upper bound of the possibility P(A). After these two are shown, we can simply invoke the standard Monotone Convergence Theorem for compact spaces and show that Pµ(A) converges to P(A) (Folland, 1999; Wilkins, 2016).
Step 1: Show that Pµ(A) monotonically decreases under granular partition. This is equivalent to show that the upper Darboux sum of a d−dimensional Riemann integral monotonically decreases as the partition get finer-grained. This result is Lemma 6.4 in (Wilkins, 2016).
Step 2: Show that Pµ(A) ≥ P(A). This is equivalent to say that d−dimensional upper Darboux sum is an upperbound for its Darboux-Riemann integral. This result is Lemma 6.6 in (Wilkins, 2016).
Step 3: Show that Pµ(A) converges to P(A). By step 1 and step 2, we can construct a sequence of monotonically decreasing upper Darboux sums. By the Riemann-integrability of the function mA and the compactness of the set Ω, [0, 1], we can conclude that by Monotone Convergence Theorem (Folland, 1999), this conclusion holds.
Corollary 1 Following the condition in definition 4, if in addition mA is Lipschitz-continuous, then the convergence rate in 1 is O(1/n). If mA instead has bounded variation, then the convergence rate is also O(1/n), where n is the number of partitions.
Proof. In our case we defined the simple fuzzy set membership function as:
fn(x) = n∑
i=1
sup f(x)1(x ∈ Ui)
where (fn)n∈N , f are measurable functions on the space (U,F , µ) and 1 is the indicator function, and we have U = ⋃n i=1 Ui a partition of the universe. To derive a bound for convergence rate, we need to evaluate ∫ |fn − f |dµ = ∫ (fn − f)dµ, since fn ≥ f . and we have the following result:
∫ (fn − f)dµ = n∑
i=1
sup f(x)1(x ∈ Ui)− n∑
i=1
∫
Ui
fdµ
=
n∑
i=1
∫
Ui ( sup x∈Ui f(x)− f(x) ) dµ
Now suppose that f is Lipschitz continuous with constant L (a much stronger assumption), then we have that
∀x ∈ Ui, | sup x∈Ui f(x)− f(x)| ≤ Lµ(Ui)
Then we have ∫ (fn − f)dµ = n∑
i=1
∫
Ui ( sup x∈Ui f(x)− f(x) ) dµ
≤ n∑
i=1
∫
Ui
Lµ(Ui)dµ = n∑
i=1
Lµ(Ui) 2
without loss of generality, let {Ui} be an even partition and let µ(U) = 1, then we have that µ(Ui) ∝ O( 1n) and so µ(Ui)2 ∝ O(1/n2). Hence Equation 6 decays with O(1/n) rate.
Suppose that f has bounded total variation V (f) <∞, then we have that: ∫ (fn − fd)µ = n∑
i=1
∫
Ui ( sup x∈Ui f(x)− f(x) ) dµ
≤ n∑
i=1
V (f)
n µ(Ui)
Again, we have the conclusion that the error decays with O(1/n).

heading: D Details on Experiments
text: D.1 Baselines For our taxonomy expansion task, we compare with existing methods that use vector embeddings or geometric embeddings. For vector embedding methods, we include also models that use advanced structures of taxonomy data. To summarize, the baselines we compare with are:
• TAXI(Panchenko et al., 2016): This is a vectorbased embedding model that relies heavily on hypernym and hyponym relations between entities.
• HypeNet(Shwartz et al., 2016): This is a vectorbased embedding model that leverages dependency paths between entity pairs.
• Bert+MLP(Yu et al., 2020): This is a vectorbased embedding method that uses Bert (Devlin et al., 2019) to generate entity embeddings. Bert used in this model and in our own model is finetuned with a smaller learning.
• TaxoExpan(Shen et al., 2020): This is a vectorbased embedding method leverages local egographs to model pair dependencies. It uses graph neural networks (GNN).
• STEAM(Yu et al., 2020): This is a vector-based embedding method that samples dependency paths from taxonomy for better structured entity embedding.
• BoxTaxo(Jiang et al., 2023): This is a geometric embedding method that uses box embedding for entities in taxonomy. It is able to capture asymmetric relationships between entities.
D.2 Evaluation Metrics For evaluating the taxonomy expansion task, we follow (Jiang et al., 2023). For the i-th query, denote ai the true anchor and âi the top-1 predicted anchor and let N be the total number of test samples, then the three metrics we use are the following:
• Accuracy (ACC): evaluates the prediction’s overall correctness
ACC = 1
N
N∑
i=1
I(âi = ai)
• Mean Reciprocal Rank (MRR): evaluate the rank of the correct prediction in all predictions
MRR = 1
N
N∑
i=1
1
rank(ai)
• Wu & Palmer similarity (Wu& P) (Wu and Palmer, 1994): measures the semantic similarity between concepts in a taxonomy
Wu&P = 1
N
N∑
i=1
2× depth(LCA(âi, ai)) depth(âi) + depth(ai)
where LCA is the least common ancestor of two inputs and depth is the depth in the taxonomy tree.
D.3 Implementation Detail for Base Model In figure 5 and 6, the base model Fuzzy Set Embedding (FUSE) is the configuration of model with un-normalized global weights corresponding to volumes of each partition. In later ablation studies, we examine the impact of normalization on the volumes, corresponding to a choice of different measure. The number of partition of FUSE on the science dataset is 500, while the number of partition used on the environment dataset is 350. For training stability, we also normalize the fuzzy set embedding by their Euclidean norm before multiplication with volume weights. In addition, to make a fair comparison against baselines, we use the same optimization setup as in (Jiang et al., 2023) and provide a version of FUSE (FUSE (λ = 1.0)) with equal weights on intersection and asymmetry loss.

heading: E Scope and Limitation
text: This work is the first attempt to use fuzzy set to model concepts in taxonomy expansion. To examine only the effectiveness of fuzzy set representation, we only use simple neural architectures and use only the child-parent pairs. The full capacity of FUSE should be further examined using datasets that contain First Order Logic (FOL) statements, since by construction, fuzzy sets should satisfy all the fuzzy logic axioms (Chen et al., 2022). This suggests future directions to expand taxonomy datasets with more complicated queries, and to handle more graph-structured data in social analysis and text mining (Ren and Leskovec, 2020; Ren et al., 2020; Chen et al., 2022; Zhu et al., 2022; Ju et al., 2023). Moreover, we can explore more explicit form of fuzzy membership function, such as a mixture of Gumbel boxes, to make the learning more concrete.

heading: Future Work
text: This suggests future directions to expand taxonomy datasets with more complicated queries, and to handle more graph-structured data in social analysis and text mining (Ren and Leskovec, 2020; Ren et al. , 2020; Chen et al. , 2022; Zhu et al. , 2022; Ju et al. , 2023). Moreover, we can explore more explicit form of fuzzy membership function, such as a mixture of Gumbel boxes, to make the learning more concrete