Decomposing Co-occurrence Matrices into Interpretable Components as Formal Concepts (2024)

Abstract: This study addresses the interpretability of word representations through an investigation of a count-based co-occurrence matrix. Employing the mathematical methodology of Formal Concept Analysis, we reveal an underlying structure that is amenable to human interpretation. Furthermore, we unveil the emergence of hierarchical and geometrical structures within word vectors as consequences of word usage. Our experiments on the PPMI matrix demonstrate that the formal concepts that we identified align with interpretable categories, as shown in the category completion task.

heading: 1 Introduction
text: Word vector representations are central to natural language processing, as they capture semantic and syntactic features (Lenci, 2018). Their significance has been amplified in recent times, as they are used as input for Transformer-based language models (Vaswani et al., 2017), where static embeddings are contextualized. Their effectiveness has been explained by the distributional hypothesis (Harris, 1954) linking similar semantics and similar distribution (Jurafsky and Martin, 2009). However, the interpretability of their dimensions remains an active research topic (Şenel et al., 2018). Levy and Goldberg (2014a) found neural word embeddings to be uninterpretable while acknowledging that sparse vectors capture some latent topics. Geva et al. (2022), among others, pioneered efforts to interpret dynamic embeddings in GPT-2 (Radford et al., 2019) by projecting them into the vocabulary space, though a systematic approach to interpret dimensions of embeddings remains an open issue.
Many preceding studies have investigated the semantic properties of word embeddings and revealed that word vectors in a vector space capture relational meanings. The most well-known example is the parallelogram formed in the vector space
by the embeddings of words in analogical relations (e.g. king:queen::man:woman) (Mikolov et al., 2013c). Other semantic relationships also exhibit geometrical counterparts, such as semantic composition with vector addition (Mikolov et al., 2013b; Mitchell and Lapata, 2008), hypernymy captured by linear projection (Fu et al., 2014), and polysemy as a linear combination of vectors (Arora et al., 2018). Regarding the theoretical analysis of embeddings, Levy and Goldberg (2014b) suggested that word2vec (Mikolov et al., 2013a) is equivalent to the factorization of a word co-occurrence matrix. Arora et al. (2016) proposed a generative model in which PMI-based word embeddings exhibit linear structures. These related studies collectively hint that the latent structure in the co-occurrence matrix reflects linguistic regularities and is inherently embedded within vector representations. Therefore, understanding the word co-occurrence matrix represents a cornerstone in elucidating the interpretability of word representations.
In this study, we directly address the mathematical structure of a word co-occurrence matrix to uncover underlying linguistic patterns and to interpret the dimensions of word embeddings. We claim that a formal concept, as mathematically defined in the matrix, corresponds to human-interpretable categories. We substantiate our claim through the category completion task. Specifically, we used Formal Concept Analysis (FCA), a field of applied mathematics (Ganter and Wille, 2012), to formally characterize the internal structure of a matrix. We define a group of words as interpretable if it can be descriptively labeled. Furthermore, we demonstrate that a hierarchical structure of formal concepts emerges as a geometric formation in the vector space, which explains why relational meanings are captured by word embeddings.
Our contributions are threefold. First, we propose two methods that apply FCA to real-valued data: binarization by varying thresholds and fuzzi-
4683
fication of FCA. Second, we empirically show that the formal concepts in the co-occurrence matrix coincide with interpretable categories. Third, we present a novel algorithm to detect formal concepts, which is capable of disambiguating polysemous words. To our knowledge, this is the first study to apply FCA to a word-word co-occurrence matrix. Our study offers a new approach to uncover latent linguistic structures in co-occurrence matrices.

heading: 2 Formal concept analysis of word co-occurrence matrix

heading: 2.1 Basics of FCA
text: FCA is related to order theory and abstract algebra. It mathematizes concepts and conceptual hierarchy (Ganter and Wille, 2012). A concept comprises a pair of its extents (objects) and its intents (attributes). Concepts can form a hierarchy known as a lattice. FCA has been empirically applied for data mining and ontology (Poelmans et al., 2013), especially in bioinformatics (Roscoe et al., 2022).
A formal context K := (G,M, I) consists of two sets G,M and a binary relation I ⊆ G ×M . The elements of G and M are called objects and attributes, respectively. For g ∈ G and m ∈M , a relation (g,m) ∈ I means that the object g has the attribute m. We define two derivation operators; ↑ : 2G → 2M maps a subset of objects to a subset of attributes, and its reverse ↓ : 2M → 2G maps attributes to objects. For A ⊆ G,B ⊆M ,
A↑ := {m ∈M | (g,m) ∈ I (∀g ∈ A)} (1) B↓ := {g ∈ G | (g,m) ∈ I (∀m ∈ B)} (2)
A↑ ⊆ M is the set of attributes common to all objects in A, whereas B↓ ⊆ G is the set of objects that possess all the attributes in B. It can be shown that A ⊆ B↓ ⇔ B ⊆ A↑, which is a structure-preserving (order-reversing) correspondence between ordered sets known as a Galois connection (Davey and Priestley, 2002).
A formal concept of the context (G,M, I) is defined as a pair (A,B) ∈ 2G × 2M where both A↑ = B and B↓ = A hold. A and B are considered the extent and intent, respectively, of the formal concept (A,B). The compositions of two derivation operators ↑↓ : 2G → 2G and ↓↑ : 2M → 2M are closure operators (Davey and Priestley, 2002), with a formal concept defined as the fixed point of these operations. If a formal context is represented as a binary matrix, it corresponds to a maximal rectangular (submatrix) with
all ones in its entries when the rows and columns are appropriately reordered.
A formal concept can also be equated with a maximal biclique, i.e., a complete subgraph of a bipartite graph (Chiaselotti et al., 2015). All elements of A and B are completely connected within that subgraph.

heading: 2.2 Rational and benefit of using FCA
text: A word co-occurrence matrix, used as input data to learn word embeddings, is constructed by counting the frequency of a target-context word pair that cooccurs in the neighborhood. By regarding target words as objects and context words as attributes, we can express this co-occurrence as a binary relation. Thus, we can treat a co-occurrence matrix as a formal context.
FCA is effective in analyzing co-occurrence matrices for three reasons. First, it can characterize a local structure within the matrix. Second, formal concepts can capture relations between more than three words, which cannot be represented by individual pairwise relationships, yielding a richer analysis of the structure. Third, we can define (partial) order relation between formal concepts. A semantic relationship such as hypernymy can be formalized by such an order relation. We further demonstrate the function of FCA in Section 3.
To apply the crisp (binary) FCA to a real-valued co-occurrence matrix, we tested two approaches. First, we simply binarized the matrix values by thresholds, with a varying threshold method deployed to flexibly locate formal concepts (Section 4). Second, we extended the crisp FCA to an FCA built on fuzzy logic (Section 5).

heading: 3 Demonstration using synthetic data

heading: 3.1 Artificial toy corpus
text: We examined how FCA handles a word cooccurrence matrix using a toy corpus. We demonstrated that formal concepts capture semantic categories emerging from word usage in the corpus and introduced a concept lattice of FCA to illustrate the hierarchical structure of concepts.
The demonstration contains 1) a corpus of 24 synthetic sentences with 17 words (Appendix A), 2) a co-occurrence matrix obtained from the corpus, and 3) word vectors acquired from the matrix (Fig. 1). The corpus is designed to replicate a geometric formation of the analogy relation. Specifically, we targeted eight words—king, queen, man,
woman, and their plurals—so that their vectors formed a parallelepiped. The sentences were expressed analogously: E.g., ‘king (queen) live in palace”, whereas “man (woman) live in house”. The co-occurrence matrix X ∈ {0, 1}17×17 is bi-
nary, where Xij = 1 if two words co-occur in a sentence and Xij = 0 otherwise. Each row of this matrix represents a word vector. Projected on the 3-dimensional space, the eight word vectors form a parallelepiped (Fig. 2).

heading: 3.2 Detecting formal concepts
text: We now apply FCA to the matrix X . Although formal concepts can be determined by applying the closure operator ↑↓, a simplified method is to find a rectangular in the matrix. For example, the submatrix of rows i ∈ {1, 3, 4, 7} and columns j ∈ {1} represents a formal concept, as all its entries are 1s and no other rectangular matrix contains it. This concept represents a pair of the extent {king, queen, kings, queens} and the intent {palace}, interpreted as "royal."
There are a total of 28 formal concepts in this matrix (see Appendix B for the list and notation). They are classified into five types, including two trivial ones wherein one element is empty. Examples of the three non-trivial types include the following:
f1 := ({king,man, kings,men}, {tie}) (3) e1 := ({king,man}, {tie, alone}) (4) v1 := ({king}, {tie, palace, alone}) (5)
To see hierarchical relations between formal concepts, we first define the order relation. Let B(G,M, I) be the set of all concepts of (G,M, I). Given (A1, B1), (A2, B2) ∈ B(G,M, I),
(A1, B1) ≤ (A2, B2) def⇐=⇒ A1 ⊆ A2 ⇔ B1 ⊇ B2 (6) Thus, if the extent A1 is contained by the extent A2, then the formal concept (A1, B1) is less than or equal to (A2, B2). Owing to the Galois connection, A1 ⊆ A2 holds if and only if B1 ⊇ B2. Then, ⟨B(G,M, I) : ≤⟩ is a complete lattice known as a concept lattice, a nonempty ordered set where a join and a meet exist for all elements and subsets. Fig. 3 visualizes all ordered relations between the formal concepts identified in the matrix X . We
observe that the lattice of formal concepts (Fig. 3) corresponds to the parallelepiped (Fig. 2). This suggests that geometric relations between word vectors reflect the hierarchical structure latent in the word co-occurrence matrix.

heading: 3.3 Three implications of FCA
text: First, FCA allows us to easily interpret the identified formal concepts. For example, f1 should be labeled as masculine from its extent {king, kings,
man, men}, whereas f6, with the extent {queen, queens, woman, women}, must be labeled as feminine. The other f -type concepts can be labeled as royal, common, singular and plural. Thus, formal concepts coincide with semantic categories.
Second, v1 (king) can be seen as the intersection of three others—f1, f3, f5— analogous to a vertex included in three faces. Semantically, king is something royal, masculine, and singular. This relation can be algebraically formulated as v1 = f1∧f3∧f5 where ∧ is a meet operation.
Third, pairs of opposing faces in the parallelepiped form complementary concepts such as masculine vs. feminine. Mathematically, we can construct a formal concept algebra by defining additional operations as axioms (Wille, 2004). Using this algebra, the formal concept of masculine can be demonstrated to complement that of feminine; ¬f1 = f6 where ¬ is a negation. The observation that king v1 = f1 ∧ f3 ∧ f5 and queen v5 = f6 ∧ f3 ∧ f5 share f3 and f5 explains the phenomenon that both synonyms and antonyms appear close to each other in the vector space (Turney and Pantel, 2010).
In summary, the co-occurrence matrix exhibits the geometrical and algebraic structures formed by interpretable formal concepts.

heading: 4 Experiment 1: FCA by binarization
text: We now demonstrate that formal concepts can be defined on actual word co-occurrence data and correspond to both semantic and syntactic categories.

heading: 4.1 Algorithm to identify formal concepts
text: We designed a novel algorithm to locate formal concepts through the conversion of two derivation operators (Eq. 1 and 2). The corresponding pseudoalgorithm is shown in Algorithm 1. Given a cooccurrence matrix X and set of target words S as a seed, the algorithm returns a formal concept (S↑↓, S↑), which is a pair of two subsets of the vocabulary. Here, S↑↓ is the closed set of S.
The first derivation operator ↑ must identify context words that co-occur with all target words in S. In other words, a context word is selected when it has all entry values exceeding the threshold t for the target words in S. Equivalently, any entry value that the seed words have with the context word should not be less than t, meaning that their minimum must be greater than or equal to t. As indicated in Line 3, the algorithm finds
Algorithm 1 Varying Threshold Method
Input: X ∈ RN×N , S := {wi}i∈IS , k ∈ N Output: FC := (S↑↓, S↑), t ∈ R
1: function FINDFORMALCONCEPT(S, k) 2: for j ← 1 to N do 3: mj ← mini∈IS Xij 4: end for 5: Sort [mj ] in descending order← [mp(j)] 6: JS↑ ← {p(j)}j≤k 7: S↑ ← {wj}j∈JS↑ 8: t← mp(k) 9: IS↑↓ ← ∅
10: for i← 1 to N do 11: µi ← minj∈JS↑ Xij 12: if µi ≥ t then 13: IS↑↓ ← IS↑↓ ∪ {i} 14: end if 15: end for 16: S↑↓ ← {wi}i∈IS↑↓ 17: return (S↑↓, S↑), t 18: end function
the minimum value that the seed words (in rows ∀i ∈ IS) have against a certain context word (in a column j ∈ {1, . . . , N}), sorts them in descending order (Line 5), and selects the first k context words (columns) S↑ (Line 6). The threshold is automatically determined as the kth largest minimum value (Line 8). Next, an inverse operation executes. Given S↑, the algorithm finds a minimum value over the context words S↑ (JS↑ in the column index) against a target word in a row i (Line 11) and selects the target words (rows IS↑↓) with minimum values exceeding the threshold (Line 13), which form S↑↓. IS↑↓ and JS↑ are subsets of rows and columns corresponding to S↑↓ and S↑, respectively. t is the determined threshold. The algorithm ensures that a submatrix (Xij)i∈IS↑↓, j∈JS↑ satisfies:
Xij ≥ t (i ∈ IS↑↓, j ∈ JS↑) (7) Xij < t (∀j /∈ JS↑, ∃i ∈ IS↑↓) (8) Xij < t (∀i /∈ IS↑↓,∃j ∈ JS↑) (9)
Note that the submatrix of IS↑↓ × JS↑ is discriminated from its neighbouring area. Its inner region has higher values than t (Eq. 7), whereas each of its exterior rows and columns horizontally (Eq. 8) and vertically (Eq. 9) adjacent to the submatrix contains at least one cell below the threshold. In other words, the higher entry values discriminate
the submatrix of a formal concept from its neighbors, forming a local plateau-like structure that is not necessarily captured by the cosine similarity.

heading: 4.2 Category completion test
text: The experiment was conducted to verify that the formal concepts identified from the co-occurrence matrix coincide with interpretable categories.
Test set We adopted two existing test sets from Lindh-Knuutila and Honkela (2015) containing semantic categories: the Battig set (Bullinaria and Levy, 2012), comprising 53 categories with 10 words for each, and BLESS (Baroni and Lenci, 2011), which contains 17 categories with 5-17 words for each. We also compiled two additional sets: Series and Syntactic. The tested categories are listed in Appendix C.
Procedure For each category, we systematically furnished the algorithm with all possible word pairs as seeds derived from the category’s word set. Next, we identified the optimal seed that yields the most extensive set of accurately classified words. We then assessed how effectively the algorithm retrieves the correct words from the optimal seed for the given category (Precision, Recall). Because the word sets are not necessarily exhaustive, we also regarded those missed words as correct, based on our human judgement (Extended precision)1.
Baseline We used a similarity-based approach as a baseline. Specifically, we applied the k-nearest neighbor algorithm with cosine similarity. To ensure a fair comparison, we utilized the identical optimal seeds derived by the FCA method and found the nearest vectors to their mean vectors.
Data The co-occurrence matrix was constructed from the English Wikipedia dump (20171001)2 (2.9B tokens), counted with a window of 10. We adopted PPMI (positive point-wise mutual information) as it yields the best results in the semantic task (Bullinaria and Levy, 2012). To keep the matrix size manageable, we limited the vocabulary to the 10K most frequent words.

heading: 4.3 Results
text: Qualitative results Table 1 presents the output samples produced by the algorithm. When given {large, huge} as a seed, the algorithm returned
1The annotation was done by one of the authors, who is non-native but has educational experience in the U.S.
2CC BY-SA 3.0; https://dumps.wikimedia.org/legal.html
{large, huge, enormous, vast} as the extent and {sums, amounts, quantities} as the intent, which constitutes a formal concept. All PPMI values within this concept exceeded 3.95. This formal concept can be labeled as "largeness" or Adjective of size, which implies that it is indeed interpretable. Interestingly, another formal concept consisting of {large, small} arises from the different seed instead. Similar results held for other seeds.
Quantitative results Table 2 shows that 61.5– 84.3% of the identified extent words matched the category labels in the test sets (Extended precision). Furthermore, 56.3–76.8% of the words in the test sets were retrieved by the algorithm (Recall). Semantic categories in Battig, BLESS, and Series were more effectively captured by formal concepts than syntactic categories. We also observed that homogeneous categories (e.g., Country) frequently formed formal concepts. With the exception of the Extended Precision metric for the Syntactic test set, our proposed method consistently achieved higher scores compared to the baseline.
The use of optimal seeds in the evaluation is justified because the objective is to measure the extent to which a mathematically identified formal concept best matches categories provided in the test set. Other non-optimal seeds return different formal concepts, which indicate the heterogeneity of human-made categories in the test set. See Appendix E for performance spread and a further discussion on the roles of seed words.

heading: 4.4 Analysis
text: The results suggest that formal concepts overlap with interpretable categories, which are defined as a set of words that human can descriptively label. Furthermore, the FCA method exhibited a notable advantage over the cosine similaritybased approach in concept retrieval. This is because the latter broadly identifies related words, whereas the former delves into specifying the underlying context. For example, given the seed words {church, chapel}, FCA additionally retrieves {cathedral, shrine}, emphasizing the context of "religious buildings." In contrast, the cosine method returns {cathedral, catholic} as output, failing to extract the feature of "buildings."
This advantage of FCA stems from its ability to locate mathematical structures within the matrix. Higher PPMI values discriminate the submatrix of a formal concept from its neighbors, forming a
local plateau-like structure that is not necessarily captured by the cosine similarity (Eq. 7–9). This insight offers a use case for the proposed algorithm.
Disambiguating polysemy A target word can participate in multiple formal concepts. By inputting seed words with different associations, we found that polysemous words such as tie and spring have multiple formal concepts, as shown in Table 3. We observed that separate formal concepts (e.g., clothing, match, fasten) may contain the same word (e.g., tie) in their extents. Three separate plateaus may share the same row as visualized in Fig. 4.
Arora et al. (2018) discovered that the embeddings of polysemous words can be decomposed as linear combinations of sense vectors. Our finding suggests that these vectors reflect separate formal concepts, and that the embeddings inherit the inner structure of the co-occurrence matrix.
Measuring a similarity in subspace The proposed algorithm generates a byproduct that can be used to investigate the relationship between multiple vectors (the rows of the matrix) in a subspace.
Reusing Lines 3–5 in Algorithm 1, we can determine whether target words in a seed share certain context words in limited dimensions and are semantically related in the shared context.
Specifically, we propose the subspace similarity ϕ(S) defined as
ϕ(S) := 1
k
k∑
i=1
mp(i) (10)
for a group of words S = {wi}i∈IS , where mj := mini∈IS Xij , p(i) is a permuted index in descending order and k is a hyperparameter for the scope of subspace. The notation is the same as in Algorithm 1. The subspace similarity is the mean of the thresholds t determined over different parameters values up to k. Fig. 5 shows the computed values of the subspace similarity for several word groups. Semantically related groups show significantly higher values than the randomly chosen word group. These results indicate that semantically related groups share certain context words locally, even if their cosine similarities are
low. Generally, randomly chosen vectors in highdimensional space tend to be orthogonal, which implies a low chance of detecting correlations in any dimension. In contrast, a higher subspace similarity should suggest that a certain structure can be defined more than incidentally.

heading: 5 Experiment 2: Applying Fuzzy FCA

heading: 5.1 Fuzzification of FCA
text: Our second application of FCA to a real-valued matrix involves the fuzzification of the crisp FCA by incorporating fuzzy set theory (Belohlavek, 2007). A fuzzy set formalizes an ambiguous set, such as "a set of tall people," by assigning a degree of membership to each element. In Appendix D, we give the definition of a fuzzy formal concept and show that it is equivalent to a rank-one submatrix under our proposed specification. Thus, the problem of finding fuzzy formal concepts can be regarded as that of identifying nonnegative rank-one submatrices in a PPMI matrix.
Because it is NP-hard to exactly decompose a matrix into nonnegative factors (Vavasis, 2010), we
obtained an approximation by deploying nonnegative matrix factorization (NMF; Lee and Seung, 1999), as its L1 regularization is considered effective in making them as sparse as possible. We controlled the sparseness so that the decomposed submatrices became disjoint. NMF decomposes X ∈ Rm×n+ into two matrices W ∈ Rm×r+ and H ∈ Rn×r+ so that X = WHT = ∑r k=1wkh T k , where wk and hk are the kth columns of W and H , respectively. The outer product wkhTk is of rank one and preferably sparse, thereby approximating a fuzzy formal concept. The loss function is Lα(W,H) = 12∥X −WHT ∥2F + α ( n∥W∥1 +
m∥H∥1 ) . We recursively applied NMF 3 over three rounds—first to the PPMI matrix as in Section 4, then twice to the positive residual matrices resulting from decomposition—factorizing into r = 300 components each round. Parameters for the L1 norms were set to α = 5, 3, 1 × 10−4 for each round.

heading: 5.2 Results
text: We manually labeled 900 rank-one submatrices by reviewing the words corresponding to the largest entries in wk and hk (see Appendix F.1 for details). We then classified the submatrices among four categories to assess how well the labels describe the words in each formal concept4 (Table 4). Out of 900 acquired formal concepts, 95.7% were labeled descriptively or partially descriptively, or at least consisted of meaningfully related words.

heading: 5.3 Analysis
text: We found that Fuzzy FCA reveals the same formal concepts as the crisp FCA. For example, all categories listed in Table 2 also appear as rank-one submatrices. Of the 108 formal concepts identified in Experiment 1, 89 formal concepts (82.4%)
3NMF from Scikit-learn library: BSD license. 4The same as the footnote 1.
were included by those found by the fuzzy method (Supplemental statistics in Appendix F.3). In fact, Fuzzy FCA detected more eligible words (e.g. immense, massive for Largeness, shrine for Religious Buildings). This observation demonstrates the robustness of FCA, as well as the correlation between the two methods.
Another interesting finding is that two types of rank-one submatrices were discovered: a clique type with identical rows and columns, and a biclique type with different rows and columns. An example of the latter is ({explain, describe, discuss, ...}, {beliefs, concepts, ideas,...}), which represents a verb phrase for an act of communication.

heading: 6 Discussion

heading: 6.1 Why do formal concepts correspond to interpretable categories ?
text: As noted in Section 2, a formal concept is equivalent to a biclique, which means that the words in it are densely connected. A group of words forms a dense community if the words are used repeatedly together. Furthermore, if the same latent state always emits the same set of words, then those words are repeatedly counted as co-occurrence, thereby forming a formal concept. The random walk model of Arora et al. (2016) captures the same mechanism to generate linearly structured embeddings.
However, a latent state is not necessarily limited to a topic, i.e., a state based on thematic proximity. As revealed in Section 4, formal concepts may reflect functional proximity, e.g. the comparative. Furthermore, we observed phrasal proximity, as in a verb phrase. Thus, a broad range of semantic and syntactic patterns of word usage can be captured as a formal concept.
These results open questions about the potential relationship between formal concepts and human
cognition (e.g., Wu et al., 2022 showed that fMRI patterns contain information to solve analogical reasoning), which may be the subject of future studies of semantic cognition. Our approach may provide a quantitative method to address these questions.

heading: 6.2 How can formal concepts be fully captured ?
text: We designed two methods that apply FCA to a real-valued matrix to detect interpretable formal concepts, although we do not yet have a theory to assess and relate the two methods.
In general, the challenge of FCA in applied studies is scalability stemming from computational complexity, which must be addressed when increasing the size of a co-occurrence matrix. Another challenge is posed by heterogeneous data from large corpora. Specifically, we observed that interpretable formal concepts are detected at different threshold levels (Section 4) and by layered factorization (Section 5). The latent structures at different scales indicate that multiple formal contexts coexist in the matrix as if they were superposed, and they were probably generated separately. Thus, the rank-one submatrices may be disjoint, superposed, or overlapping. To extract such latent structures more precisely, the algorithm must depend upon the modeling of generative processes, which is also a topic for a future study.

heading: 6.3 What do embeddings represent after all?
text: Recall that formal concepts defined as rank-one submatrices appear as components of matrix factorization X = WHT (Section 5). While a column of W corresponds to a fuzzy set that constitutes each formal concept, a row of W is used as a word embedding. Thus, a value in each dimension of the embedding can be seen as the "coordinate" of the corresponding formal concept. The other matrix H is considered to encode attributes. The embeddings, acquired by matrix factorization or implicit factorization (Levy and Goldberg, 2014b), must inherit the structures of formal concepts, as the factor matrices can be mutually transformed.

heading: 6.4 Are FCA methods beneficial in practice?
text: The FCA method exhibited an advantage over the cosine similarity-based approach in the category completion task (Section 4). Although both methods can capture a similarity between words, a fundamental distinction lies in the subspace where
these similarities are assessed. While cosine similarity utilizes the entire vector space and treats vectors as static entities, the FCA method dynamically narrows the subspace based on a given set of words. It identifies subvectors with significantly high occurrences, a task that cosine-based methods cannot perform. This merit makes FCA beneficial in various tasks such as polysemy disambiguation and concept completion (e.g., Shani et al., 2023).
Another potential benefit of FCA is its use as an analytical tool for contextualized embeddings. Dar et al., 2023 reveals that the inner representations of GPT-2 can be interpreted by projecting vectors into the vocabulary space. They report actual pairs of words processed in layers of GPT-2, some of which seem to be similar to the formal concepts identified in Experiment 2. Thus, there is a possibility that the contextualization process can be interpreted as certain algebraic operations on formal concepts, though this is still speculative.

heading: 7 Related studies
text: Several studies demonstrated that sparse embeddings are interpretable. Murphy et al. (2012) and Biggs et al. (2008) applied nonnegative matrix factorization with a sparsity constraint to worddocument co-occurrence data and discovered topics. Other studies (Faruqui et al., 2015; Park et al., 2017; Jang and Myaeng, 2017) investigated word embeddings to restore interpretability by using sparsity. We mathematically formalized the latent structure in the word co-occurrence matrix, which prior studies might have empirically detected.
FCA has been applied in linguistics (Priss, 2005), primarily for ontology. Cimiano et al. (2005) applied FCA for the automatic acquisition of taxonomies from a corpus. Moraes and Lima (2012) built a semantic structure by setting the S-V-C tuples of the annotated corpora as a formal context. Berend et al. (2018) used FCA by binarizing sparse word embedding for hypernymy discovery. In contrast to these studies, we deployed FCA to explore the structure of the matrix itself, which revealed the underlying structure of word-word co-occurrence matrices.
Gastaldi (2021) delved into the underlying mechanism of word embeddings from a linguisticphilosophical perspective and pointed out simultaneous codetermination or bi-duality between terms and contexts as a significant feature of language, which we believe to have successfully formalized
via FCA. Our mathematical approach to interpreting co-occurrence data may shed light on the structure of language, as Bradley et al. (2023) frames language in category theory.

heading: 8 Summary
text: This study establishes a mathematical characterization of semantic relations represented as geometrical formations in a vector space, employing FCA to investigate a word co-occurrence matrix. Our experiments demonstrate that identified formal concepts align with interpretable categories. Using synthetic data, we also illustrated the emergence of hierarchical structures from word usage. Subsequent challenges include theoretical sophistication in applying FCA, exploring generative modeling, and delving into cognitive inquiries.

heading: 9 Limitations and risks
text: Our study is inherently exploratory, with the aim of communicating critical insights in a timely manner before exhaustively diving into a comprehensive analysis. Consequently, a more thorough investigation and nuanced analysis are deferred to future work, acknowledging that the current study serves as a preliminary exploration that lays the foundation for deeper scrutiny. One direction is to identify the entire formal concepts and construct a concept lattice to identify all the possible semantic relations within a vocabulary. Another direction is to investigate more thoroughly the relationship between formal concepts and linguistic concepts to quantify how far the distributional hypothesis holds.
Another limitation of this work stems from the reliance on a singular dataset for our analysis. Although our findings reveal compelling patterns within the chosen dataset, generalizability across diverse data sets remains an unexplored avenue. We anticipate similar trends in other data sets, but a comprehensive cross-validation across various sources is pending. Future research efforts should extend our methodology to encompass a wider spectrum of data sets, ensuring the robustness and applicability of our observed trends across different contexts.
The study constitutes a fundamental analysis aimed at identifying mathematical properties within linguistic statistical data, thus enhancing interpretability. Notably, no significant material risks were identified throughout the investigation and will not be seen due to the nature of the analytical
approach employed.

heading: Acknowledgements
text: This work was supported by the JSPS KAKENHI grant number JP23H0369, JP22K19790, JP24KJ1202, the JST PRESTO grant number JPMJPR20C9, and the JST CREST JPMJCR23P4, Japan.

heading: A Toy corpus
text: The corpus contains 24 synthetic sentences shown in Table 5. The target words—king, queen, man, woman and their plurals—are subjects of the sentences. Each of the eight words appears with three verbs—live-in, wear, eat—once for each. The remaining six words—palace, house, tie, dress, alone, together—discriminate the subject words so that they are in the analogical relations of three dimensions.

heading: B List of formal concepts
text: There are 28 formal concepts in the co-occurrence matrix derived from the toy corpus.
Suppose that the set of objects (target words) and the set of attributes (context words) be G,M respectively, defined as:
G = {king,man, queen, queens, kings,men, queens, women}
M = {tie, dress, palace, house,
alone, together}
Then, all the formal concepts are identified as below:
T = (G, ∅) f1 = ({king,man, kings,men}, {tie}) f2 = ({man,woman,men,women}, {house}) f3 = ({king, queen,man,woman}, {alone}) f4 = ({kings, queens,men,women}, {together}) f5 = ({king, queen, kings, queens}, {palace}) f6 = ({queen,woman, queens, women}, {dress}) e1 = ({king,man}, {tie, alone}) e2 = ({king, kings}, {tie, palace}) e3 = ({man,men}, {tie, house}) e4 = ({kings,men}, {tie, together}) e5 = ({king, queen}, {palace, alone}) e6 = ({man,woman}, {house, alone}) e7 = ({kings, queens}, {palace, together}) e8 = ({men,women}, {house, together}) e9 = ({queen,woman}, {dress, alone}) e10 = ({queen, queens}, {palace, dress}) e11 = ({woman,women}, {house, dress}) e12 = ({queens, women}, {dress, togther}) v1 = ({king}, {tie, palace, alone}) v2 = ({man}, {tie, house, alone}) v3 = ({kings}, {tie, palace, together}) v4 = ({men}, {tie, house, together}) v5 = ({queen}, {dress, palace, alone}) v6 = ({woman}, {dress, house, alone}) v7 = ({queens}, {dress, palace, together}) v8 = ({women}, {dress, house, together}) B = (∅,M)

heading: C Category completion test
text: We used the four test sets for the category completion test: Battig, BLESS, Series and Syntactic.
Battig test (Bullinaria and Levy, 2012), originated from Battig and Montague (1969), contains 53 categories with 10 words for each category, of which we used 44 categories in the experiments, since the others have less than two words in our vocabulary of the co-occurrence matrix.
BLESS (Baroni and Lenci, 2011) contains 17 categories with 5-17, of which we used 12 categories for the same reason.
Both of Series and Syntactic are developed by the authors to supplement Battig and BLESS, which contain only common nouns. Series is hinted by Hashimoto et al. (2016) that proposed the series completion task (penny:nickel:dime:?) for word embeddings. Syntactic is motivated by our early finding that comparative adjectives such as quicker, faster, ... emerge as a salient formal concept with a high threshold in the binary FCA experiment. In both test sets, each category consists of 4 to 5 words, which are manually selected by one of the authors. In the development process, we partly use AI assistance5 to generate a list of candidates for a category and its word set, by prompting with an example "Direction: north, east, south, west".
Examples of a category in each test set are shown below (Table 6)
We used only the categories that contain more than or equal to three words in our vocabulary, which are listed in Table 7.

heading: D Fuzzification of FCA
text: Formally, a fuzzy set A is a function A : X → L where X is a ground set and L = [0, 1], which assigns the value to each member of X . A subsumption relation A ⊆ B holds if and only if A(x) ≤ B(x) for all x ∈ X . In Fuzzy FCA, a formal concept is K := (G,M, I, L). We consider two fuzzy sets A ∈ LG, B ∈ LM as objects and
5https://chat.openai.com/
attributes and a fuzzy relation I ∈ LG×M . Mathematically, L can be generalized to a residuated lattice that includes [0, 1] as its special case. Similar to the crisp setting, two fuzzy derivation operators ↑ : LG → LM and ↓ : LM → LG are defined as follows: For all m ∈M and g ∈ G,
A ↑ (m) := ∧
g∈G
( A(g)→ I(g,m) ) ∈ L (11)
B ↓ (g) := ∧
m∈M
( B(m)→ I(g,m) ) ∈ L (12)
Note that A↑ ∈ LM , B↓ ∈ LG and (→) : L × L→ L , which is a binary operation defined on L. In plain English, the degree to which an object g belongs to the fuzzy set A should imply the level of co-occurrence between g and an attribute m, which retrospectively should determine the degree to which the attribute m belongs to another fuzzy set A↑. Then, fuzzy formal concepts are defined as a pair of fuzzy sets (A,B) where A↑ = B and B↓ = A hold as in the crisp FCA.
We need to specify operations such as (→) to numerically compute them. Three specifications, named as Lukasiewicz, Gödel and Goguen, have already been proposed (Belohlavek and Vychodil, 2012), but instead we propose our own specification tailored to the analysis of a word co-occurrence matrix.
a→ b := { b/a if a > 0
⊤ if a = 0 (13)
where ⊤ is the greatest element in L. This specification is a slight modification of the one proposed by Goguen. The meet ∧ is numerically calculated as a minimum.
Our specification is equivalent to defining (A,A↑) and (B↓, B) as a rank-one submatrix. Recall that the fuzzy set A ∈ LG assigns a value x ∈ L to the element g. Similarly, the fuzzy set A↑ ∈ LM assigns a value y ∈ L to the element m. Thus, the specification in Eq. (13) ensures that y = xy/x for x > 0. This means that nonnegative entries in both fuzzy sets A,A↑ constitute a rank-one submatrix.

heading: E Role of seed words and performance spread
text: In Algorithm 1 (Section 4), seed words are required as input to the algorithm since a formal concept is detected as a closed set containing those seed words. A closed set is the fixed point of a closure operator. In this algorithm, any set of words can be a seed. If seed words are randomly chosen, then the algorithm will find and return a formal concept in an unsupervised way. Different sets of words return the same formal concept if all of the used words belong to the same closed set, and derive different formal concepts otherwise due to the mathematical property of a closure operator and closure system.
In the category completion test in Experiment 1, all possible pairs from the word list of the same category are used as seeds (2 words) to derive a formal concept. Therefore, different formal concepts can be identified by a chosen pair of words from the test set.
Table 8 shows the statistics of the distribution of recalls calculated for all combinations of two words, reflecting the degree to which the FCA retrieved elements from the test set based on seed choices. To verify, cross-reference the numbers in the Max column with the corresponding recalls reported in Table 2 where the optimal seed pairs were selected. In cases where a word set within a specific category in the test set comprises 10 words, there exist 10C2 = 45 possible pairs. We computed the minimum, maximum, and median values for each category, subsequently averaging them between categories for each test set and the entire dataset.
These statistics suggest that employing a "right seed" (optimal pair) results in the formal concept covering 56.3% (Syntactic) to 76.8% (Series). On the contrary, the use of a different seed may yield a distinct formal concept. This divergence can be attributed to the non-cohesive nature of word groups within the test set.
For instance, the "Occupation or profession" category of Battig comprises words such as doctor, lawyer, teacher, engineer, professor, carpenter, salesman, nurse, psychologist (with one word omitted due to limited vocabularies in the matrix). Notably, the FCA found that the maximum formal concept within this category is only four words: lawyer, nurse, psychologist, teacher, which seem to represent the "profession" part of the category.

heading: F Decomposition by NMF
text: F.1 Decomposed submatrices by NMF
We applied NMF recursively in three rounds. In the first round, we decomposed the PPMI matrix as in X0 ≈W1HT1 into 300 components (α = 0.0005). In the second round, we applied NMF to the positive residual matrix after the first decomposition: X1 := max(X0−W1HT1 , 0) as decomposed as in X1 ≈ W2HT2 (α = 0.0003). In the third round, the residual matrix X2 := max(X1 −W2HT2 , 0) was decomposed into X2 ≈W3HT3 (α = 0.0001). Note that each component (rank-one matrix) wkhTk was forced to be sparse by L1 regularization. Thus, their nonnegative rows and columns make a nonnegative rank-one submatrix, which we regard as a fuzzy formal concept.
The components derived in the first round were indexed from 1 to 300. Similarly, those in the second round were indexed from 301 to 600, and ones in the third round were indexed from 601 to 900. We ordered each component according to the Frobenius norm within each round. Therefore, the smaller ID number implies that the submatrix has a greater norm in each round.
Samples of the components are presented in Table 12. The class was evaluated by one of the authors according to the definition given in Appendix F.2. The author also labeled a category from the words that comprise the submatrix wkhTk . More specifically, for each vector wk and hk, we picked 20 words that correspond to the largest elements in the vectors, respectively. In Table 12, the only four top words are presented for both wk as extents and hk as intents. For ease of visibility, categories were labeled with more general expressions, although they could be labeled with more focused category names.
Table 9 shows a supplemental analysis of the type of relatedness between words participating in each submatrix.
F.2 Types of qualitative classes The set of words corresponding to the largest dimensions within each component is classified into four qualitative classes, as in the below definition (Table 11), following Lindh-Knuutila and Honkela (2015) . These classes indicate how well an identified formal concept (a rank-one matrix) is interpretable as a category.
F.3 Overlap of two FCA methods In addition, we performed an analysis of set overlap at the word level. For each of the 89 groups, we calculated the set overlap using the Jaccard index, which is defined as the number of words in the intersection divided by the number in the union. The results are presented in Table 10 as percentages.

heading: Future Work
text: Consequently, a more thorough investigation and nuanced analysis are deferred to future work, acknowledging that the current study serves as a preliminary exploration that lays the foundation for deeper scrutiny. One direction is to identify the entire formal concepts and construct a concept lattice to identify all the possible semantic relations within a vocabulary. Another direction is to investigate more thoroughly the relationship between formal concepts and linguistic concepts to quantify how far the distributional hypothesis holds. Another limitation of this work stems from the reliance on a singular dataset for our analysis. Although our findings reveal compelling patterns within the chosen dataset, generalizability across diverse data sets remains an unexplored avenue. We anticipate similar trends in other data sets, but a comprehensive cross-validation across various sources is pending. Future research efforts should extend our methodology to encompass a wider spectrum of data sets, ensuring the robustness and applicability of our observed trends across different contexts. The study constitutes a fundamental analysis aimed at identifying mathematical properties within linguistic statistical data, thus enhancing interpretability. Notably, no significant material risks were identified throughout the investigation and will not be seen due to the nature of the analytical
approach employed