Venue: NAACL 2024

# Subspace Representations for Soft Set Operations and Sentence Similarities

Yoichi Ishibashi\({}^{1,2}\)  Sho Yokoi\({}^{3,4}\) Katsuhito Sudoh\({}^{1}\) Satoshi Nakamura\({}^{1}\)

\({}^{1}\) Nara Institute of Science and Technology  \({}^{2}\) Kyoto University

\({}^{3}\) Tohoku University   \({}^{4}\) RIKEN

{ishibashi.yoichi.ir3, sudoh, s-nakamura}@is.naist.jp yokoi@tohoku.ac.jp

Work done while at Nara Institute of Science and Technology.Our code is publicly available at [https://github.com/yoichi1484/subspace](https://github.com/yoichi1484/subspace)

###### Abstract

In the field of natural language processing (NLP), continuous vector representations are crucial for capturing the semantic meanings of individual words. Yet, when it comes to the representations of _sets_ of words, the conventional vector-based approaches often struggle with expressiveness and lack the essential set operations such as union, intersection, and complement. Inspired by quantum logic, we realize the representation of word sets and corresponding set operations within pre-trained word embedding spaces. By grounding our approach in the linear subspaces, we enable efficient computation of various set operations and facilitate the _soft_ computation of membership functions within continuous spaces. Moreover, we allow for the computation of the F-score directly within word vectors, thereby establishing a direct link to the assessment of sentence similarity. In experiments with widely-used pre-trained embeddings and benchmarks, we show that our subspace-based set operations consistently outperform vector-based ones in both sentence similarity and set retrieval tasks. 1

Footnote 1: Our code is publicly available at [https://github.com/yoichi1484/subspace](https://github.com/yoichi1484/subspace)

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 3512–3524 June 16-21, 2024 ©2024 Association for Computational Linguistics

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 3512–3524 June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

Embedding-based word representations have become fundamental in the field of natural language processing (NLP). Models like word2vec Mikolov et al. (2013) and GloVe Pennington et al. (2014), along with recent Transformer-based architectures Vaswani et al. (2017); Devlin et al. (2019), have underscored the significance of embeddings in capturing the complexities of linguistic semantics.

The importance of representing collections of words is pivotal in understanding concepts and relationships within language contexts Zaheer et al. (2017); Zhelezniak et al. (2019). For instance, while words like "apple" and "orange" each carry their distinct meanings, together they represent the broader concept of fruits. Another example of important application is a sentence representation Zaheer et al. (2017). The set of words in a sentence captures the overall meaning, allowing for computations such as text similarity Agirre et al. (2012).

Against this backdrop, our research recognizes the significance of applying set operations in NLP and explores a new approach. Set operations enable a richer representation of relationships between collections of words, leading to more accurate semantic analysis based on context. For example, employing set operations allows for a clearer understanding of shared semantic features and differences among word groups within a text. This directly benefits tasks like determining semantic similarity and expanding word sets.

In response to these challenges, our study introduces a novel methodology that exploits the principles of _quantum logic_ Birkhoff and Von Neumann (1936), applied within embedding spaces to

Figure 1: Superiority of subspace representations: Our subspace representation (blue) surpasses the traditional vector set representation (gray) in both text similarity and text concept set retrieval tasks.



define set operations. Our proposed framework adopts a subspace-based approach for representing word sets, aiming to maintain the intricate semantic relationships within these sets. We represent a word set as a subspace which is spanned by pre-trained embeddings. Additionally, it adheres to the foundational laws of set theory as delineated in the framework of quantum logic. This compliance ensures that our set operations, such as union, intersection, and complement, are not only mathematically robust but also linguistically meaningful when applied them in pre-trained embedding space.

We first introduce a subspace set representation along with basic operations (\(\cap\), \(\cup\), and \(\in\)). Subsequently, to highlight the usefulness of our proposed framework, we introduce two core set computations: text similarity and set membership. The empirical results consistently point towards the notable superiority of our approach; our straightforward approach of spanning subspaces with pretrained embedding sets enables a rich set representation, and we demonstrated its consistent performance enhancement in downstream tasks (Figure 1). Our research contributions include:

1. The introduction of continuous set representations and a framework for set operations, enabling more effective manipulation of word embedding sets (SS4).
2. We propose _SubspaceBERTScore_, an extension of the embedding set-based text similarity method, BERTScore Zhang et al. (2020). By simply transitioning from a vector set representation to a subspace, and incorporating a subspace-based indicator function, we observe a salient improvement in performance across all text similarity benchmarks (SS5).
3. We apply subspace-based basic operations (\(\cap\), \(\cup\), and \(\in\)) to set expansion task and achive high performance (SS6).

## 2 Preliminaries

To make the following discussion clear, we define several symbols. The sets of tokens in two sentences (\(A\) and \(B\)) are denoted as \(A=\{a_{1},a_{2},\dots\},B=\{b_{1},b_{2},\dots\}\) respectively. The sets of contextualized token vectors are denoted as \(\mathbf{A}=\{\boldsymbol{a}_{1},\boldsymbol{a}_{2},\ldots\},\mathbf{B}=\{ \boldsymbol{b}_{1},\boldsymbol{b}_{2},\ldots\}\), where \(\boldsymbol{a}\) and \(\boldsymbol{b}\) are token vectors generated by the pre-trained embedding model such as BERT. The subspace spanned by \(\mathbf{A}\) is denoted as \(\mathbb{S}_{A}=\mathrm{span}(\boldsymbol{a}_{1},\boldsymbol{a}_{2},\dots)\). Note that the bases of the subspace is orthonormalized.

## 3 Symbolic Set Operations

We first formulate various set operations in a pre-trained embedding space. Among many types of operations for practical NLP applications, this work focuses on _set similarity_:

\[A =\{A,\mathit{boy},\mathit{walks},\mathit{in},\mathit{this}, \mathit{park}\}, \tag{1}\] \[B =\{\mathit{Thek},\mathit{kind},\mathit{runs},\mathit{in},\mathit{the },\mathit{square}\},\] \[\mathrm{Similarity}(A,B),\]

_set membership_ (\(\in\)) and _basic operations_ (\(\cap\), \(\cup\)):

\[\mathit{Color}=\{\mathit{red},\mathit{blue},\mathit{green}, \mathit{orange},\dots\}, \tag{2}\] \[Fruit=\{\mathit{apple},\mathit{orange},\mathit{peach},\dots\},\] \[\mathit{orange}\in\mathit{Color}\cap Fruit.\]

For this purpose, we need following representations on a pre-trained embedding space2:

Footnote 2: These operations do not include some set operations such as cardinality, but are sufficient for expressing the practical forms of sets such as Eq. (1).

An element and a set of elementsThe representations of an element and a set of elements are the most basic ones. To exploit word embeddings, we represent a word (e.g., \(\mathit{orange}\)) as an element and a group of words (e.g., \(\{\mathit{red},\mathit{blue},\mathit{green},\mathit{orange},\dots\}\)) as a word set.

Quantification of set membership (indicator function)_Membership_ denotes a relation in which word \(w\) is an element of set \(A\), i.e., \(w\in A\). We quantify it based on vector representations. Although the membership is typically a binary decision identical to that in a symbolic space, it can also be measured by the degree of closeness in a continuous vector space. Membership can be computed as an indicator function. The indicator function \(\mathbbm{1}_{\mathrm{set}}\) quantifies whether the word \(w\) is included (1) or not (\(0\)) in the set in a discrete manner:

\[\mathbbm{1}_{\mathrm{set}}[w\in A]=\begin{cases}1&\text{if~{}~{}$w\in A$,}\\ 0&\text{if~{}~{}$w\notin A$.}\end{cases} \tag{3}\]

Similarity between discrete symbol setsSet similarity, such as recall and precision, is an essential operation when calculating the similarity of texts. Despite its simplicity, the word overlap-based sentence similarity serves as a remarkably effective approximation and has found widespread 

practical application, as evidenced by numerous studiesBojar et al. (2018); Zhang et al. (2020); Cer et al. (2017); Zhelezniak et al. (2019). They stand out as excellent similarity metrics based on embeddings. BERTScore Zhang et al. (2020), which utilizes embeddings for its computation, is grounded in recall and precision 3. The typical computations for recall (\(R\)) and precision (\(P\)) are as follows4:

Footnote 3: Unlike the symbolic set similarity, which do not consider word order, contextualized embeddings enable the capture of word sequence information.

Footnote 4: For simplicity in explanation, we present \(A\) and \(B\) in Eq. (4) and Eq. (5) as a set of tokens.

\[R=\frac{1}{|A|}\sum_{a_{i}\in A}\mathbbm{1}_{\text{set}}[a_{i} \in B], \tag{4}\] \[P=\frac{1}{|B|}\sum_{b_{i}\in B}\mathbbm{1}_{\text{set}}[b_{i} \in A], \tag{5}\]

Basic set operationsWe need three basic set operations: _intersection_ (\(A~{}\cap~{}B\)), _union_ (\(A~{}\cup~{}B\)), and _complement_ (\(\overline{A}\)). They allow us to represent various sets using such different combinations as _Color_\(\cap~{}Fruit\).

## 4 Subspace-based Set Representations

We propose the representations of a word set and set operations based on _quantum logic_Birkhoff and Von Neumann (1936). They hold advantages of geometric properties in an embedding space, and the set operations are guaranteed to hold for the laws of a set defined in quantum logic.

### Quantum logic

While word embedding represents a word's meaning as a vector in linear space, quantum mechanics similarly represents a quantum state as a vector in linear space. These two intuitively different fields are very close to each other in terms of the representation and the operation of information.

Quantum logic Birkhoff and Von Neumann (1936) theory describes quantum mechanical phenomena. Intuitively, it is a framework for _set operations in a vector space_. In quantum logic, a set of vectors is represented as a linear subspace in a Hilbert space, and such set operations as union, intersection, and complement are defined as operations on subspaces. Quantum logic, which employs a complete orthomodular lattice as its system of truth values, guarantees to hold various set operations, such as De Morgan's laws \(\overline{(A\cap B)}=\overline{A}\cup\overline{B}\) and \(\overline{(A\cup B)}=\overline{A}\cap\overline{B}\), idempotent law: \(A\cap A=A\), and double complement: \(\overline{\overline{A}}=A\).

```
0:\(\{v^{(1)},\dots,v^{(k)}\}\subseteq\mathbb{R}^{1\times d}\): Word embeddings to span subspace \(\mathbb{S}_{A}\)
0:\(\mathbf{S}_{A}\in\mathbb{R}^{r\times d}\): Bases of \(\mathbb{S}_{A}\)
0:\(\mathbf{A}\in\mathbb{R}^{k\times d}\leftarrow\textsc{stack\_rows}(\mathbf{v}^{(1)} ,\dots,\mathbf{v}^{(k)})\)\(\mathbf{S}_{A}\in\mathbb{R}^{r\times d}\leftarrow\textsc{(ortho\_normal}(\mathbf{A}^{ \top}))^{\top}\)\(\triangleright\)Orthonormalize the bases. \(r\) is the rank of \(\mathbf{A}\)return\(\mathbf{S}_{A}\)
```

**Algorithm 1** Computing basis of a subspace

In quantum logic, a set of vectors is represented by a vector, and a set is represented by a vector, and a set is represented by a vector, and a set is represented by a vector. In quantum logic, a set is represented by a vector, and a set is represented by a vector, and a set is represented by a vector, and a set is represented by a vector, and a set is represented by a vector, and a set is represented by a vector, and a set is represented by a vector, and a set is represented by a vector corresponding to its elements. Here we assume an element, i.e., word \(w\), is represented by vector \(\mathbf{v}_{w}\), and a word set is represented by linear subspace \(\mathbb{S}_{A}\subset\mathbb{R}^{n}\) spanned by word vectors:

\[\mathbb{S}_{A}\coloneqq\mathrm{span}(\mathbf{A})\coloneqq\mathrm{span}(\mathbf{a} _{1},\mathbf{a}_{2},\dots). \tag{6}\]

Hereinafter we simply refer to _linear subspace_ as _subspace_. Algorithm 1 is the pseudocode for computing the basis of the subspace.

Basic set operationsThe _complement_ of set \(A\), denoted by \(\overline{A}\), is represented by the orthogonal complement of subspace \(\mathbb{S}_{A}\):

\[\mathbb{S}_{\overline{A}}\coloneqq(\mathbb{S}_{A})^{\perp}=\{\mathbf{v}\mid \exists\mathbf{a}\in\mathbb{S}_{A},\mathbf{v}\cdot\mathbf{a}=0\}. \tag{7}\]

The _union_ of two sets, \(A\) and \(B\), denoted by \(A\cup B\), is represented by the sum space of two 

subspaces, \({\mathbb{S}}_{A}\) and \({\mathbb{S}}_{B}\):

\[{\mathbb{S}}_{A\cup B}\coloneqq{\mathbb{S}}_{A}+{\mathbb{S}}_{B}=\{\boldsymbol{a }+\boldsymbol{b}\,|\,\boldsymbol{a}\!\in\!{\mathbb{S}}_{A},\boldsymbol{b}\! \in\!{\mathbb{S}}_{B}\}. \tag{8}\]

The _intersection_ of two sets, \(A\) and \(B\), denoted by \(A\cap B\), is represented by the intersection of two subspaces, \({\mathbb{S}}_{A}\) and \({\mathbb{S}}_{B}\):

\[{\mathbb{S}}_{A\cap B}\coloneqq{\mathbb{S}}_{A}\cap{\mathbb{S}}_{B}=\{ \boldsymbol{v}\mid\boldsymbol{v}\in{\mathbb{S}}_{A},\boldsymbol{v}\in{ \mathbb{S}}_{B}\}. \tag{9}\]

The basis of the intersection can be computed based on singular value decomposition (SVD). The bases are the vectors shared by the two subspaces.

Hard membershipThe set membership in the embedding space (e.g., \(boy\in Male\)) can be represented by the inclusion of a vector into a subspace (e.g., \(\boldsymbol{v}_{boy}\in{\mathbb{S}}_{Male}\)) and given by the following indicator function:

\[{\mathbbm{1}}_{\mathrm{hard}}(\boldsymbol{v},{\mathbb{S}}_{A})\coloneqq \begin{cases}1&(\boldsymbol{v}\in{\mathbb{S}}_{A}),\\ 0&(\boldsymbol{v}\notin{\mathbb{S}}_{A}).\end{cases} \tag{10}\]

However, this binary decision fails to exploit the geometric properties of the word embedding space regarding semantic similarity. Suppose we quantify the degree of membership of word \(boy\) for word set \(Male\) consisting of many masculine nouns other than \(boy\). Even if \(\boldsymbol{v}_{boy}\) is located very close to \({\mathbb{S}}_{Male}\) due to its semantic similarity to masculine nouns, \({\mathbbm{1}}_{\mathrm{hard}}(\boldsymbol{v}_{boy},{\mathbb{S}}_{Male})\) must return \(0\) because \(\boldsymbol{v}_{boy}\) must not be located _exactly_ on subspace \({\mathbb{S}}_{Male}\)

\begin{table}
\begin{tabular}{c c c} \hline \hline
**Symbolic Set Representation** & \multicolumn{2}{c}{**Subspace-based Set Representation**} \\ \hline \hline Element & & Vector \\ \(king\) & & \(\boldsymbol{v}_{king}\) \\ \hline Set & & Subspace \\ \(Male=\{king,man,\dots\}\) & & \({\mathbb{S}}_{Male}=\operatorname{span}(\boldsymbol{v}_{king},\boldsymbol{v}_{ man},\dots)\) \\ \hline Complement & & Orthogonal complement \\ \(\overline{Male}\) & & \(({\mathbb{S}}_{Male})^{\perp}\) \\ \hline Union & & Sum space \\ \(Male\cup Female\) & & \({\mathbb{S}}_{Male}+{\mathbb{S}}_{Female}\) \\ \hline Intersection & & Intersection \\ \(Color\cap Fruit\) & & \({\mathbb{S}}_{Color}\cap{\mathbb{S}}_{Fruit}\) \\ \hline Set membership & & Subspace indicator function \\ \({\mathbbm{1}}_{\mathrm{set}}[boy\in Male]\) & & \({\mathbbm{1}}_{\mathrm{subspace}}(\boldsymbol{v}_{boy},{\mathbb{S}}_{Male})\) \\ \hline Set similarity & & SubspaceBERTScore \\ Recall & & \(R_{\mathrm{subspace}}\) \\ Precision & & \(P_{\mathrm{subspace}}\) \\ \({\mathbb{Q}}\) & & \(\boldsymbol{x}\times\boldsymbol{0}\) \\ F-score & & \(\boldsymbol{0}\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Correspondence between symbolic set representations and subspace-based set representations: We demonstrate that union, intersection, and complement, which are formulated in quantum logic, and our new formulations of set membership and word set similarity hold in pre-trained word embedding space.

 

defined by \({M}a\). It must return \(1\) based on the masculine property of word \(boy\). Such _hard_ membership defined by Eq. (10) is incompatible with an embedding space.

**Soft membership: Subspace indicator function** Instead, we define another membership function called _subspace indicator function_\(\mathbbm{1}_{\mathrm{subspace}}\) that returns continuous values from \(0\) to \(1\) depending on the following minimum angle between vector \(\mathbf{v}_{w}\) and subspace \(\mathbb{S}_{A}\) (the first canonical angle):

\[\mathbbm{1}_{\mathrm{subspace}}(\mathbf{v},\mathbb{S}_{A})\coloneqq \max\bigg{\{}\frac{|\mathbf{a}\cdot\mathbf{v}|}{\|\mathbf{a}\||\mathbf{v}\|}\bigg{|}\,\mathbf{a} \in\mathbb{S}_{A}\!\bigg{\}}. \tag{11}\]

This captures the degree of membership between a word and a word set, represented by the angle between a word vector and a subspace. It is a natural extension of \(\mathbbm{1}_{\mathrm{hard}}\), i.e., \(\mathbbm{1}_{\mathrm{subspace}}\) returns \(1\) when \(\mathbf{v}_{w}\in\mathbb{S}_{A}\) and \(0\) when \(\mathbf{v}_{w}\in\mathbb{S}_{\overline{A}}\).

The key distinction of our subspace indicator function approach lies in its ability to leverage the comprehensive information encapsulated within pre-trained word embedding space. The subspace indicator function does not simply find the nearest individual word from the set. Instead, we consider the _closeness_ of the query word to the entire set as a whole, **by projecting the query word into the subspace spanned by the pre-trained embeddings** (as illustrated in the figure of the subspace indicator function function in Table 1). This way, we account not just for the individual word similarities, but also for the overall semantic coherence of the word set. The detailed process for computing this subspace indicator function can be found in Algorithm 2.

### Set Similarity

**Limitation of symbolic set similarities** Suppose we quantify the set similarity between \(A=\{\textit{A}\), _boy_, _walks_, _in_, _this_, _park_\(\}\) and \(B=\{\textit{The}\), _kid_, _runs_, _in_, _the_, _square_\(\}\), which represent semantically similar sentences. The challenge with traditional symbolic set similarities, such as recall, is that they primarily rely on the exact overlap of words between the sets. These semantically similar sentences share only one word \(\{\mathit{in}\}\) between \(A\) and \(B\colon|A\cap B|=1\). To address this shortcoming, it is essential to compute vector-based set similarity, such as BERTScore.

**Three types of similarity in BERTScore** BERTScore is a method that uses embeddings to approximately calculate \(R\), \(P\), and the \(F\)-score:

\[R_{\mathrm{BERT}} =\frac{1}{|A|}\sum_{\mathbf{a}_{i}\in\mathbf{A}}\mathbbm{1}_{ \mathrm{vectors}}(\mathbf{a}_{i},\mathbf{B}), \tag{12}\] \[P_{\mathrm{BERT}} =\frac{1}{|B|}\sum_{\mathbf{b}_{i}\in\mathbf{B}}\mathbbm{1}_{ \mathrm{vectors}}(\mathbf{b}_{i},\mathbf{A}),\] (13) \[F_{\mathrm{BERT}} =2\frac{P_{\mathrm{BERT}}\cdot R_{\mathrm{BERT}}}{P_{\mathrm{ BERT}}+R_{\mathrm{BERT}}}, \tag{14}\]

where a sentence is represented as a set of token vectors \(\mathbf{A}\) and \(\mathbf{B}\). \(\mathbbm{1}_{\mathrm{vectors}}\) is the indicator function for vector sets. Intuitively, this indicator function represents the calculation of _selecting one token from the sentence_ and serves as a flexible extension of the binary indicator function \(\mathbbm{1}_{\mathrm{set}}\). It returns a continuous similarity score between \(-1\) and \(1\) for a token, depending on its similarity with the tokens in the sentence. Specifically, \(\mathbbm{1}_{\mathrm{vectors}}(\mathbf{a}_{i},\mathbf{B})\) quantifies to what extent the \(i\)-th token vector \(\mathbf{a}_{i}\) in sentence \(A\) is semantically included in sentence \(B\) by taking the maximum cosine similarity between \(\mathbf{a}_{i}\) and the token vectors in \(B\):

\[\mathbbm{1}_{\mathrm{vectors}}(\mathbf{a}_{i},\mathbf{B})=\max_{\mathbf{ b}_{j}\in\mathbf{B}}\cos(\mathbf{a}_{i},\mathbf{b}_{j})\in[-1,1], \tag{15}\]

where \(\cos(\mathbf{a}_{i},\mathbf{b}_{j})\) is the cosine similarity between \(\mathbf{a}_{i}\) and \(\mathbf{b}_{j}\).

**Limitations of BERTScore's Indicator Function** The indicator function \(\mathbbm{1}_{\mathrm{vector}}\) lies at the heart of BERTScore, playing a crucial role in the computation of \(P_{\mathrm{BERT}}\), \(R_{\mathrm{BERT}}\), and \(F_{\mathrm{BERT}}\). However, a critical limitation arises from BERTScore's reliance on maximum cosine similarity for its indicator function, which severely hinders its ability to capture the rich and nuanced meanings conveyed by a sentence. Figure 2 starkly illustrates this limitation through the visualization of BERTScore's \(\mathbbm{1}_{\mathrm{vectors}}\) calculation. Consider the sentence _We are the king and queen_, which evokes a broader, more abstract concept of _royalty_ through the co-occurrence of _king_ and _queen_. When seeking an alignment for _royalty_, BERTScore's indicator function heavily favors the token with the highest cosine similarity -- in this case, _king_. This approach leads to a severe alignment bias towards the meaning of a single word, while the complex and implicit meanings conveyed by the entire sentence are overlooked.

**SubspaceBERTScore** To overcome the limitations of BERTScore regarding the expressiveness 

of its indicator function, we propose Subspace-BERTScore. This method extends BERTScore by employing the concept of subspace-based sentence representation and indicator function.

Extension of \(P\), \(R\), \(F\) with SubspacesBased on the above discussions, we propose _Subspace-BERTScore_, which calculates BERTScore's \(R\), \(P\), \(F\) using the subspace representation of sentences and the subspace indicator function:

\[R_{\mathrm{subspace}}=\frac{1}{|A|}\sum_{\mathbf{a}_{i}\in A}\mathbbm{1}_{ \mathrm{subspace}}(\mathbf{a}_{i},\mathbb{S}_{B}), \tag{16}\]

\[P_{\mathrm{subspace}}=\frac{1}{|B|}\sum_{\mathbf{b}_{i}\in B}\mathbbm{1}_{ \mathrm{subspace}}(\mathbf{b}_{i},\mathbb{S}_{A}), \tag{17}\]

\[F_{\mathrm{subspace}}=2\frac{P_{\mathrm{subspace}}\cdot R_{\mathrm{subspace}} }{P_{\mathrm{subspace}}+R_{\mathrm{subspace}}}, \tag{18}\]

where \(R_{\mathrm{subspace}}\), \(P_{\mathrm{subspace}}\), and \(F_{\mathrm{subspace}}\) are the final evaluation measures of SubspaceBERTScore.

Weighting by ImportancePrevious study Banerjee and Lavie (2005); Vedantam et al. (2015) has shown that infrequently occurring words play a more important role in sentence similarity than general words. We apply importance weightings to our method as follows:

\[R_{\mathrm{subspace}}=\frac{\sum_{\mathbf{a}_{i}\in A}\operatorname{weight}(\mathbf{a _{i}})\mathbbm{1}_{\mathrm{subspace}}(\mathbf{a}_{i},\mathbb{S}_{B})}{\sum_{\mathbf{a }_{i}\in A}\operatorname{weight}(\mathbf{a_{i}})}, \tag{19}\]

\[P_{\mathrm{subspace}}=\frac{\sum_{\mathbf{b}_{i}\in B}\operatorname{ weight}(\mathbf{b_{i}})\mathbbm{1}_{\mathrm{subspace}}(\mathbf{b}_{i},\mathbb{S}_{A})}{ \sum_{\mathbf{b}_{i}\in B}\operatorname{weight}(\mathbf{b_{i}})}, \tag{20}\]

where \(\operatorname{weight}(\cdot)\) is a weighting function. We use the L2 norm of the vector Yokoi et al. (2020); Oyama et al. (2023).

```
0:\(\mathbf{S}_{A}\in\mathbb{R}^{k\times d}\): Bases of \(\mathbb{S}_{A}\) Input:\(\mathbf{v}_{w}\in\mathbb{R}^{1\times d}\): A word vector Output:\(\sigma\in\mathbb{R}\): Membership degree if\(k=0\)then return\(0\) else \(\widetilde{\mathbf{v}}_{w}\leftarrow\frac{\mathbf{v}_{w}}{||\mathbf{v}_{w}||}\in\mathbb{R}^{1 \times d}\)\(\mathbf{U}^{\top}\in\mathbb{R}^{k\times k},\sigma\in\mathbb{R},\mathbf{V} \in\mathbb{R}\leftarrow\text{SVD}(\mathbf{S}_{A}\widetilde{\mathbf{v}}_{w}^{\top})\)return\(\sigma\in\mathbb{R}\)\(\triangleright\) The output of \(\mathbbm{1}_{\mathrm{subspace}}(\mathbf{v}_{w},\mathbb{S}_{A})\) is always nonnegative because it is a singular value endif
```

**Algorithm 2** Subspace indicator function \(\mathbbm{1}_{\mathrm{subspace}}(\mathbf{v}_{w},\mathbb{S}_{A})\)

## 5 Semantic Textual Similarity Task

In this section, we examine the effectiveness of SubspaceBERTScore through the semantic textual similarity task (STS; Agirre et al., 2012).

TaskAn STS task calculates the similarity between two sentences. For the STS evaluation protocol, we follow Gao et al. (2021). Its evaluation is based on the correlation between the similarity calculated by the model and corresponding human judgments. We used datasets from the SemEval shared task 2012-2016 Agirre et al. (2012, 2013, 2014, 2015, 2016), STS benchmark (STS-B; Cer et al., 2017), and SICK-Relatedness (SICK-R; Marelli et al., 2014). We used Spearman's \(\rho\).

EmbeddingsWe used 768-dimensional BERTbase5Devlin et al. (2019), which was pre-trained with BookCorpus and Wikipedia. We used hidden states in the last layer.

Figure 2: Comparison between the proposed SubspaceBERTScore and BERTScore. We visualize the alignment process between the word _royalty_ and the words in the sentence \(B\). SubspaceBERTScore represents \(B\) as the subspace \(\mathbb{S}_{B}\) and calculates the similarity (canonical angle) between \(\mathbb{S}_{B}\) and the _royalty_ vector (\(\mathbf{a}_{4}\)). Our approach provides a “softer” alignment, capturing the overall semantic context of the sentence. On the other hand, BERTScore adopts a “harder” alignment strategy, selecting only the word from the sentence with the maximum cosine similarity.

 

BaselinesWe compared our method **Subspace-BERTScore** with other baseline similarity metrics. The baselines included **Avg-cos**(Arora et al., 2017), the cosine similarity between the averaged vectors, **CLS-cos**(Gao et al., 2021), the cosine similarity between the \([CLS]\) representations of the pre-trained language model, **DynaMax**(Zhelezniak et al., 2019), a set similarity based on fuzzy sets, Word Mover's Distance (**WMD**; Kusner et al., 2015), a metric based on optimal transport cost, and Word Rotator's Distance (**WMD**; Yokoi et al., 2020), an optimal transport-based metric that improves WMD.

Main resultsThe results are shown in Table 2. In comparison to BERTScore, our method achieves superior correlation with human judgments across all three key metrics: F-score, precision, and recall. An important observation is that the performance consistently improves by subspace representation of the set. The results suggest that simply replacing the representation of embedding sets and the indicator function with subspace-based alternatives significantly enhances our ability to capture and express the depth of linguistic semantics.

We also conduct an experiment using L2 norm as a weighting factor for the indicator function. This method has previously been proven effective in the STS task (Yokoi et al., 2020). We see that both our proposed method and BERTScore improve their performance underlining the effectiveness of this weighting approach in both cases. Notably, our proposed method continued to outperform BERTScore even when L2 norm was used for weighting.

Our similarity also outperforms the fuzzy-set based similarity of DynaMax. This result suggests that the proposed subspace-based approach represents a set and set operations better than the fuzzy set-based approach in embedding space.

## 6 Text Concept Set Retrieval Task

In this section, we evaluate the capability of our proposed set operations (\(\cap\), \(\cup\), and \(\in\)) in effectively representing word sets.

TaskWe evaluate our set operations by the set expansion task introduced by Zaheer et al. (2017). In this task, the model is given a set of words that share a common concept or theme. The objective is to expand this set by retrieving relevant words from a vocabulary that fit the same concept. For instance, if the initial set includes words like "apple", "banana", and "peach", the task would be to identify and add other fruit names (e.g., "orange") to this set. For the evaluation, we follow Zaheer et al. (2017). We report recall (**R@k**) and **Median**, that indicate whether the words in the test set can be ranked higher.

EmbeddingsWe used the most standard pretrained word embeddings in all of our experiments:

\begin{table}
\begin{tabular}{l c r r r r r r r r r} \hline \hline

- & .215 & .321 & .213 & .379 & .442 & .203 & .424 & .314 \\ Avg-cos &
- & .309 & .599\$ & 477 & .603 & .637 & .473 & .582\$ & .526 \\ WMD &
- & .238 & .443 & .389 & .531 & .532 & .384 & .509 & .432 \\ WRD &
- & .241 & .502 & .410 & .573 & .573 & .421 & .527 & .464 \\ DynaMax &
- & .322 & .518 & .432 & .616 & .639 & .452 & .560 & .506 \\ \hline \hline  & \(F\) &
- & .312 & .546 & .450 & .602 & .636 & .446 & .553 & .506 \\ BERTScore & \(P\) &
- & .261 & .532 & .462 & .576 & .622 & .443 & .559 & .494 \\  & \(R\) &
- & .350 & .527 & .416 & .602 & .623 & .430 & .522 & .496 \\ \hline  & \(F\) &
- & **.335** & **.573** & **.476** & **.610** & **.650** & **.479** & **.562** & **.526** \\ SubspaceBERTScore & \(P\) &
- & **.282** & **.550** & **.488** & **.580** & **.630** & **.475** & **.568** & **.511** \\  & \(R\) &
- & **.369*** & **.552** & **.436** & **.611** & **.639** & **.462** & **.530** & **.514** \\ \hline \hline  & \(F\) & L2 & .321 & .540 & .452 & .613 & .640 & .454 & .558 & .511 \\ BERTScore & \(P\) & L2 & .274 & .529 & .468 & .589 & .627 & .450 & .565 & .500 \\  & \(R\) & L2 & .348 & .520 & .414 & .610 & .624 & .437 & .524 & .497 \\ \hline  & \(F\) & L2 & **.342** & **.568** & **.477** & **.621** & **.653*** & **.486*** & **.568** & **.531*** \\ SubspaceBERTScore & \(P\) & L2 & **.292** & **.547** & **.492*** & **.592** & **.634** & **.479** & **.574** & **.516** \\  & \(R\) & L2 & **.367** & **.544** & **.434** & **.620*** & **.640** & **.468** & **.532** & **.515** \\ \hline \hline \end{tabular}
\end{table}
Table 2: A comprehensive comparison of similarity metrics in the STS task. The scores are Spearman’s \(\rho\). The methods with the highest values, using the same pre-trained embeddings, are highlighted in \(\bigstar\). Scores that showed improvement from BERTScore are denoted in **bold**.
 

300-dimensional GloVe6(Pennington et al., 2014), which was pre-trained with Common Crawl, and 300-dimensional word2vec7(Mikolov et al., 2013), which was pre-trained with Google News.

Footnote 5: [https://www.facebook.com/en/facebook/](https://www.facebook.com/en/facebook/)

Footnote 6: [https://nlp.stanford.edu/projects/glove/](https://nlp.stanford.edu/projects/glove/)

Footnote 7: [https://code.google.com/archive/p/word2vec/](https://code.google.com/archive/p/word2vec/)

**Set Expansion with Subspace Indicator Function** To illustrate our subspace-based set expansion method (**Subspace Set**), we consider a set of fruit-related words. For example, let's take \(S_{\text{fruit}}=\{apple,banana,\dots\}\). This set is divided into two subsets: a 'span' subset used for creating a subspace representation, and a 'test' subset for evaluation. Let's assume \(\mathit{orange}\notin S_{\text{fruit\_span}}\) is a target word for testing. (1) From the 'span' subset, we generate a subspace: \(\mathbb{S}_{Fruit}=\operatorname{span}(S_{\text{fruit\_span}})\). For instance, if \(S_{\text{fruit\_span}}=\{apple,banana,\dots\}\), then \(\mathbb{S}_{Fruit}=\operatorname{span}(\boldsymbol{v}_{apple},\boldsymbol{ v}_{banana},\dots)\). (2) We define a subspace indicator function, which computes the degree to which a word vector belongs to the subspace. For a word \(w\), the membership score is calculated as \(\mathrm{score}=\mathbbm{1}_{\mathrm{subspace}}(\boldsymbol{v}_{\mathit{ orange}},\mathbb{S}_{Fruit})\). This score reflects the extent to which \(w\) aligns with the semantic characteristics of the subspace. This method effectively expand the set \(S_{\text{fruit}}\) by identifying words that share semantic properties with the subspace defined by the initial set.

BaselinesWe compared several baselines, which don't require training on word sets, to our method. **Random** just selects words randomly from the dataset's vocabulary. A simple unsupervised baseline with word embeddings uses the nearest neighbors in the embedding space (**Near**)8. We also compare a method based on fuzzy sets (**Fuzzy set**; Zhelezniak et al., 2019) with our method. Similar to our method, their method is designed to exploit both the flexibility of word vectors and rich set operations. Fuzzy Set represents word set \(A\) by max-pooled word vectors \(\boldsymbol{s}=\max_{w\in A}\boldsymbol{v}_{w}\). One major difference from our method is that Fuzzy set represents a set of word vectors by compressing them into a vector of fixed dimensions. Although the Text Concept Set Retrieval task requires computing the degree of a word's membership for a word set, their method does not provide it. We instead used cosine similarity \(\cos(\boldsymbol{v}_{w},\boldsymbol{s})\) between word vector \(\boldsymbol{v}_{w}\) of word \(w\in V\) and \(\boldsymbol{s}\) as the degree of membership to apply fuzzy sets to the task.

Footnote 8: While Zaheer et al. (2017) does not provide details about this method, we have inferred through our replication experiments that it uses a method based on the cosine similarity between the query word vector and other vectors to obtain the nearest neighbor.

DatasetWe used a previously created dataset (Zaheer et al., 2017), which was denoted by "LDA-1k, Vocab = 17k." in the paper. The dataset (\(\mathbf{D^{Set}}\)) contains 100 word sets, each of which consists of 50 words sampled from a common topic9. Five pre-determined words from each set were used as the word set \(S\). An additional 800 word sets were used to train the models that require training on word sets. Table 3 shows an example of the data and the number of test sets.

Footnote 9: This work used Latent Dirichlet Allocation (LDA; Blei et al., 2003) as a topic model.

To evaluate the union and intersection sets, we 

prepared additional data through the union and intersection operations on two randomly-selected word sets from the original word sets (\({\bf D}^{\rm Set}\))10. The number of words in each set in \({\bf D}^{\rm Union}\) was limited to 50 to match the original dataset (\({\bf D}^{\rm Set}\)). The number of words in each set in \({\bf D}^{\rm Intersect}\) was set to a minimum of 10. Finally, 100 unions and intersections were randomly selected from these word sets with zero elements excluded. See Table 3 for examples and statistics of the datasets.

Footnote 10: Note that these were based on the dataset LDA-1k, which was automatically generated by Zaheer et al. (2017) using LDA, so the quality depends on their method.

ResultsIn experiments on union and intersection, we compared our method only with Fuzzy Set. The proposed method and Fuzzy Set can induce representations for the union and intersection using set operations defined in the word embedding space; the others cannot do so directly. Table 4 shows the experimental results. Here our subspace-based set operation method (Subspace set) is the best among the methods that did not require training. The results suggest that combining off-the-shelf pre-trained embeddings with appropriate set-oriented operations makes linguistic computation on sets feasible without additional training. The results in \({\bf D}^{\rm Union}\) and \({\bf D}^{\rm Intersect}\) show that the our method outperform Fuzzy Set in most metrics. As methods for achieving set operations in vector spaces, the proposed method is empirically more promising than the existing fuzzy set-based method.

## 7 Related Work

Symbol-based similarities between word sets have been proposed, such as Jaccard coefficient Jaccard (1901); Manning and Schutze (2001); Thada and Jaglan (2013) and TF-IDF-based similarity Jurafsky (2000). Unfortunately, symbol-based methods cannot capture the semantic similarity of similar sets or words when the symbols are different.

While many studies have explored representing word sets in pre-trained embedding spaces Kusner et al. (2015); Yokoi et al. (2020), they primarily focus on set similarity. Our approach, however, extends beyond this by developing a comprehensive framework for various set operations within these spaces. Utilizing subspace properties, our method not only represents word sets but also performs a range of versatile operations, such as calculating textual similarities and membership degrees.

Many methods for learning the representation of sets have been proposed because of the wide range of possible applications Zaheer et al. (2017); Pellegrini et al. (2021); Lee et al. (2019); Vilnis and McCallum (2015); Athiwaratkan and Wilson (2017). In contrast, our approach does not require additional training. This enables us to compute set representations and operations using popular general-purpose language models, which are trained on the general domain Brown et al. (2020).

## 8 Conclusion

This study introduces a novel framework for set representation and operations within pre-trained embedding spaces, employing linear subspaces grounded in quantum logic. This approach extends the scope of conventional embedding set operations by incorporating vector-based representations.

### Ethical Considerations

We recognize the importance of addressing the inherent biases in pre-trained models, such as gender stereotypes. In our experiment, we used RoBERTa, which has gender biases Sharma et al. (2021). We used this model in its original state to preserve the experimental conditions of BERTScore, acknowledging that such biases may influence our results. However, we would like to emphasize that the focus of our work, which lies in sentence similarity, does not inherently add to or magnify these ethical concerns.

### Limitations

Our SubspaceBERTScore is built upon the foundation of BERTScore, which presents a limitation in that our results and findings are inherently dependent on the characteristics and performance of BERTScore. While we chose BERTScore due to its robustness and popularity in the field, potential biases or shortcomings intrinsic to BERTScore might be incorporated into our extension. Nevertheless, this constraint also suggests future research possibilities, such as applying our subspace-based approach to other base sentence similarity metrics, further expanding the versatility and applicability of our method.

The experiments we conducted were exclusive to BERT and RoBERTa. Testing our methodology with other pre-trained models, like GPT-3 Brown et al. (2020), could broaden its applicability and establish its robustness across various pre-trained models.

 

We evaluated our methodology primarily using English datasets. This decision was made to streamline our initial explorations rather than due to an inherent language-specific bias in our approach. We expect that our subspace-based methodology will be effective across various languages.

## Acknowledgments

This work was supported by JSPS KAKENHI Grant Number 22H03654, 22H03651, and JST, ACT-X Grant Number JPJMAX200S, Japan.

## References

* Agrirre et al. (2015) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M. Cer, Mona T. Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability. In _Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2015, Denver, Colorado, USA, June 4-5, 2015_, pages 252-263. The Association for Computer Linguistics.
* Agirre et al. (2014) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M. Cer, Mona T. Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 Task 10: Multilingual Semantic Textual Similarity. In _Proceedings of the 8th International Workshop on Semantic Evaluation, SemEval@COLING 2014, Dublin, Ireland, August 23-24, 2014_, pages 81-91. The Association for Computer Linguistics.
* Agirre et al. (2016) Eneko Agirre, Carmen Banea, Daniel M. Cer, Mona T. Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In _Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016_, pages 497-511. The Association for Computer Linguistics.
* Agirre et al. (2012) Eneko Agirre, Daniel M. Cer, Mona T. Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity. In _Proceedings of the 6th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2012, Montreal, Canada, June 7-8, 2012_, pages 385-393. The Association for Computer Linguistics.
* Agirre et al. (2013) Eneko Agirre, Daniel M. Cer, Mona T. Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. "SEM 2013 shared task: Semantic Textual Similarity. In _Proceedings of the Second Joint Conference on Lexical and Computational Semantics, *SEM 2013, June 13-14, 2013, Atlanta, Georgia, USA_, pages 32-43. Association for Computational Linguistics.
* Arora et al. (2017) Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A Simple but tough-to-Beat Baseline for Sentence Embeddings. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
* August 4, Volume 1: Long Papers_, pages 1645-1656.
* Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In _Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, Ann Arbor, Michigan, USA, June 29, 2005_, pages 65-72. Association for Computational Linguistics.
* Birkhoff and Von Neumann (1936) Garrett Birkhoff and John Von Neumann. 1936. The logic of quantum mechanics. _Annals of mathematics_, pages 823-843.
* Blei et al. (2003) David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. _J. Mach. Learn. Res._, 3:993-1022.
* Ondw\({}_{r}\)ej Bojar et al. (2018) Ondw\({}_{r}\)ej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT'18). In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_, pages 272-303, Belgium, Brussels. Association for Computational Linguistics.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighian, Rewon Child, Aditya Ramesch, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Cer et al. (2017) Daniel M. Cer, Mona T. Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. In _Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval@ACL 2017, Vancouver, Canada, August 3-4, 2017_, pages 1-14. Association for Computational Linguistics.

 

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_, pages 4171-4186. Association for Computational Linguistics.
* Gao et al. [2021] Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_, pages 6894-6910. Association for Computational Linguistics.
* Jaccard [1901] Paul Jaccard. 1901. Distribution de la flore alpine dans le bassin des dranses et dans quelques regions voisines. _Bulletin de la Societe Vaudoise des Sciences Naturelles_, 37:241-72.
* Jurafsky [2000] Dan Jurafsky. 2000. _Speech & language processing_. Pearson Education India.
* Kusner et al. [2015] Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. 2015. From Word Embeddings To Document Distances. In _Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015_, volume 37 of _JMLR Workshop and Conference Proceedings_, pages 957-966. JMLR.org.
* Lee et al. [2019] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks. In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_, volume 97 of _Proceedings of Machine Learning Research_, pages 3744-3753. PMLR.
* November 1, 2018_, pages 671-688. Association for Computational Linguistics.
* Manning and Schutze [2001] Christopher D. Manning and Hinrich Schutze. 2001. _Foundations of statistical natural language processing_. MIT Press.
* Marelli et al. [2014] Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaeella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In _Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26-31, 2014_, pages 216-223. European Language Resources Association (ELRA).
* Mikolov et al. [2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In _Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States._, pages 3111-3119.
* Ogama et al. [2023] Momose Ogama, Sho Yokoi, and Hidetoshi Shimodaira. 2023. Norm of word embedding encodes information gain. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_, pages 2108-2130. Association for Computational Linguistics.
* Pellegrini et al. [2021] Giovanni Pellegrini, Alessandro Tibo, Paolo Frasconi, Andrea Passerini, and Manfred Jaeger. 2021. Learning Aggregation Functions. In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021_, pages 2892-2898. ij-caic.org.
* Pennington et al. [2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global Vectors for Word Representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL_, pages 1532-1543.
* Sharma et al. [2021] Shanya Sharma, Manan Dey, and Koustuv Sinha. 2021. Evaluating gender bias in natural language inference. _CoRR_, abs/2105.05541.
* Thada and Jaglan [2013] Vikas Thada and Vivek Jaglan. 2013. Comparison of jaccard, dice, cosine similarity coefficient to find best fitness value for web retrieved documents using genetic algorithm. _International Journal of Innovations in Engineering and Technology_, 2(4):202-205.
* Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 5998-6008.
* Vedantam et al. [2015] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015_, pages 4566-4575. IEEE Computer Society.
* Vilnis and McCallum [2015] Luke Vilnis and Andrew McCallum. 2015. Word Representations via Gaussian Embedding. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Yokoi et al. [2020] Sho Yokoi, Ryo Takahashi, Reina Akama, Jun Suzuki, and Kentaro Inui. 2020. Word Rotator's Distance. In 

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 2944-2960. Association for Computational Linguistics.
* Zaheer et al. (2017) Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander J. Smola. 2017. Deep Sets. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 3391-3401.
* Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Zhelezniak et al. (2019) Vitalii Zhelezniak, Aleksandar Savkov, April Shen, Francesco Moramarco, Jack Flann, and Nils Y. Hammerla. 2019. Don't Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectors. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.

## Appendix A Pseudocodes

The pseudocodes for our SubspaceBERTScore and basic set operations are shown in Algorithms 3, 4, 5, and 6.

```
0:\({\bf S}_{A}\in\mathbb{R}^{k\times d}\): Bases of \(\mathbb{S}_{A}\) Input:\({\bf S}_{B}\in\mathbb{R}^{\ell\times d}\): Bases of \(\mathbb{S}_{B}\) Output:\({\bf S}_{A\cup B}\in\mathbb{R}^{r\times d}\): Bases of \(\mathbb{S}_{A}\cup\mathbb{S}_{B}\) \({\bf M}\in\mathbb{R}^{(k+\ell)\times d}\leftarrow\text{concat\_rows}({\bf S}_ {A},{\bf S}_{B})\) \({\bf S}_{A\cup B}\in\mathbb{R}^{r\times d}\leftarrow\text{ortho\_normal}({\bf M })\vartriangle r\) is rank of \({\bf M}\) return\({\bf S}_{A\cup B}\)
```

**Algorithm 3** Union

```
0:\({\bf S}_{A}\in\mathbb{R}^{k\times d}\): Bases of \(\mathbb{S}_{A}\) Input:\({\bf S}_{B}\in\mathbb{R}^{\ell\times d}\): Bases of \(\mathbb{S}_{B}\) Output:\({\bf S}_{A\cup B}\in\mathbb{R}^{r\times d}\): Bases of \(\mathbb{S}_{A}\cup\mathbb{S}_{B}\) \({\bf M}\in\mathbb{R}^{(k+\ell)\times d}\leftarrow\text{concat\_rows}({\bf S}_ {A},{\bf S}_{B})\) \({\bf S}_{A\cup B}\in\mathbb{R}^{r\times d}\leftarrow\text{ortho\_normal}({\bf M })\vartriangle r\) is rank of \({\bf M}\) return\({\bf S}_{A\cup B}\)
```

**Algorithm 4** Intersection

```
0:\({\bf S}_{A}\in\mathbb{R}^{k\times d}\): Bases of \(\mathbb{S}_{A}\) Input:\({\bf S}_{B}\in\mathbb{R}^{\ell\times d}\): Bases of \(\mathbb{S}_{B}\) (\(\ell\geq k\)) \(\alpha\): Threshold below which the cosine of canonical angles is considered zero.
0:\({\bf S}_{A\cap B}\in\mathbb{R}^{m\times d}\): Bases of \(\mathbb{S}_{A}\cap\mathbb{S}_{B}\) \({\bf M}\in\mathbb{R}^{k\times\ell}\leftarrow{\bf S}_{A}{\bf S}_{B}^{\top}\) \({\bf U}\in\mathbb{R}^{k\times\ell},{\mathbf{\Sigma}}\in\mathbb{R}^{\ell\times\ell },{\mathbf{V}}^{\top}\in\mathbb{R}^{\ell\times\ell}\gets{\rm SVD}({\bf M})\) \(\triangleright~{}{\mathbf{\Sigma}}={\rm diag}(\sigma_{1},\dots,\sigma_{\ell})~{}( \sigma_{1}\geq\cdots\geq\sigma_{\ell})\) has cosines of the canonical angles between \(\mathbb{S}_{A}\) and \(\mathbb{S}_{B}\). \({\bf W}\in\mathbb{R}^{m\times d}\leftarrow{\bf U}[:,1:\sigma_{m}]\) \(\triangleright~{}m\) is the maximum index that satisfies \(|\sigma_{i}-1|\leq\alpha\). return\({\bf S}_{A\cap B}\)
```

**Algorithm 5** Complement

```
0:\({\bf S}_{A}\in\mathbb{R}^{k\times d}\): Bases of \(\mathbb{S}_{A}\) Output:\({\bf S}_{\overline{A}}\in\mathbb{R}^{d\times d}\): Bases of \(\mathbb{S}_{\overline{A}}\) \({\bf U}\in\mathbb{R}^{d\times d},{\mathbf{\Sigma}}\in\mathbb{R}^{d\times k},{\mathbf{V }}^{\top}\in\mathbb{R}^{k\times k}\leftarrow{\rm SVD}({\bf S}_{\overline{A}}^{ \top})\) \({\bf S}_{\overline{A}}\in\mathbb{R}^{(d-k)\times d}\leftarrow({\bf U}[:,k:d])^{\top}\) return\({\bf S}_{\overline{A}}\)
```

**Algorithm 5** Complement

## Appendix B WMT Results

An example of a practical task where our proposed method can be directly utilized is the automatic evaluation of machine translation systems. We applied our SubspaceBERTScore to the WMT18 Ma et al. (2018). The table 5 presents the results of applying both BERTScore and our 

proposed SubspaceBERTScore to various X-to-English translation settings in the WMT18 competition. Our proposed method consistently outperforms BERTScore. Specifically, for all X-to-English translation settings, we observed an increase in Kendall's \(\tau\) for F-score, Precision (P), and Recall (R). It is evident that SubspaceBERTScore shows an improvement across all the considered metrics, underscoring the effectiveness of our proposed method in evaluating machine translation systems.

## Appendix C Vector vs. Set of Vectors

Table 6 presents the comparison results on the STS-B dataset. We utilized embeddings from BERT and SimCSE Gao et al. (2021). Avg-cos and CLS-cos are based on the cosine similarity of vectors, employing methods that compress information into a single vector. In contrast, BERTScore among other methods is based on sets of vectors. A key takeaway from these results is the suggestion that methods using sets of vectors outperform those using a single vector in evaluating similarity. It is particularly interesting that even SimCSE, which aims to optimize average cosine similarity, shows superior score when using set-based similarity.

\begin{table}
\begin{tabular}{l l c} \hline \hline
**Emb.** & **Method** & **STS-B** \\ \hline \multirow{3}{*}{BERT-base} & CLS-cos & .424 \\  & Avg-cos & .582 \\  & BERTScore\({}^{\dagger}\) & .446 \\  & SubspaceBERTScore\({}^{\dagger}\) & .478 \\ \hline \multirow{3}{*}{SimCSE} & CLS-cos & .769 \\  & AVg-cos & .786 \\  & BERTScore\({}^{\dagger}\) & .798 \\  & SubspaceBERTScore\({}^{\dagger}\) & .801 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Comparison of vector-based similarity and set-based similarity metrics (\({}^{\dagger}\)) on the STS-B dataset.

\begin{table}
\begin{tabular}{l r r r r r r r r r} \hline \hline
**Method** & **Metric** & **cs-en** & **de-en** & **et-en** & **fi-en** & **ru-en** & **tr-en** & **zh-en** & **Avg.** \\ \hline \multirow{3}{*}{BERTScore} & F & .404 & .550 & .397 & .296 & .353 & .292 & .264 & .365 \\  & P & **.387** & .541 & .389 & .283 & .345 & .280 & .248 & .353 \\  & R & .388 & .546 & .391 & .304 & .343 & .290 & .255 & .360 \\ \hline \multirow{3}{*}{SubspaceBERTScore} & F & **.411** & **.557** & **.403** & **.309** & **.358** & **.303** & .264 & **.372** \\  & P & .382 & **.548** & **.393** & **.290** & **.352** & **.294** & .248 & **.358** \\ \cline{1-1}  & R & **.391** & **.547** & **.392** & **.313** & **.358** & **.292** & **.259** & **.365** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Comparison of BERTScore and SubspaceBERTScore on WMT18 for X-to-English translation tasks.