AoE: Angle-optimized Embeddings for Semantic Textual Similarity (2024)

Abstract: Text embedding is pivotal in semantic textual similarity (STS) tasks, which are crucial components in Large Language Model (LLM) applications. STS learning largely relies on the cosine function as the optimization objective to reflect semantic similarity. However, the cosine has saturation zones rendering vanishing gradients and hindering learning subtle semantic differences in text embeddings. To address this issue, we propose a novel Angle-optimized Embedding model, AoE. It optimizes angle differences in complex space to explore similarity in saturation zones better. To set up a comprehensive evaluation, we experimented with existing short-text STS, our newly collected long-text STS, and downstream task datasets. Extensive experimental results on STS and MTEB benchmarks show that AoE significantly outperforms popular text embedding models neglecting cosine saturation zones. It highlights that AoE can produce highquality text embeddings and broadly benefit downstream tasks. The code is available at: https://github.com/SeanLee97/AnglE

heading: 1 Introduction
text: Text embeddings, essential language features, are foundations of semantic textual similarity (STS) tasks, which quantify how similar two text pieces are in semantics (Kiros et al., 2015; Hill et al., 2016; Conneau et al., 2017; Cer et al., 2018; Reimers and Gurevych, 2019; Gao et al., 2021). They broadly benefit downstream tasks, such as information retrieval (Asai et al., 2023) and clustering (Xu et al., 2023), and are particularly helpful in many recent LLMs-based applications (OpenAI, 2022a; Touvron et al., 2023); e.g., many RAG tasks employ text embeddings for retrieval (Asai et al., 2023).
The existing STS training commonly involves optimizing cosine functions — the learning ob-
∗Previously known as “AnglE” †Corresponding author
jective to indicate the similarity of pairwise text embeddings (Reimers and Gurevych, 2019; Gao et al., 2021; Su, 2022; Zhuo et al., 2023). However, the cosine has saturation zones, resulting in gradient vanishing in optimization regardless of the network depth (Roodschild et al., 2020). The gradient will be close to zero for embedding pairs falling in the saturation zone, preventing parameters from updating in backpropagation. Because embedding pairs in saturation zones are nearly aligned or antialigned, it hinders text embedding models from discerning subtle, implicit differences that appear similar yet are actually dissimilar in semantics.
Such pairs commonly appear in STS training data from Natural Language Inference (NLI) datasets, such as the Multi-Genre NLI (MNLI) (Williams et al., 2018) and the Stanford NLI (SNLI) (Bowman et al., 2015). They typically include three labels of entailment, neutral, and contradict; pairs in saturation zones may render obscure cross-label boundaries. To illustrate this point, Figure 1 shows an example from the SNLI dataset. The “neutral” pair shows a high appearance similarity (with many shared words) instead of semantically similar. The similar appearance similarity results in them falling into cosine’s saturation zones, causing vanishing gradients during optimization. Consequently, the model mistakenly considers their relations as “entailment” instead of their correct label “neutral.”
1825
Viewing these concerns, we aim to tackle the negative effects of the cosine’s saturation zones in embeddings and propose a novel Angle-optimized Embedding (AoE) model for STS. It decomposes an embedding into real and imaginary components through complex division, aiming to employ the real component for reflecting appearance differences and the imaginary component for subtle differences. It allows AoE to involve the optimization of the angle difference to understand subtle differences in text pairs for similarity learning.
To the best of our knowledge, we are the first to explore the negative effects of cosine’s saturation zones and optimize angle differences through division in complex space for text embedding learning.
In the STS experimental setup, we observed that most existing STS benchmarks focus on evaluating models on short texts. Unfortunately, limited datasets are available to evaluate the STS performance on long texts. However, long texts are prevalent in real-world applications such as financial and legal documents (Li et al., 2023). To tackle this challenge, we present a high-quality long-text STS dataset collected from GitHub Issues with roughly 22K samples. It allows for a more comprehensive evaluation of STS performance with long texts.
We first experimented with short- and long-text STS datasets in the standard and in-domain STS tasks, where AoE outperforms non-trivial baselines in varying embedding backbones. Then, AoE shows consistently superior results in facilitating various downstream tasks, indicating its benefits in diverse scenarios. In particular, AoE achieves SOTA results on the Massive Text Embedding Benchmark (MTEB) (Muennighoff et al., 2022) at the same model scale. Next, an ablation study indicates that all modules positively contribute to AoE. Finally, we further discuss how AoE learns better embeddings in cosine saturation zones.
In summary, our contributions are as follows:
• We investigate the effects of cosine saturation zones for STS and optimize angle differences in complex space for improving text embedding.
• We extend the existing STS benchmark with a new long-text dataset from Github Issues to allow more comprehensive STS empirical studies.
• We present extensive experiments demonstrating that AoE effectively handles cosine saturation zones to broadly benefit text embedding learning and create positive effects in various scenarios.

heading: 2 Related Work
text: Our work is related to text embedding learning. Compared to early efforts focusing on word embeddings (Mikolov et al., 2013), text embeddings (a more general concept) enable semantic representation for richer context. Many prior studies (Li et al., 2020; Su et al., 2021a) employed pretrained models such as BERT (Devlin et al., 2019) for learning text embeddings without fine-tuning. More recent work showed the benefits of fine-tuning the pretrained models to improve STS. Some studies (Conneau et al., 2017; Cer et al., 2018) involved humanlabeled training data, e.g., NLI datasets. In training methods, SBERT (Reimers and Gurevych, 2019) used siamese BERT, while many others adopted contrastive learning (Zhang et al., 2020; Gao et al., 2021; Chuang et al., 2022; Zhuo et al., 2023).
Most widely-used models employ cosine to measure similarity in their learning objectives, as shown in Table 6, Appendix A. Cosine exhibits saturation zones leading to gradient vanishing (Roodschild et al., 2020). It hinders embedding models from encoding subtle differences in pairs falling in the saturation zones. However, none of the existing work considers such an issue, and we propose the angle-optimized AoE model to mitigate this gap.
AoE is inspired by complex embeddings for using complex division to exploit angle differences. In positional word embedding learning, Su et al. (2021b) adopted complex multiplication to introduce rotary position embedding into transformer architecture. In knowledge graph learning, Trouillon et al. (2016); Sun et al. (2019) presented entity embedding in complex space to model the source to target rotations for link prediction. However, their embeddings are on the word or entity level and focus on their position modeling. On the contrary, we make the first efforts to leverage complex division to compute normalized angle differences between sentence-level text embeddings for mitigating the negative effects of the cosine’s saturation zones.

heading: 3 AoE Framework
text: Here, we will introduce the AoE methods with the overall framework in Figure 2. We will first present how to encode the complex text embeddings in Section 3.1, followed by the angle objective in Section 3.2. At last, Section 3.3 will outline the final learning objective to describe AoE’s training process.

heading: 3.1 Complex Text Embeddings
text: The first step of AoE is to transform the input text into complex text embeddings. Our intuition is to employ real components for learning appearance differences (like prior practices (Reimers and Gurevych, 2019; Gao et al., 2021)) and imaginary ones for subtle semantic differences. This way, we can exploit the angle differences for embeddings’ similarity learning in cosine saturation zones.
To achieve this, we first input the text x into the embedding layer to obtain the token embeddings: E = Emb(x) ∈ R2d. Inspired by Sun et al. (2019), the token embeddings E include two sub-spaces. The first d embeddings represent the real token embeddings Ere = E1:d ∈ Rd, while the embeddings from d to 2d represent the imaginary token embeddings, denoted as Eim = Ed:2d ∈ Rd.
Then, the token embeddings are fed into Transformer encoders like BERT (Devlin et al., 2019) or LLaMA (Touvron et al., 2023) to obtain the text embeddings: X = Encoderp(E) ∈ R2d, where p means pooling. Specifically, embeddings of the “CLS” token for BERT and the last token for LLaMA represent sentence-level text embeddings. Consequently, a text embedding has real (Xre = X1:d ∈ Rd) and imaginary components (Xim ::::
= Xd:2d ∈ Rd), where wave:::: henceforth indicates the imaginary embeddings for easy reading.

heading: 3.2 Angle Objective
text: After obtaining the complex text embeddings, we present the angle objective in complex space. Specifically, for the input text pair (xi, xj), we obtain their real and imaginary text embeddings (Xrei , Ximi :::: ) and (Xrej , X im j
::::
) via the process in Section
3.1. To allow clearer formula derivation, we define:
z = a+ b : i ∈ C,w = c+ d : i ∈ C, (1)
where a = Xrei ∈ Rd, b: = Ximi :::: ∈ Rd, c = Xrej ∈ Rd, and d: = Ximj ::::
∈ Rd. Then, we conduct complex division to determine the angle difference and the factor in magnitude. Based on the complex division rule, we can measure the angle difference between embeddings z and w, ∆θzw, as follows:
∆θzw = log
  (ac+ b : d : ) + (b : c− ad : )
√ (c2 + d2
:: )(a2 + b2 :: )
  , (2)
where the denominator serves as the normalization term (naturally derived from complex division). For detailed derivation, we refer readers to Appendix C. Based on that and following Su (2022), we optimize the angle difference between input text pairs with the ranking objective function below:
Langle = log  1 + ∑
sij>smn
exp( ∆θij −∆θmn
τ )
  , (3)
where τ is a temperature hyperparameter. sij is the similarity between text xi and xj , and smn is the similarity between text xm and xn. sij > smn is from the ranking of training data labels. By optimizing the angle objective, Langle, we aim to minimize the angle difference for pairs with high similarity compared to those with low similarity. Thus, for embedding pairs in cosine saturation zones (e.g., similar ones in appearance), the angle objective helps reflect the subtle semantic differences, mitigating the negative effects of gradient vanishing.

heading: 3.3 Training Process of AoE Framework
text: In the embedding training of AoE, we optimize the angle objective (Section 3.2) with the auxiliary objective. This multi-objective approach allows the AoE framework to learn text embeddings comprehensively from multiple perspectives, enhancing the model’s overall performance. Here, we employ the widely-used supervised contrastive learning objective as the auxiliary objective Lcl, as follows:
Lcl = − ∑
b
m∑
i
log
  e cos(Xbi ,X+ bi )/τ
∑N j e cos(Xbi ,X+ bj )/τ
  , (4)
where τ is a temperature hyperparameter, b stands for the b-th batch, X+bi and X + bj
are the respective positive samples of Xbi and Xbj , m represents the number of positive pairs in b-th batch, N is the batch size, and cos(·) is the cosine similarity.
In the training, we combine the angle objective and the contrastive objective in the following manner to form the final objective function:
L = w1 · Langle + w2 · Lcl, (5)
where w1 and w2 are two hyperparameters to control the weights of balancing the two objectives.

heading: 4 Experimental Setup
text: Here, we elaborate on the experimental setup, including datasets, baselines, evaluation metrics, and implementation details. We also open-source our trained models in Appendix Section E.
Datasets. Following standard setup (Gao et al., 2021), the training data is from MultiNLI and SNLI. Our statistics reveal that 33% of the text pairs show
a similarity above 0.95 and 66% above 0.8. It means a large proportion of samples in or near cosine saturation zones, implying the challenges of learning subtle semantic differences for them.
For evaluation, we test AoE on STS tasks with existing widely-used short-text STS datasets and our newly proposed long-text GitHub Issue Similarity Dataset. Furthermore, we examine AoE on downstream data with 7 popular tasks and MTEB. • Existing Short-text STS Tasks. We first evaluate AoE on 7 widely-adopted STS datasets, namely: STS 2012-2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), SICK-R (Marelli et al., 2014), and STS-B (Cer et al., 2017). These datasets mainly consist of short text (less than 512 tokens), whereas real-world scenarios often involve long texts. Viewing this gap, we introduce a new long-text dataset called GitHub Issues Similarity Dataset (GIS) as follows for a more extensive STS evaluation. • GitHub Issues Similarity Dataset (GIS). The GIS dataset was gathered based on the GitHub duplicate issues indicating high similarity. The duplication label is easy to access because maintainers of open source organizations tend to mark these duplicate issues as closed with a comment like “closing as a duplicate of #id.” Consequently, these duplicate issues inherently serve as a source of the STS task. Here, most issues contain long text because of the large amount of code involved.
We extracted duplicated issues from 55 famous open-source projects (see Appendix B) on GitHub using GitHub API to compile the dataset. The duplicate issues served as positive samples, while the remaining ones were considered negative. These open-source projects have active participation from maintainers and volunteers to maintain the issue quality. Additionally, We randomly selected 10% of the data for manual inspection, and the quality was found to be satisfactory. 93% of the sampled data can be clearly classified as either similar with label 0 or dissimilar with label 1. Our statistics show that the proportion of long text (with token length > 512) for the train, validation, and test sets is 61.03%, 60.85%, and 60.50%, respectively. More details of GIS are presented in Appendix B.1
• Downstream Tasks. Following standard practice, we evaluate AoE on 7 downstream tasks: MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe
1The dataset can be downloaded at https://hf.co/ datasets/WhereIsAI/github-issue-similarity.
Model STS12 STS13 STS14 STS15 STS16 STS-B SICK-R Avg.
Closed-source Models
openai-ada-002 69.80 83.27 76.09 86.12 85.96 83.17 80.60 80.72 openai-text-embedding-3 72.84 86.10 81.15 88.49 85.08 83.56 79.00 82.32
et al., 2005), SST2 (Socher et al., 2013), TREC (Voorhees and Tice, 2000), and MRPC (Dolan et al., 2004). These tasks mainly evaluate the classification performance of text embeddings. We also examine AoE on the MTEB (Muennighoff et al., 2022) for a more thorough downstream task evaluation. It includes classification (12 datasets), clustering (11 datasets), pair classification (3 datasets), reranking (4 datasets), retrieval (15 datasets), STS (10 datasets), and summarization (1 dataset) tasks.
Evaluation Metrics. For STS, we follow previous studies to use SentEval (Conneau and Kiela, 2018) to compute Spearman’s correlation and report the “all” setting. For downstream tasks, we employ SentEval to assess the performance of text embeddings. For a fair comparison, we follow baselines and use the default parameters of Sen-
tEval. For MTEB, we employ the official MTEB evaluation code to test the performance of AoE.
For all our implementations, we will report the average score over five runs and the std value (±).
Baselines. Because AoE is supervised, we primarily compare it with widely used supervised embedding baselines for a fair comparison. They are: InferSent (Conneau et al., 2017), USE (Cer et al., 2018), SBERT (Reimers and Gurevych, 2019), CoSENT (Su, 2022), and supervised versions of SimCSE (Gao et al., 2021) and ConSERT (Yan et al., 2021). In particular, given different backbones, we compare AoE with SBERT and SimCSE, the two most widely-used text embedding baselines. All the above baselines are opensource embeddings. In addition, we adopt two popular closed-source baselines, OpenAI’s Ada-002 (OpenAI, 2022b) and OpenAI’s text-embeddings-3 (OpenAI, 2024), for a comprehensive comparison.
Implementation Details. We extensively examine AoE on three scales of pre-trained backbone models: BERTbase (uncased) (Devlin et al., 2019), LLaMA7B (LLaMA2-7B) (Touvron et al., 2023) and its counterpart in 13B. As for BERT, we set the initial learning rate to 5e− 5. For LLaMA, we apply the QLoRA (Dettmers et al., 2023) technique
for efficient fine-tuning with the initial learning rate to 1e − 4. For embeddings, we used the prompt “Summarize sentence {text} in one word:” to obtain the summative token and concatenate it to the last token of the text and then apply its token embeddings as the text embeddings, inspired by (Jiang et al., 2023). For the temperature in objectives, we set the τ to 0.05 following the previous practice (Gao et al., 2021). For w1 and w2 in Equation 5, we use the grid search strategy to search for their values. For a fair comparison with prior work, we follow SimCSE (Gao et al., 2021) to set the random seed to 42 for all main experiments. Yet, in Section 5.2, we test AoE without fixed random seeds to examine its robustness.

heading: 5 Experimental Results
text: Section 5.1 first presents the main comparison results, followed by an ablation study in Section 5.2. Finally, we will further discuss AoE in Section 5.3.

heading: 5.1 Main Comparison Results
text: In the experimental comparison, we examine benchmark results of STS and downstream tasks for intrinsic and extrinsic embedding evaluations.
Standard STS. We begin with the standard STS benchmark experiments for models trained using MultiNLI and SNLI datasets and evaluated on SentEval. The results are presented in Table 1, where we can draw the following observations.
First, larger backbone models generally result in better performance. It implies the larger model scales of LLMs can helpfully capture deeper semantics for the STS prediction. Second, SimCSE works better than SBERT, possibly benefitting from contrastive learning for capturing semantic similarity. Third, given any backbone, AoE consistently
performs best in all STS benchmarks. For example, compared to SimCSE, AoE demonstrates average score improvements of 0.86%, 0.72%, and 0.60% for BERTbase, LLaMA7B , and LLaMA13B , respectively. While sharing a contrastive learning objective with SimCSE, AoE’s performance gain likely comes from the novel addition of the angle objective. It allows optimizing the angle differences to explore the subtle semantic differences of training samples in cosine saturation zones, which is prevalent in the training data as we showed in Section 4. Furthermore, AoE’s improvements observed across different backbone model sizes indicate that the benefit from the angle objective is universal.
In-domain STS. To further examine the embedding training specifically, we experimented with in-domain STS tasks for STS-B with short text and GIS with long text. Here, the training and test sets are obtained from the same dataset, and BERTbase is the backbone for efficiency restrictions with long text. Table 2 presents the results. As can be seen, all models perform much worse on GIS than STS-B. It implies that long-text STS presents non-trivial challenges, requiring more indepth exploration. We also note that SimCSE performs worse than SBERT (opposite to standard STS). It indicates that contrastive learning may rely on large-scale training samples, which is inadequate for in-domain STS. Nevertheless, AoE consistently performs the best, achieving improvements of 1.35% and 10.11% compared to SBERT and SimCSE, respectively. It indicates that the angle objective may enable more efficient STS training, reducing reliance on large-scale training data.
Downstream Tasks. The above experiments concerned intrinsic evaluations. For extrinsic evalua-
tions, we assess how the embeddings can benefit downstream tasks. Here, we first consider 7 popular classification tasks and show the results in Table 3. We can see that AoE-BERTbase performs better than other BERTbase baselines, showing the subtle semantics captured by the angle objective can further benefit downstream tasks. Moreover, AoE-LLaMA13B achieves the best performance. These results indicate that AoE can produce text embeddings that helpfully assist downstream tasks.
MTEB Benchmark. We have shown the superiority of AoE embeddings on classification. Here, the leaderboard experiments of MTEB benchmark further provide a more extensive study in downstream tasks (Muennighoff et al., 2022). We trained AoE using the widely-used embedding data2 and the supervised data released by BGE (Zhang et al., 2023). In the experimental results, AoE achieved SOTA performance in BERT-large scale models, with an average score of 64.64. Specifically, AoE outperformed the top 2 open-source BERT-large models: bge-large-en-v1.5 (64.23) and ember-v1 (63.54). Moreover, it outperforms popular closedsource models: openai-text-embedding-3-large (OpenAI, 2024) (64.59), voyage-lite-01-instruct (64.49), and Cohere-embed-english-v3.0 (64.47). Most aforementioned comparison models are based on contrastive learning. It indicates that our novel angle objective design can provide performance gain for more challenging downstream tasks.

heading: 5.2 Ablation Study
text: While AoE has demonstrated overall effectiveness, we conduct ablation studies on the standard STS to investigate the contributions of AoE’s different modules. The results are shown in Table 4.
First, we examine AoE’s performance with varying objectives. Interstingly, using only the angle objective outperforms the counterpart using only the contrastive objective. It indicates that our angle objective might be more effective in learning semantic similarity than the contrastive objective. Nevertheless, combining both of them yields the best results. Second, we test AoE’s performance on four different pooling strategies and find that the “cls” pooling is the most helpful one. Third, we test how random seeds affect AoE, and the results show that AoE is not sensitive to random seeds and robustly effective across varying selections.
2https://huggingface.co/embedding-data
Model Avg. Spearman’s Correlation
Objective
AoE 82.43±0.08 only angle objective 82.36±0.14 only contrastive objective 81.53±0.19
Pooling Strategy
cls 82.43±0.11 avg 81.69±0.18 max 77.96±0.21
Random Seed

heading: 5.3 Further Discussions and Analyses
text: To provide more insight, we further probe into AoE’s output to interpret why it enables effective embedding learning as follows. Besides, we discuss its efficiency (training time) in Appendix D.
NLI Performance. Recall that we employed NLI datasets to train text embeddings, whereas the downstream task benchmarks do not involve NLI (see Table 3). We are hence interested in how AoE embeddings can benefit NLI tasks. Here, AoE is mainly compared with BERT and SimCSE text embeddings. Specifically, we input “[CLS]premise[SEP]hypothesis[SEP]” into the model and extract the representation of the “[CLS]” token for the logistic regression classification following SentEval (Conneau and Kiela, 2018). For the MultiNLI task, we report the average accuracy of validation_matched and validation_mismatched datasets. For the SNLI task, we report accuracy on the test set. The results are presented in Table 5. AoE consistently outperforms BERT and SimCSE. It suggests that AoE can well capture subtle semantics via training and thus benefit NLI tasks.
Real and Imaginary Text Embeddings in Cosine Saturation Zone. We then study what is encoded
in imaginary text embeddings to tackle text pairs in cosine saturation zones. To that end, we focus on the data in the STS-B test set’s saturation zone (weighted similarity score > 0.95) and visualize them in a 2D plot using t-SNE (Van der Maaten and Hinton, 2008). Figures 3a and 3b show that imaginary text embeddings are more scattered than the real ones. To probe into the results, we draw lines between 5 sample text pairs and observe that the lengths of the lines, i.e., distances between text pairs, are larger for imaginary text embeddings than for real ones. For instance, consider one of the sample text pairs Ukraine to implement unilateral ceasefire and Ukraine offers unilateral ceasefire. The real distance is 11.8, while the imaginary distance is 22.1. This larger imaginary distance better reflects the subtle difference between “to
implement” and “offers”. AoE optimizes the angle differences to encode such subtle differences in the imaginary embeddings, resulting in better effectiveness. Figure 3c further supports this observation, as it shows that distances between text pairs are greater for imaginary text embeddings. This can be explained by the tendency that real text embeddings primarily capture appearance semantic differences, which can be influenced by saturation zones. Meanwhile, imaginary text embeddings specialize in capturing subtle semantic differences and help mitigate the negative effects of cosine saturation zones. We also visualize the full STS-B test set in Appendix D and have similar observations.
Text Embedding Distributions. Finally, we examine embedding distributions and how they align with the human senses. Figure 4 depicts the density plots of the cosine similarities between text pairs in the STS-B test set. Figure 5 shows the golden (human-labeled) scores, where human annotations are evenly distributed across varying similarity levels. However, Figure 4 implies that SimCSE and SBERT tend to focus their predictions within larger similarity intervals; in contrast, AoE’s distribution leans towards the left, indicating its ability to utilize a broader range to diversify similarity predictions. It could be attributed to AoE’s angle optimization,
allowing imaginary embeddings to reflect subtle semantic differences. As a result, AoE’s distribution aligns more closely with the humans’ distribution.

heading: 6 Conclusion
text: In this paper, we have presented a novel text embedding model called AoE, which optimizes the angle difference in complex space to mitigate the negative effects of cosine saturation zones. To comprehensively evaluate AoE with STS tasks, we have introduced a GitHub Issues Similarity Dataset for long-text STS evaluation. Extensive experiments have suggested that AoE outperforms baselines, indicating that AoE can produce high-quality text embeddings and benefit various downstream tasks.
Ethics Statement
In this paper, we present a newly developed longtext STS dataset called GitHub Issue Similarity (GIS). The data collection process for GIS follows the guidelines of GitHub, and we use the official GitHub API to collect the necessary data. We have carefully reviewed the data and are confident that there are no ethical issues, such as offensive content. All repositories included in the GIS dataset are open source.
Limitations
One limitation of AoE lies in its performance improvement on our proposed long-text STS dataset GIS is comparatively lower than its performance on short-text STS tasks. We plan to improve AoE’s performance on long-text STS tasks in future work.

heading: Acknowledgements
text: This work is supported by the NSFC Young Scientists Fund (Project No. 62006203), a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. PolyU/25200821), the Innovation and Technology Fund (Project No. PRP/047/22FX), and PolyU Internal Fund from RC-DSAI (Project No. 1-CE1E).
Here, we sincerely thank the reviewers and ACs for their valuable input, which has greatly improved our work.

heading: A Related Work
text: Table 6 provides a list of models that include cosine similarity in their objective functions. We observe that using cosine similarity to measure similarity in objective functions that pose a gradient vanishing challenge is quite common.

heading: B Details of GIS Dataset
text: We collected GitHub issues via the official GitHub API from the following popular 55 repositories:
Figure 6 shows an example of the proposed GIS dataset. We can see that the texts are long, and there is a higher overlap among duplicate issues than non-duplicate issues.
Figure 7 shows the data source count distribution of the proposed GIS. We can observe that there is a wide range of repositories in GIS, most of which consist of over 100 samples.
Table 7 presents the data split and data size of the proposed GIS dataset, and Figure 9 depicts a violin plot illustrating the token-level text length distribution. The violin plot reveals a substantial number of lengthy texts.
Figure 8 depicts the distribution of the n-gram overlapping for non-duplicate and duplicate issue pairs. We can see that the overlapping becomes more significant as the grams decrease. Additionally, the overlapping of duplicate issue pairs is slightly larger than non-duplicate pairs. Specifically, the average overlapping of non-duplicate
pairs for 1-gram, 2-gram, and 3-gram is 22, 12, and 9, respectively. Similarly, the average overlapping of non-duplicate pairs for 1-gram, 2-gram, and 3-gram is 26, 16, and 13, respectively. These statistics highlight the importance of using deep networks, even large language models, to identify duplicate and non-duplicate issues.

heading: C Detailed Derivation of Angle Difference
text: Complex division involves determining the angle difference and the factor in magnitude. Based on this, we calculate complex division between z and w as follows:
z w = γei∆θzw
γ = rz rw =
√ a2 + b2
::
√ c2 + d2
::
∆θzw = θz − θw,
(6)
where rz and rw represent the magnitudes of z and w, while θz and θw denote the respective angles of
ahttps://huggingface.co/collections/WhereIsAI/universal-angle-embeddings-663b0618ade1a39663e48190 bhttps://huggingface.co/collections/SeanLee97/angle-nli-sentence-embeddings-6646de386099d0472c5e21c0
z and w. Next, we compute the value of zw by the division rule in complex space as follows:
z w =
a+ b : i c+ d : i = (ac+ b : d : ) + (b
: c− ad : )i
c2 + d2 ::
. (7)
After that, we combine Eq. 6 and Eq. 7 to calculate the angle difference ∆θzw between z and w. By combing Eq. 6 and Eq. 7, we can obtain the following equation:
(ac+ b : d : ) + (b : c− ad : )i
c2 + d2 ::
= γei∆θzw . (8)
Then, we apply log(·) function to both sides, as follows:
log( (ac+ b : d : ) + (b
: c− ad : )i
c2 + d2 ::
) = log(γ) + i∆θzw. (9)
Next, we move log(γ) to the left side and replace
γ to
√ a2+b2: √ c2+d2: , as follows:
log
  (ac+ b : d : ) + (b : c− ad : )i
c2 + d2 ::
×
√ c2 + d2
::
√ a2 + b2
::
  = i∆θzw.
(10)
Finally, we simplify it and follow Sun et al. (2019) to use the real and imaginary text embeddings for the calculation to obtain Eq. 2

heading: D Discussion
text: Discussion of Training Time. To evaluate the efficiency of AoE, we compare its training time with SBERT and SimCSE. We train the models on the STS-B dataset for one epoch using a single GPU (Nvidia GeForce RTX3090 Ti). For BERTbase, the training times are 14.35, 14.93, and 14.94 seconds for SBERT, SimCSE, and AoE, respectively. For LLaMA7B , the training times are as follows: 1027.02 seconds for SBERT, 1027.67 seconds for SimCSE, and 1027.18 seconds for AoE. We find that AoE’s training time is similar to SBERT and SimCSE, suggesting that AoE can achieve better performance with comparable efficiency.
Analysis of Real and Imaginary Text Embeddings in Saturation Zone. Figure 10 displays a 2D plot using t-SNE, showing the full STS-B test set’s real and imaginary text embeddings. The imaginary text embeddings are more vertically scattered than the real text embeddings shown in Figure 10a and Figure 10b. The figures also include lines representing five sample text pairs, where the lines of the imaginary text embeddings are longer than those of the real text embeddings.
Figure 10c depicts the distance distribution of text pairs. It is noticeable that the distribution
of imaginary text embeddings is shifted towards higher distances compared to the real text embeddings. Moreover, the average distance of the imaginary text embeddings is also larger than that of the real text embeddings.
This evidence suggests that the imaginary text embeddings possess stronger capabilities in distinguishing semantic differences, thereby better discerning subtle semantic differences.

heading: E Pretrained Models of AoE
text: We open source multiple AoE embeddings for various scenarios, as listed in Table 8. The universal AoE embeddings (UAE) can be used for information retrieval, retrieval-augmented generation (RAG), semantic textual similarity, code similarity, clustering, classification, and many other applications. The AoE NLI embeddings can be used for semantic textual similarity.

heading: Future Work
text: We plan to improve AoE’s performance on long-text STS tasks in future work