Hyper-CL: Conditioning Sentence Representations with Hypernetworks (2024)

Abstract: While the introduction of contrastive learning frameworks in sentence representation learning has significantly contributed to advancements in the field, it still remains unclear whether state-of-the-art sentence embeddings can capture the fine-grained semantics of sentences, particularly when conditioned on specific perspectives. In this paper, we introduce HyperCL, an efficient methodology that integrates hypernetworks with contrastive learning to compute conditioned sentence representations. In our proposed approach, the hypernetwork is responsible for transforming pre-computed condition embeddings into corresponding projection layers. This enables the same sentence embeddings to be projected differently according to various conditions. Evaluation of two representative conditioning benchmarks, namely conditional semantic text similarity and knowledge graph completion, demonstrates that Hyper-CL is effective in flexibly conditioning sentence representations, showcasing its computational efficiency at the same time. We also provide a comprehensive analysis of the inner workings of our approach, leading to a better interpretation of its mechanisms. Our code is available at https://github.com/HYU-NLP/Hyper-CL.

heading: 1 Introduction
text: Building upon the established correlation between language model performance and computational capacity (Kaplan et al., 2020), there has emerged an undeniable trend towards the adoption of everlarger language models across a diverse range of NLP applications. This trend is also evident in the computation of sentence or text representations. Despite the ongoing popularity of compact encoders such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), there is a growing inclination to leverage the capabilities of recent,
†Equal contribution. ∗Corresponding author.
HyperNetwork
The perspective of "The speed of travel" The perspective of "The mode of transportation"
Vector Space
Both depict individuals engaging in outdoor activities, traversing natural landscapes. Cycling generally entails a faster pace than hiking.
The mode of transportation 𝐶!"#! The speed of travel 𝐶$%&
Sentence 1: A hiker navigates through a dense
Forest on a winding path.
Sentence 2: A cyclist pedals along a scenic mountain
trail, surrounded by lush greenery.
similar
dissimilar
generate… generate…
Figure 1: Illustration of our approach dubbed HyperCL. In the example, two sentences are provided along with two distinct conditions, chigh and clow. Specifically, chigh (orange) denotes a condition that results in the sentences being interpreted more similarly, whereas clow (blue) leads to a perspective in which the two sentences are understood as being relatively more distinct. The identical pair of sentences are projected into different subspaces that reflect the provided conditions.
larger language models, e.g., LLaMA-2 (Touvron et al., 2023), even breaking from the conventional roles of encoders and decoders. Consequently, the enduring challenge of finding a balance between performance and computational cost—a persistent issue in sentence representation learning (Reimers and Gurevych, 2019)—continues to be elusive.
In recent years, there has been a marked improvement in the quality of sentence embeddings, a progress primarily driven by the advent of contrastive learning frameworks (Kim et al., 2021; Gao et al., 2021; Chuang et al., 2022; inter alia). However, since the performance of these embeddings
700
is generally evaluated based on their ability to encapsulate the overall meaning of the corresponding sentences—as measured by benchmarks like STSB (Agirre et al., 2012; Cer et al., 2017) and MTEB (Muennighoff et al., 2023), it remains uncertain whether they adequately capture information relating to the various aspects of the source sentences.
For instance, consider the sentences (1) “A cyclist pedals along a scenic mountain trail, surrounded by lush greenery” and (2) “A hiker navigates through a dense forest on a winding path, enveloped by the tranquility of nature”. In terms of “The mode of transportation”, these sentences should be perceived as similar since both depict individuals engaging in outdoor activities, traversing natural landscapes. However, regarding “The speed of travel”, they should be differentiated, as cycling generally entails a faster pace than hiking. Deshpande et al. (2023) reported that current models for sentence embeddings face challenges in recognizing the fine-grained semantics within sentences. In other words, the existing models struggle to accurately detect the subtle shifts in sentence nuances that occur when conditioned on specific criteria.
In the literature, three prevalent approaches have been established for constructing conditioned representations (Deshpande et al., 2023), particularly in estimating their similarity (see Figure 2). The first is the cross-encoder approach, which encodes the concatenation of a pair of sentences (s1, s2) with a condition (c), i.e., [s1; s2; c].1 The second method
1[·; ·] represents the concatenation operation.
is the bi-encoder architecture, computing separate representations of sentences s1 and s2 with the condition c—[s1; c] and [s2; c]. Despite their simplicity, both approaches share a notable limitation: the representation must be computed for every unique combination of sentences plus a condition.
On the other hand, the tri-encoder architecture utilizes pre-computed embeddings of sentences s1 and s2 along with the condition c. It then employs a separate composition function responsible for merging the semantics of the sentence and condition. Considering that the embeddings for each component can be cached and reused, this approach offers enhanced long-term efficiency. The tri-encoder architecture, despite its potential, falls short in performance compared to the bi-encoder. This is primarily due to its inherent limitation, which is the inability to model explicit interactions between sentences and conditions during the representation construction process. Therefore, there is a need to propose a revised version of the tri-encoder architecture that improves its functionality without substantially sacrificing its efficiency.
In this work, we present Hyper-CL, a method that integrates Hypernetworks (Ha et al., 2017) with Contrastive Learning to efficiently compute conditioned sentence representations and their similarity. As illustrated in Figure 2d, our proposed approach is derived from the tri-encoder architecture. It introduces an additional hypernetwork tasked with constructing a condition-sensitive network on the fly. This network projects the original sentence embeddings into a specific condition subspace. Fig-
ure 1 illustrates the effectiveness of Hyper-CL in dynamically conditioning pre-computed sentence representations according to different perspectives.
We demonstrate the effectiveness of Hyper-CL by significantly reducing the performance gap with the bi-encoder architecture in the Conditional Semantic Textual Similarity (C-STS) and Knowledge Graph Completion (KGC) tasks. In particular, for C-STS, Hyper-CL demonstrates an improvement of up to 7.25 points in Spearman correlation compared to the original tri-encoder architecture. Furthermore, compared to the bi-encoder approach, our method shows superior efficiency by reducing the running time by approximately 40% on the CSTS dataset and 57% on the WN18RR dataset.

heading: 2 Background and Related Work
text: In this paper, conditioning refers to the presence of two or more signals, each represented as a natural language expression c. These signals impact the interpretation of a sentence s, highlighting a specific aspect of the sentence (Galanti and Wolf, 2020). Here, we describe two representative tasks that involve conditioning, along with an introduction to the concept of hypernetworks.
Conditional Semantic Textual Similarity (CSTS) (Deshpande et al., 2023) is a task composed of four elements in one quadruplet: two sentences s1 and s2, a condition c to consider when calculating the similarity between the two sentences, and a similarity score y. Unlike the original Semantic Textual Similarity (STS) dataset (Agirre et al., 2012; Cer et al., 2017), C-STS computes similarity scores for the same sentence pair s1 and s2 under two distinct conditions chigh and clow. The similarity scores of the conditioned sentences are expected to be high for chigh and low for clow. Therefore, a model for this task is required to compute distinct representations for the same sentence under two different viewpoints. To this end, a few basic architectures illustrated in Figure 2 have been proposed by Deshpande et al. (2023). Our objective is to present a revision to the previous methods, pursuing the balance between performance and computational efficiency.
Knowledge Graph Completion (KGC) is the task focused on automatically inferring missing relationships or entities in a knowledge graph. The knowledge graph is represented as a set of triplets (h, r, t), consisting of a head entity h, a relation r, and a tail entity t. Link prediction, a subtask in
KGC,2 aims to uncover unestablished yet plausible and novel relationships between entities (Bordes et al., 2013; Toutanova and Chen, 2015). When given a head entity and a relation, the task of identifying the most suitable tail entity is known as head entity prediction. Conversely, the task of determining the appropriate head entity when a tail entity and relation are provided is termed tail entity prediction.
While two types of methodologies are generally available for KGC—embedding-based methods and text-based methods—our primary focus is on text-based methods that rely on the processing of textual information by language models. We further categorize them into three types: cross-encoder, encoder-decoder, and bi-encoder architectures. Approaches such as KG-BERT (Yao et al., 2019) and MTL-KGC (Kim et al., 2020), which concatenate all triple elements (i.e., [h; r; t]), are classified as cross-encoder. Methods like StAR (Wang et al., 2021) and SimKGC (Wang et al., 2022), which separately embed [h; r] and t in tail prediction tasks, are classified as bi-encoder. Lastly, GenKGC (Xie et al., 2022) and KG-S2S (Chen et al., 2022), which directly generate tail entity text based on the remaining [h; r], are classified as encoder-decoder.
On the other hand, hypernetworks refer to a type of neural network that generates the weights or parameters for another neural network, known as the primary network (Ha et al., 2017; Chauhan et al., 2023; Majumdar et al., 2023). In essence, a hypernetwork enables the dynamic construction of the primary network, allowing its function to adapt flexibly based on the input or condition. For instance, Galanti and Wolf (2020) demonstrate that, even with a compact primary network, hypernetworks can effectively learn and apply diverse functions for various inputs, provided the hypernetwork itself is sufficiently large. In our settings, we also endeavor to harness the advantages of hypernetworks, ensuring that conditioned sentence representations are dynamically computed and adapted in response to changing conditions.

heading: 3 Proposed Method: Hyper-CL

heading: 3.1 Motivation
text: As briefly mentioned in §1, current approaches to sentence conditioning demonstrate a clear trade-off between performance and computational efficiency.
2While this work focuses solely on link prediction, note that other relevant tasks exist within the domain of KGC.
Approaches that enable direct interaction between a sentence and a condition within encoders—the cross-encoder and bi-encoder architectures—gain enhanced performance but at the cost of reduced efficiency. In contrast, the tri-encoder architecture, while enabling efficient conditioning with pre-computed sentence and condition embeddings, tends to have inferior performance compared to its counterparts. Note that this trade-off becomes more pronounced as the number of sentences and conditions to be processed increases.
Formally, consider a computationally intensive language model-based encoder f , and a less demanding composition network g, which is required in the case of the tri-encoder architecture. In terms of the bi-encoder, obtaining the set of every possible conditioned embedding H = {hsc|∀s∀c, s ∈ S, c ∈ C} necessitates |S| × |C| repetitions of the heavy operation imposed by f , where |S| denotes the total number of sentences and |C| represents the total number of possible conditions. In contrast, the tri-encoder architecture only requires heavy operations by f for each sentence s and condition c just once. This implies that only |S|+ |C| heavy operations are needed, followed by |S| × |C| lightweight operations by g to obtain the conditioned embeddings. As a result, the tri-encoder and its variants
become more efficient if the cost of computing g is markedly lower than that of f , thereby amortizing the cost for computing H.
In this work, we aim to develop a new architecture for sentence conditioning that inherits the efficiency merits of the tri-encoder architecture, while simultaneously outperforming the original in terms of performance. To achieve this, we propose the use of hypernetworks to implement g, facilitating the dynamic construction of conditioning networks while maintaining reasonable cost-efficiency.

heading: 3.2 Framework and Training Procedure
text: The framework of Hyper-CL and its training procedure are listed as follows (also see Figure 3):
1. First, it computes the embeddings of a sentence s and a condition c using the same embedding model f : hs = f(s) and hc = f(c).
2. Given the condition embedding hc, a hypernetwork q : RNh → RNh×Nh outputs a linear transformation matrix Wc for conditioning: Wc = q(hc).3
3. We encode the condition-aware sentence embedding hsc based on the matrix Wc and the sentence embedding hs: hsc = Wc · hs.
4. For training, we perform contrastive learning with the conditional sentence embeddings hsc, whose details are explained in the following.

heading: 3.3 Contrastive Learning in Subspaces
text: The conditioning network composed of Wc is a linear neural network. In other words, it can be interpreted as a linear transformation function g : RNh → RNh , mapping from the original semantic space of sentence embeddings to a specialized condition subspace. We demonstrate that conducting contrastive learning within the subspace of specific viewpoints yields greater effectiveness compared to performing the same process in the general space. We apply separate task-oriented contrastive learning objectives for the tasks, C-STS and KGC.
C-STS The C-STS task entails providing conditions chigh and clow for two sentences, s1 and s2. This setup induces different interpretations of the relationship between the two sentences—one being more similar under chigh and the other more dissimilar under clow. In a given instance from the dataset,
3Nh: The dimensionality of sentence embeddings, which is equal to the size of the hidden states of an encoder model.
Hyper-CL generates two pairs of conditioned sentence embeddings, i.e., (hs1chigh , hs2chigh) and (hs1clow , hs2clow). Since these pairs correspond to positive and negative pairs in the contrastive objective, we directly utilize them for training.
Considering that the C-STS dataset already contains gold-standard similarity values for the two sentences under each condition, it seems reasonable to employ the Mean Squared Error (MSE) objective in conjunction with contrastive learning. However, as training progresses, we can speculate that MSE objectives that utilize labels will provide relatively more fine-grained granularity compared to contrastive objectives that do not. Therefore, to mitigate the relatively strong impact of the contrastive objective, we apply the InfoNCE (Oord et al., 2018) loss with high temperature, as follows:
LCL = − log e ϕ(hs1chigh ,hs2chigh )/τ
e ϕ(hs1chigh ,hs2chigh )/τ +eϕ(hs1clow ,hs2clow )/τ ,
where ϕ is the cosine similarity function and τ is a temperature hyperparameter. The MSE objective is as follows:
LMSE = ∥ϕ(hs1c, hs2c)− y∥22,
where c can be either chigh or clow. By combining the two above formulas, the final form of our training objective for C-STS becomes:
L = LMSE + LCL.
Note that L is averaged over data instances in the training set.
KGC We follow the setting of SimKGC (Wang et al., 2022), except that we leverage Hyper-CL instead of the bi-encoder architecture. For each triplet of head entity, relation, and tail entity (h, r, t), we treat entities as sentences and relations as conditions, framing KGC as a conditioning task. Furthermore, given that Hyper-CL is simple and flexible enough to adopt various techniques from SimKGC, such as the use of self-negative, pre-batch negatives, and in-batch negatives, we decide to apply these tricks to our method as well. In conclusion, the final training objective for KGC is as follows (refer to Wang et al. (2022) for more details):
LCL = − log e (ϕ(hhr,ht)−γ)/τ
e(ϕ(hhr,ht)−γ)/τ+ ∑N
j=1 e ϕ(hhr,ht′ )/τ
where hhr, ht and h′t are the relation-aware head embedding, tail embedding, random embedding
(i.e., self-negative, pre-batch negative, and in-batch negative), respectively. The relation-aware head embedding corresponds to a conditional sentence embedding. γ is an additive margin, ϕ is the cosine similarity, and τ is a learnable parameter.

heading: 3.4 Optimization of Hypernetworks
text: In the original formulation presented in §3.2, the number of parameters for the hypernetwork q : RNh → RNh×Nh is the cube of Nh, which could lead to cost inefficiency. To address this issue, we propose decomposing the network into two lowrank matrices, drawing inspiration from low-rank approximation (Yu et al., 2017; Hu et al., 2021). In particular, we introduce two smaller hypernetworks of the same size: q1 : RNh → RNh×NK and q2 : RNh → RNh×NK to generate two low-rank matrices Wc1 = q1(hc) and Wc2 = q2(hc), where Nk is much smaller than Nh. We then obtain the final matrix through their product: Wc = Wc1W Tc2. 4

heading: 3.5 Caching Conditioning Networks
text: In the tri-encoder architecture, once computed, the sentence and condition embeddings hs and hc can be cached and subsequently reused whenever conditioned embeddings related to them need to be computed. Hyper-CL, building upon the tri-encoder framework, not only inherits this advantage but also further improves time efficiency by caching the parameters of the entire conditioning networks Wc = q(hc) generated by the hypernetwork. It is important to note that this approach is viable because the computation of the matrix Wc depends solely on hc, without requiring any other inputs.

heading: 4 Experiments
text: We apply Hyper-CL to various embedding models (i.e., encoders) and fine-tune them on the target task, denoting the resulting models with the subscript hyper-cl. If the rank of the hypernetwork (Nk) is different from Nh, we denote this value as hyperK-cl. We set K as 64 and 85, for the base and large models respectively. A detailed explanation for the selection of K can be found in the appendix B. We show the effectiveness of HyperCL by evaluating it on two downstream tasks.

heading: 4.1 Conditional Semantic Textual Similarity
text: We use DiffCSE (Chuang et al., 2022) and SimCSE (Gao et al., 2021), adaptations of RoBERTa (Liu
4A similar method was proposed in Majumdar et al. (2023).
et al., 2019), for the embedding model f . Note that the key difference between the original tri-encoder architecture and Hyper-CL lies in the composition network, g. The original uses the simple Hadamard product, while Hyper-CL employs hypernetworks to learn linear layers for this composition.
Table 1 summarizes the results of Hyper-CL in addition to baselines on C-STS. Compared to the tri-encoder baselines, Hyper-CL demonstrates improvements with up to a 7.25-point increase in Pearson correlation when based on SimCSEbase. This reduces the performance gap between the biencoder and tri-encoder from 13.3 to 6.05 points.
In addition, even when Hyper-CL is developed with low-rank approximation (i.e., hyper64-cl, hyper85-cl), its performance remains consistent. This indicates that the memory usage of hypernetworks can be effectively controlled, while both performance and time efficiency are maintained.

heading: 4.2 Knowledge Graph Completion
text: In KGC, the link prediction task entails computing relation-aware embeddings for head or tail entities and subsequently retrieving the top-K embeddings based on their similarity scores. We consider two datasets for KGC: WN18RR (Bordes et al., 2013) and FB15k-237 (Toutanova and Chen, 2015).
We employ text-based KGC models as baselines for evaluation. Specifically, we use the SimKGC model that leverages all negatives (i.e., in-batch negative, pre-batch negative, and self-negative), which is also true when applying Hyper-CL to
SimKGC. We consider an extra baseline of applying the tri-encoder architecture to SimKGC with different g—(1) Hadamard: performs the Hadamard product between the representations of a condition c and a sentence s: g1(hc, hs) = hc⊙hs. (2) Concatenation: merge the two vectors, apply a dropout function, and halve the dimension using a linear layer: g2(hc, hs) = W · d([hc;hs]).
Table 2 presents the outcomes of our method and baselines on KGC, measured by MRR (Mean Reciprocal Rank) and Hits@K. BERTbase is leveraged for the embedding model f . As a result, SimKGChyper-cl, representing the application of Hyper-CL to SimKGC, shows that there is no significant difference in performance compared to the original SimKGC, especially in terms of Hits@10. Even for SimKGChyper64-cl, while there is a slight decrease in performance, it still yields competitive results and does not significantly trail behind other baselines. Moreover, the performance of other methods in the tri-encoder architecture falls significantly short of Hyper-CL’s. It is worth noting that our implementation is based on the tri-encoder architecture, which guarantees significantly more efficiency in running time compared to the original SimKGC. The details of this analysis are in §5.1.

heading: 5 Analysis

heading: 5.1 Efficiency Comparison between Bi-Encoder and Tri-Encoder
text: To assess the running time efficiency of Hyper-CL, enabled by its caching capability, we compare the
execution time of our method with that of the biencoder and tri-encoder architectures. We measure execution time, cache hit rate, and memory usage in scenarios with caching enabled. Initially, the cache is empty, and embeddings are added to the cache upon each cache miss.
Specifically, we estimate the inference time required for the C-STS and WN18RR datasets. To simulate a realistic scenario where the number of data samples to be processed is substantial, we utilize all dataset splits (training, validation, and test) from the datasets. For the C-STS dataset, we also evaluate the results of SimCSEhyper64-cl as a solution for cases with large cache sizes. For Hyper-CL, the transformation matrix Wc is saved instead of embeddings. We set the batch size as 1 to minimize the overhead of cache storage and retrieval.
As observed in Table 3 and 4, Hyper-CL reduces the running time by approximately 40% on the C-STS dataset and 57% on the WN18RR dataset than the bi-encoder architecture. Compared to the naïve tri-encoder architecture, Hyper-CL requires slightly more time, but we believe this is acceptable given its significantly improved performance.
In terms of cache hit rate, the tri-encoder architecture, including Hyper-CL, achieves a much higher rate as it embeds each input separately, whereas the bi-encoder architecture has a lower rate because both the sentence and condition com-
bination must match to result in a cache hit. Considering the significant gap in cache hit ratio between the bi-encoder and tri-encoder architectures, we expect that the efficiency of Hyper-CL will be more pronounced when deployed to process real-time streaming data from a large pool of users. In such scenarios, the diversity of input sentences and their conditions (relations) would be much higher than in our experimental settings, implying that the efficiency gap between the bi-encoder and tri-encoder architectures will be more severe.5

heading: 5.2 Analysis of Embedding Clusters
text: In this subsection, we explore the impact of HyperCL on generating conditioned sentence representations by visualizing the computed embeddings and analyzing them using clustering tools.
We visualize the vector space of sentence embeddings before and after transformation by Wc, which is the weight of a linear layer from the Hyper-CL’s hypernetwork. This helps us observe if embeddings based on the same condition cluster after transformation, indicating proper differentiation based on conditions. For this analysis, we choose three sets
5The ratio of the number of unique items required to be cached by the tri- and bi-encoder architectures, i.e., (|C| + |S|)/(|C|×|S|), will approach close to 0 if |C| and |S| become very large. This implies that in practice, it is infeasible to utilize caching techniques for the bi-encoder architecture if the sentences and conditions of interest are sufficiently diverse.
of 20 random sentences from the C-STS validation dataset, with sentences within each group sharing the same conditions. The three conditions we select are: ‘The number of people’, ‘The sport’, and ‘The name of the object’. As expected, Figure 4 shows that sentence embeddings transformed with the same Wci form clusters, meaning each embedding has projected to respective subspaces.
We complement the visual analysis with a quantitative evaluation. We perform K-means clustering on the sentence embeddings before and after transformation.6 Following the clustering, we compute the average impurity (entropy) of each sentence group, where a lower value suggests better conditioning of sentence embeddings. Formally, the impurity I based on the entropy of each (conditional) sentence group E(i) is given by:
I = ∑
i |Ci| |S| E(i) = − ∑ i |Ci| |S| ∑ j |Lij | |Ci| log |Lij | |Ci| ,
6K is set to 3, same as the number of chosen conditions.
where |S| is the total number of sentences, |Ci| is the number of sentences that should be labeled as condition i, and |Lij | represents the number of examples clustered as j by K-means clustering within the ith (conditional) sentence group. We discover that after projection done by Hyper-CL, I changes from 0.739 to 0.270, indicating that Hyper-CL effectively projects sentence embeddings into distinct subspaces based on different conditions.

heading: 5.3 Generalization Capabilities of Hyper-CL
text: We examine the generalization capabilities of Hyper-CL, focusing on its ability to generalize to unseen conditions during training. This enables a fine-grained evaluation of the conditioning ability and confirms its feasibility in realistic settings.
For the targeted study, we define two separate subsets of the C-STS validation set.7 The first category is referred to as the ‘unseen’ dataset, which consists only of data instances with conditions not present during training. The second is named the ‘seen’ dataset, which comprises data instances with
7Due to the absence of publicly available gold-standard labels for the C-STS test set, we use the validation set for evaluation purposes. Additionally, we partition the C-STS training data in a 9:1 ratio to establish a new validation set. Note also that we concentrate on C-STS in this section, as WN18RR and FB15K-237 do not provide ‘unseen’ conditions.
conditions already seen in the training phase. Statistically, the number of data instances for the ‘unseen’ portion is 731 (25.79% of the overall dataset), and for the ‘seen’ portion, it is 2,103.
Experimental results on the ‘overall’, ‘unseen’, and ‘seen’ datasets are listed in Table 5. For the embedding model f , we employed SimCSElarge. Compared to the original tri-encoder (the first row), Hyper-CL shows a clear performance improvement of 16-22 points in both ‘seen’ and ‘unseen’ settings, with particularly superior performance in the ‘unseen’ setting. These findings highlight the superior generalization capabilities of Hyper-CL, enabling it to excel at handling unseen data.

heading: 5.4 Ablation Study on Contrastive Learning
text: In §3.3, we argued that the joint utilization of hypernetworks and contrastive learning yields the best performance among the available options. To verify this, we evaluate four different variations of the tri-encoder architecture on C-STS, whose details are as follows: (1) SimCSEbase: the tri-encoder architecture trained only with LMSE ; (2) SimCSEbase+cl: the tri-encoder trained with both LMSE and LCL but hypernetworks excluded; (3) SimCSEbase+hyper64: a variant of Hyper-CL (K=64) but trained only with LMSE ; (4) SimCSEbase+hyper64−cl: a normal Hyper-CL with low-rank approximation (K=64). For a fair comparison, we ensure that the total number of parameters for each variant remains consistent, guaranteeing equal expressive power.
Table 6 shows that the contrastive learning objective (LCL) is more effective when combined with hypernetworks. This trend is clearly observed when comparing the performance increase from SimCSEbase+hyper64 to SimCSEbase+hyper64-cl and that from SimCSEbase to SimCSEbase+cl.

heading: 5.4.1 Why is Contrastive Learning More Effective with Hypernetworks?
text: The weight matrix Wc = q(hc), generated by the hypernetworks of Hyper-CL, is responsible for a
linear transformation of a sentence embedding. On the other hand, the Hadamard product of a sentence embedding and a condition embedding, which is computed in the original tri-encoder architecture, can also be considered as a linear transformation, formulating the condition embedding hc as a diagonal matrix Wc′ = diag(hc).
To gauge the expressiveness of the two different transformations induced by Wc and Wc′ , we calculate the variance of the Frobenius norm of these matrices during inference on a subset of the C-STS validation set. For matrices with varying valid element counts (i.e., Wc and Wc′), we normalize their Frobenius norm by dividing by the square root of the number of valid elements. Experimental results show that the variance of the Frobenius norm, a measure of the matrices’ expressive power, is significantly higher (0.0248) for Hyper-CL’s transformations (Wc) compared to the Hadamard product (0.001; Wc′). These findings imply that hypernetworks endow the transformation with enhanced expressive power. Consequently, it is reasonable to expect that the contrastive learning process that leverages hypernetworks would also exhibit greater effectiveness.

heading: 6 Conclusion
text: We propose Hyper-CL, a method that combines hypernetworks with contrastive learning to generate conditioned sentence representations. In two representative tasks requiring conditioning on specific perspectives, our approach successfully narrows the performance gap with the bi-encoder architecture while maintaining the time efficiency characteristic of the tri-encoder approach. We further validate the inner workings of Hyper-CL by presenting intuitive analyses, such as visualizations of the embeddings projected by Hyper-CL. In future work, we plan to explore a broader range of applications for Hyper-CL and to investigate its refinement.
Limitations
We have only explored applying our approach to encoder models, leaving room for applications on decoder models. Additionally, despite the variety of existing contrastive learning methodologies, we adhere to utilizing the contrastive learning objectives provided by the tasks.
Ethics Statement
In this study, we utilized models and datasets publicly available from Huggingface. All datasets for evaluation are open-source and comply with data usage policies. However, some datasets (e.g., FB15k-237) are derived from Freebase, a large collaborative online collection that may contain inherently unethical information. We conducted a thorough inspection to check if our dataset contained any unethical content. No harmful information or offensive topics were identified during the human inspection process.

heading: Acknowledgements
text: This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.RS-2020-II201373, Artificial Intelligence Graduate School Program(Hanyang University)). This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) under the artificial intelligence semiconductor support program to nurture the best talents (IITP-2024-RS-202300253914) grant funded by the Korea government(MSIT). This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(*MSIT) (No.2018R1A5A7059549). *Ministry of Science and ICT.

heading: A Training Details
text: In this section, we describe the hyperparameters used for training Hyper-CL on the two evaluation tasks employed in this paper. We implemented Hyper-CL using the Transformers package (Wolf et al., 2020). For both tasks, Hyper-CL utilized the [CLS] token embedding computed by an encoder model as a sentence representation.
Conditional Semantic Textual Similarity (CSTS): We conducted a hyperparameter search over learning rates ∈ {1e-5, 2e-5, 3e-5}, weight decays ∈ {0.0, 0.1}, and temperatures ∈ {1.0, 1.5, 1.7, 1.9}. The hyperparameter set yielding the best scores on the C-STS validation set with three random seeds was used for the final evaluation of the test set. As a result, we adopted the hyperparameters as shown in Table 7.
Knowledge Graph Completion (KGC): We utilized the same set of hyperparameters proposed in SimKGC (Wang et al., 2022).

heading: B Ablation Study on the Selection of K
text: The selection of K for constructing lightweight hypernetworks is closely related to the size of sentence embeddings. We empirically evaluated the validation set to determine suitable values for K by dividing the embedding sizes of the base (768) and large (1024) encoder embeddings with various divisors (1, 4, 8, 12, 16, 24). According to Table 8, we observed that for SimCSElarge, the performance difference between K=128 and K=85 is just 0.07 points, while trainable parameters increase 1.5x. In conclusion, we found that setting the divisor to 12 (resulting in K values of 64 and 85 for the base and large models, respectively) achieves an optimal balance between performance and efficiency.

heading: Future Work
text: In future work, we plan to explore a broader range of applications for Hyper-CL and to investigate its refinement. Limitations
We have only explored applying our approach to encoder models, leaving room for applications on decoder models. Additionally, despite the variety of existing contrastive learning methodologies, we adhere to utilizing the contrastive learning objectives provided by the tasks