Triple-Encoders: Representations That Fire Together, Wire Together (2024)

Abstract: Search-based dialog models typically reencode the dialog history at every turn, incurring high cost. Curved Contrastive Learning, a representation learning method that encodes relative distances between utterances into the embedding space via a bi-encoder, has recently shown promising results for dialog modeling at far superior efficiency. While high efficiency is achieved through independently encoding utterances, this ignores the importance of contextualization. To overcome this issue, this study introduces triple-encoders, which efficiently compute distributed utterance mixtures from these independently encoded utterances through a novel hebbian inspired co-occurrence learning objective in a self-organizing manner, without using any weights, i.e., merely through local interactions. Empirically, we find that triple-encoders lead to a substantial improvement over bi-encoders, and even to better zeroshot generalization than single-vector representation models without requiring re-encoding. Our code 1 and model2 are publicly available.

heading: 1 Introduction
text: Traditional search-based approaches in conversational sequence modeling like ConveRT (Henderson et al., 2020) represent the entire context (query) in one context vector (see Figure 1). This has two major drawbacks: (a) Recomputing the entire vector at each turn is computationally expensive, and (b) it is difficult to compress the context’s relevant information for any possible candidate response into a single vector. Furthermore, the encoder models are limited to a maximum number of tokens, usually 512.
Curved Contrastive Learning (CCL) (Erker et al., 2023) demonstrated that it is possible to encode
1 UKPLab/Triple-Encoders 2 UKPLab/Triple-Encoders-DailyDialog
utterances separately in a latent space and accumulate sequence likelihood based on solely cosine similarity, thanks to treating cosine similarity not as a semantic but as a directional relative dialog turn distance measure between utterance pairs (through two sub-spaces representing a temporal direction: before and after). This relativistic approach tackles (a), by enabling sequential search with a constant complexity, as only the latest utterance needs to be encoded and computed during inference as shown in Figure 1. (b) Furthermore, each candidate utterance can interact with every independently projected utterance, allowing a richer interaction. However, encoding utterances indepen-
5317
dently means they are not contextualized, disregarding a crucial feature of conversation. An example is illustrated in Figure 2.
For the first time, in this paper we propose a method that contextualizes utterance embeddings in dialog sequences in a self-organizing manner, without the use of additional weights, i.e, merely through local interactions (in form of efficient vector algebra) between separately encoded utterances after appropriate pre-training. While previous work has shown that mean pooling is a strong method for sentence composition from tokens (Pagliardini et al., 2018; Reimers and Gurevych, 2019), we demonstrate that this can be generalized to a higher abstraction level: distributed pairwise sequential composition (illustrated in Figure 1). To realize this, we present triple-encoders, which segment the context space of CCL into two distinct latent spaces denoting the relative order of utterances in the context. By linearly combining (averaging) representations from these sub-spaces through a co-occurrence learning objective, we create new contextualized embeddings that we can incorporate into CCL, resulting in Contextualized Curved Contrastive Learning (C3L). At inference time, our method efficiently contextualizes independently encoded utterances based on solely local interactions (without any additional weights): Our method applies only (1) mean pooling, a (2) matrix multiplication for computing the similarity and one (3) summation (across the sequential dimension) operation to aggregate similarity scores.
While we focus on modeling dialog in this paper, the sequential modularity of our method can in principle be used for any text sequence. Pilot experiments on next sentence selection of children stories are reported in Appendix E, while we leave thorough exploration to future work. Our experiments are aimed at the following research questions:
RQ1: What is the effect of triple-encoder training (C3L) + triple encoder at inference compared to CCL?
RQ2: What is the effect of triple-encoder training (C3L) while encoding utterances at inference time without contextualization (like CCL)?
Our experimental results suggest that our approach improves substantially over standard CCL. Notably, our method outperforms ConveRT (Henderson et al., 2020) in a zero-shot setting while our method requires no additional learnable parameters for contextualization. While triple-encoder train-
ing alone improves the performance considerably (RQ2), using triple-encoder contextualization at inference time (RQ1) leads to additional performance gains while keeping linear complexity.

heading: 2 Related Work
text: We will start with related work on embedding compositionality, conversational sequence modeling and self-organizing maps. Next, we will describe retrieval methods that have been a motivation to our distributed representations. Lastly, we will discuss CCL as the core foundation of our work.

heading: 2.1 Composition and Self-Organization
text: Weight-less compositionality of embeddings is a well-studied problem for word representations (Mitchell and Lapata, 2008; Rudolph and Giesbrecht, 2010; Mikolov et al., 2013; Mai et al.), but has received little attention for larger text units such as sentences or utterances. For these, investigations are limited to small contexts such as pairwise sentence relations (e.g. NLI) (Sileo et al., 2019) or sentence fusion (Huang et al., 2023), and are outperformed by parameterized composition operators. In the context of conversational sequence modeling, conventional methodologies typically employ parameterized functions (learned weights) that act as an external force to contextualize utterance embeddings that are computed indepen-
dently (Liu et al., 2022; Zhang et al., 2022). As far as we are aware, this is the first method in which contextualization in conversational sequence modeling has been achieved solely through local interactions, without the reliance on additional weights, where all information is stored in the geometry of the latent space. This approach aligns with the self-organization principle found in nature that
describes the emergence of global order from local interactions between components of a system without supervision by external directing forces (Rezaei-Lotfi et al., 2019)
demonstrating how global order within dialogue sequences can emerge from localized interactions (mean pooling and cosine similarity) among utterance embeddings. The self-organization principle has previously been applied to machine learning in self-organizing maps (Kohonen, 1982).

heading: 2.2 Retrieval
text: Typically, neural response retrieval systems like ConveRT (Henderson et al., 2020) (see Figure 1) produce a single context embedding per turn that is then compared to candidate utterance embeddings. This leads to weak interactions with candidate utterances as not all information can be compressed into one vector. Previous work in retrieval has addressed the weak interaction of bi-encodings through several techniques. Previous work like MORES (Gao et al., 2020), PreTTR (MacAvaney et al., 2020) or PolyEncoders (Humeau et al., 2020) tackled this problem by encoding each candidate representation with a query via a late-stage selfattention mechanism to enable a richer interaction. Though this technique outperformed traditional biencoders, the attention mechanism does not scale with large search spaces. Another technique that was the inspiration for our average and maximum similarity based approaches is ColBERT (Khattab and Zaharia, 2020) and ColBERTV2 (Santhanam et al., 2022) which has shown that this concept works well on word token level.

heading: 2.3 Uncontextualized CCL via Bi-Encoder
text: We build upon the previous work on Curved Contrastive Learning (CCL) (Erker et al., 2023), a selfsupervised representation learning technique based on sentence embedding methods like SentenceBERT (Reimers and Gurevych, 2019). Similar to how our universe is made up of a stage between
space and time, CCL learns a stage between semantics and the directional relative turn distance of utterance pairs in multi-turn dialog. As Figure 3 illustrates, the resulting embeddings are inspired by the concept of relativity (Einstein, 1921): By embedding utterances with special before ([B]) and after ([A]) tokens into two distinct subspaces, directional temporal distances become relative to the observer. Concretely, as Figure 3 shows, when traveling through this space from t− 1 to the next turn t, CCL linearly decreases the similarity to every previous utterance us, s < t and increase the similarity to every us, s > t as part of the sequence. Formally, given a sequence of utterances u0, u1, ..., un, choose a window size w, then the pretraining objective of CCL is
cos(E([B] ui),E([A] uk)) = 1− k − i w ,
which we enforce through an MSE loss for 0 < k − i < w. E refers to a text encoder such as SBERT. We also refer to this model as the biencoder as it uses a dual encoder. This training objective together with directional and random hard negatives shows strong performance in sequence modeling and planning tasks (Erker et al., 2023). While ConveRT encodes t utterances at step t, resulting in an overall complexity of O(n2) utterance encodings, this approach only encodes one new utterance at every step, resulting in an overall complexity of O(n) utterance encodings.
However, we hypothesize that the lack of contextualization does not reflect the dialogs’ highly contextual dependency which prohibits even better performance. With our triple-encoder we address this core limitation as we will show empirically in this paper.

heading: 3 Contextualized CCL via Triple-encoders
text: We design an extension to the CCL framework that addresses the aforementioned issue while retaining the same order of encoding complexity at inference as the bi-encoder model (O(n)), resulting in Contextualized Curved Contrastive Learning (C3L).
To enhance the CCL embeddings with contextualization, two additional special tokens are added to the before space, [B1] and [B2]. These tokens denote the relative order of the utterances in the dialog, i.e. [B1]ui ∧ [B2]uj ⇔ i < j to add positional information. To model the interaction between these different context utterances, we choose a simple mean operation as shown on the right of Figure 4. Given that the [B1] and [B2] tokens create distinct representations, they’re effectively projecting the utterance into two different embedding spaces. By combining utterance states of [B1] and [B2] through a mean operation, a new combined state that carries information from both original states is created.
Erker et al. (2023) have shown that the curved scores (a linear decrease of similarity) outperform classic hard positives/negatives in sequence modeling and enhance sequential information that
enables better planning capability. To keep the phenomenon of moving in a relativistic fashion through a continuous temporal dimension with respect to the after space, we construct the pairwise utterance mixture distance to a following utterance as the average of the two individual distances, as defined in CCL. Formally, for window size w and 0 < i < j < k and k − i < w, if the distance between [B1]ui and [A]uk is 1− k−iw , and the distance between [B2]uj and [A]uk is 1− k−jw , then the joint representation of utterances in the before space should have distance norm(2 − 2k−(i+j)w ). Hence, we enforce for positive examples:
cos
( E([B1]ui) +E([B2]uj)
2 ,E([A]uk)
)
=norm ( 2− 2k − (i+ j)
w
) ,
(1)
where norm normalizes the values from [ 1w , 2− 3w ] to [ 1w , 1] via min-max scaling to match the range of cosine similarity. Figure 4 illustrates this procedure. Like in CCL, this objective can be computed efficiently when moving from step t− 1 to t, since only the last utterance has to be encoded at every inference step, as Figure 5 illustrates.

heading: 3.1 Co-occurrence Learning Through Hard Negatives
text: With the positive training examples from above, the model does not necessarily have to learn cooccurrence information, because it suffices to identify one context utterance in the input to reach a low training error. We introduce hard negative examples to mitigate this. By training every true context representation (both [B1] and [B2]) with one random utterance as hard negatives, we enable a novel co-occurrence learning paradigm that only lets a candidate representation (in the after space) wire to its mixed contextualized representation if both context representations fire in a sequence together. Hard negatives are constructed from random utterances ur, u′r sampled from the training set:
cos ( E([B1]ui)+E([B2]ur) 2 ,E([A]uk) ) = 0.0
cos ( E([B1]ur)+E([B2]ui) 2 ,E([A]uk) ) = 0.0
cos ( E([B1]ur)+E([B2]u′r) 2 ,E([A]uk) ) = 0.0
(2) These are generated for every 1 ≤ i < k and k − i < w.
We employ three additional components: First, as a preliminary step, the model is pre-trained as a bi-encoder, i.e., standard CCL. We found this to improve results slightly (Appendix B). Second, we employ the same auxiliary NLI learning objective as is used in standard CCL. Third, like in CCL, we indicate odd and even turns through additional speaker tokens, following (Erker et al., 2023).

heading: 4 Application of Curved Contrastive Learning
text: This section describes how the bi-encoder and triple-encoder are used after training to solve the two previously introduced tasks (Erker et al., 2023) sequence modeling (Section 4.1) and short-term planning (Section 4.2). Note, when using the tripleencoder as bi-encoder, we use the same setup as CCL bi-encoders.

heading: 4.1 Dialog Sequence Modeling
text: In sequence modeling, given a context prefix C = u1, . . . , uk from a dialog u1, . . . , un, the task is to find the true utterance uk+1 among a set of randomly sampled future utterances. For every candidate utterance uf ∈ UF , the relative likelihood p(uf |C) (similarity score) is computed. For evaluation, we measure the rank of p(uk+1|C) of all utterances in the test corpus at the same depth. The bi-encoder and triple-encoder differ in how p(u|C) is computed.

heading: 4.1.1 Bi-Encoder
text: Following Erker et al. (2023), with the bi-encoder the relative likelihood is computed as the cosine similarity between the candidate utterance and every context utterance (encoded separately) as
p(uf |C) = ∑
ui∈C cos(E([B]ui),E([A]uf )).
While the accumulation is very efficient and worked fairly well, we demonstrate that our triple encoder trained with C3L significantly improves the performance thanks to the extra contextualization.

heading: 4.1.2 Triple-Encoder
text: Similar to the bi-encoder we accumulate the likelihood of a sequence based on the pairwise mixed representations of the entire sequence length n, i.e. the training window size w does not apply during inference. We construct the relative likelihood for every candidate utterance uf ∈ UF for a context C := [u1, .., un] of length n as:
P (uf |C) = n−1∑ i=1 n∑ j=i+1 cos ( E([B1]ui)+E([B2]uj) 2 ,E([A]uf ) ) (3)
As the distributed pairwise mixed representations are superimposed (Equation 3), our embedding space emerges multiple local maxima in the latent space that enables a richer interaction for the candidate (after space) as shown in Figure 1. This stands in stark contrast to traditional search with only one context vector (like ConveRT (Henderson et al., 2020)) where the candidate space is mapped to this one global maximum. This late interaction lets us build upon previous token-based techniques
like ColBERT (Khattab and Zaharia, 2020). While Equation 3 effectively captures their averaging approach, in Appendix A we also experiment with their default maximum-based approach, which is 100 times slower than simple averaging in our experimental context.
With our contextualized representations, the number of representations grows to the triangular numbers (n(n+1)2 ) as shown in the arising triangle of Table 1 (in the appendix). However, the relative number of computations at each turn is strictly linear (shown by the relative growth). Therefore at each turn, we simply compute all additional states from t− 1 to t as one matrix multiplication of size (|UF |, e_dim) and (n, e_dim)T , where e_dim denotes the embedding dimension and |UF | the size of the candidate space. Lastly, we add the score matrix of step n− 1 and rank the candidates in the next step.
As the model is trained only on a fixed window size of w, taking the full triangle on longer sequence lengths might lead to unwanted distortions as distances out of this window are not part of the learning objective. Therefore, we also experiment with the l last rows of Table 1. In other words, we here only take the last l utterances in the [B2] space and contextualize them with the entire sequence in the [B1] space with the previously discussed order constraint [B1]ui & [B2]uj ⇔ i < j. Notably this l last rows version with linear complexity has the same efficiency as the entire average since we only compute the latest row at each turn (equivalent to l = 1 as shown in Table 1).

heading: 4.1.3 Efficiency Compared to CCL
text: When it comes to the number of transformer computations triple-encoders enable a similar relativistic state accumulation in sequence modeling as the traditional CCL. As Figure 5 demonstrates, during inference it is only necessary to encode the utterance at turn n with the [B2] token and average it with every previous utterance [[B1]u1, . . . , [B1]un−1]. Only in the next turn it is necessary to encode the new utterance with [B1] which can be done during the dialog partner’s turn. We provide a detailed complexity comparison between all approaches in Table 7 in the appendix.

heading: 4.2 Short-Term Planning
text: Erker et al. (2023) have shown that the sequential information of CCL is especially useful to determine whether a candidate utterance is leading to
a goal over multiple turns by just measuring their relative distance. The short-term planning experiments are conducted as follows: A dialog context c[: l] of fixed length l is given to a dialog transformer, which generates 100 utterances for each context. The true utterance of dialog at that position is added to these 100 candidates. We then rank all candidates by cosine similarity between the candidate (in the before space) and the goal g (in the after space). This goal g = c[l + gd] is defined as the utterance of the true dialog gd turns in the future.

heading: 4.2.1 Bi-Encoder
text: Here the true utterance should be closest to the goal in the imaginary space. We measure the cosine similarity between every candidate (in the before space) and the goal (in the after space) as ∀c ∈ Candidates : cos(E([B]c, )E[A]g)) where we rank the score of the true utterance. Notably, as mentioned by (Erker et al., 2023) the goals are picked at fixed but arbitrary positions. Hence, we can not ensure low ambiguity: For example, a response like "ok, okay" as the goal is achievable through various dialog paths, making 100% accuracy unrealistic.

heading: 4.2.2 Triple-Encoder
text: Through the relativistic property and the independence assumption in classical imaginary embeddings, the candidates are in no interaction with the context. With the triple-encoders, this shortcoming can be surpassed (1) through contextual aware training and (2) through contextual combination at inference. In particular, instead of determining the likelihood of candidates leading us to the goal over multiple turns as simple cosine similarity between the candidate and the goal, we combine the likelihood of the goal independently with its contextualized version. In particular, by the mean of the candidate [B2]c with every context utterance [[B1]u1, . . . , [B1]un] as the linear combination. The relative likelihood for a candidate c to the other candidates is summarized as
cos(E([B2]c),E([A]g))+
1
n
n∑
i=1
cos
( E([B1]ui) +E([B2]c)
2 ,E([A]g)
)
(4)
Here n is the entire context length. We then rank the true utterance among the candidates.

heading: 5 Experiments
text: Our experiments are conducted on the newly introduced GTE (general-purpose text embedding) model (Li et al., 2023) as well as the RoBERTabase models (Liu et al., 2019) from the CCL paper (Erker et al., 2023) which we will use as baselines. Furthermore, we add a non-relativistic approach for sequence modeling evaluation, ConveRT (Henderson et al., 2020). Apart from that, we investigate ablations to our introduced triple-encoders using the same special tokens but without the curved property of the temporal dimension akin to curved contrastive learning as well as every component separately. The sequence modeling models are trained and evaluated on two datasets, DailyDialog (Li et al., 2017) and MDC (Li et al., 2018), a task-oriented dialogue dataset. The models are also evaluated on zero-shot performance on PersonaChat (Zhang et al., 2018). We will furthermore evaluate the triple-encoder on short-term planning proposed by (Erker et al., 2023) where we evaluate our approach also with Hits@k. To compare to previous work we use the same setup, with candidates generated with DialoGPT (Zhang et al., 2020) (top_p = 0.8, temperature=0.8) on history lengths of 2, 5, 10 with goal distances of 1, 2, 3, 4. Our code including all hyperparameters can be found in our GitHub repository.

heading: 5.1 Self-Supervised Training
text: All models are finetuned versions of the state-ofthe-art text embedder GTE (Li et al., 2023). We pre-train our models with CCL (Erker et al., 2023), while we also experiment with training the tripleencoder from scratch. All models are trained on a window size of w = 5, a batch size of 32, a learning rate of 2 · 10−5, a weight decay of 0.01, utilize an Adam optimizer (Kingma and Ba, 2015) and use a linear warmup scheduler with 10% of the training data as warmup steps. We perform model selection on the validation set after 10 epochs of training.

heading: 6 Evaluation & Discussion
text: We start the evaluation with the sequence modeling performance of triple-encoder in Section 6.1, followed by the short-term planning performance in Section 6.2. We will address the research ques-
tion RQ1 by comparing triple-encoders trained with C3L to CCL bi-encoders. To answer RQ2 we will furthermore compare to C3L bi-encoders (triple-encoder as bi-encoder), e.g training with C3L (Section 3) but using the model at inference as bi-encoder by only using the [B2] token (Section 4.1.1). We provide a comprehensive analysis on different training setups in the Appendix B and an analysis of the contribution of every component in the triple-encoder setup during inference in sequence modelling (Appendix C). In summary, we find that pre-training with CCL (which includes directional negatives) and then continuing training with triples yields the best performance. Furthermore, all our model components bring a benefit.

heading: 6.1 Sequence Modeling

heading: 6.1.1 DailyDialog
text: We compare the previously discussed architectures in terms of the sequence modeling performance on the DailyDialog corpus over different context lengths in Figure 6 (left). We start with the average triple-encoder that beats all baselines across all context lengths with an average rank of 21.25, outperforming its non-curved (hard positives ablation) triple-encoder by 34.37% and CCL (with the same GTE encoder base) (RQ1) by 31.46%. When it comes to the different variations, we find that MaxSim (average rank of 20.16) on the entire triangle yields best performance within the context size of 5 (the training window). However, computing the maximum is 100 times slower, while performing only marginally better than averaging. Therefore, we recommend using average triple-encoders. Over context length of 5 we find that the l = 2 last rows variant of average triple-encoders performs best (Table 2). On this model we find in an ablation (Figure 7) that incorporating representations outside of the training window with size w = 5 gains a 9.58% lower average rank, demonstrating that our co-occurrence objective improves performance beyond its initial training window.

heading: 6.1.2 Triple-Encoders as Bi-Encoders
text: C3L demonstrate their versatility when used as bi-encoders (RQ2). With the triple-encoder achieving an average rank of 21.25, and the GTE biencoder (Erker et al., 2023) achieving 31.01, the performance of the triple-encoder when treated as a bi-encoder sits impressively closer to the tripleencoder than to the bi-encoder with an average rank of 25.48. Our evaluations suggest that the difference is not merely attributed to the negatives. An ablation study using bi-positives and triple negatives achieves an average rank of 27.30, indicating that the positives play a pivotal role in narrowing the gap to 25.48. This hints to a principle from neuroscience:
Neurons that fire together, wire together. (Hebb, 1949)
In the context of our triple-encoder, the cooccurrence of context utterances during training (as they "fire" together) leads to stronger associations or "wiring" between them in the embedding space, specifically by pushing co-occurring representations that wire together to a representation in the after space closer together. This leads to the phenomenon that even when processed separately, the embeddings have a stronger linear additivity
to the candidate (after space) when being superimposed. We investigate this in more detail in Appendix D. While training with triplets provides the model with rich contextual information, the persistence of learned associations during bi-encoder inference allows the model higher efficiency than triple-encoders with contextualization.

heading: 6.1.3 Task-Oriented Dialog Performance
text: One major shortcoming of CCL is its weak performance on task-oriented dialog corpora (Erker et al., 2023). As shown in Figure 6 (middle), our curved triple-encoder improves upon the curved biencoder by 46% significantly (RQ1). Overall we observe that contextualization brings the biggest benefit to task-oriented corpora, as both the noncurved and curved triple-encoders outperform biencoders. In contrast to Erker et al. (2023), we find that the curvature of triple-encoders is essential on task-oriented corpora as well, yielding a 20% performance boost over the hard positives tripleencoders ablation. As Figure 6 (middle) shows, on larger context size the l = 2 triple-encoder outperforms the standard average triple-encoder similar to the DailyDialog experiments. Again, we observe that triple-encoder as bi-encoder also outperforms CCL (RQ2) substantially.

heading: 6.1.4 Zero-Shot Performance
text: For out-of-distribution dialogs on PersonaChat (Zhang et al., 2018) in Figure 6 (right) we find that the 2-last rows of our emerging triangle of mixed representations (Table 1) are crucial for generalization to larger context sizes. As the gap in the finetuned experiments is much smaller, we find that longer turn distances are much weaker modeled in zero-shot settings. Nonetheless, the hard positives ablation still performs significantly worse than using curved scores. The fact that our last 2 rows (l = 2) average triple-encoder outperforms ConveRT shows that the co-occurrence objective has nonetheless strong generalization capability. Here we also observe how our distributed representations over ConveRT’s one context vector information bottleneck comes into play. Initially, ConveRT demonstrates superior performance, but as shown in Figure 6 (right), our model progressively improves with longer context lengths. It starts outperforming ConveRT when the context length reaches 5 and continues to exhibit improvement, in contrast to ConveRT’s performance plateau. Notably, the triple-encoder as bi-encoder generalizes also better on zero-shot scenarios compared to simple CCL (Erker et al., 2023) (RQ2).

heading: 6.2 Short-Term Planning
text: We evaluate the triple-encoder also in the shortterm planning scenario on DailyDialog. As expected the extra contextualization helps on this task as shown in Table 3, most significantly on the Hits@5 metric. We find that the gap from contextualized triple-encoder to the triple-encoder as bi-encoder is significantly closer than in other tasks. This demonstrates the versatility of the pre-training alone (RQ2) while the contextualization at inference shows additional small gains (RQ1).

heading: 7 Conclusion
text: In this paper, we presented a novel approach for conversational sequence modeling, addressing the limitations of traditional methods such as ConveRT. Our triple-encoder leverages the concept of Curved Contrastive Learning and enhances it by incorporating contextualization through a Hebbian-inspired co-occurrence learning where representations that fire in a sequence together, wire together. This enables a more efficient and effective representation of dialog sequences without the need for additional weights, merely through local interactions, a first-of-its-kind approach that exhibits these selforganizing properties. As a result, our method outperforms single vector representation models on long sequences in zero-shot settings.
Our work demonstrates the distributed modularity of sequential representations by only mapping sequential properties within latent sub-spaces, i.e. all information is stored in the geometry of the latent space. For future work, we envision the exploration of triple-encoders for sequence modeling tasks other than dialog and story modeling. To encourage the community to contribute in this direction we release our model and open-source our code.

heading: 8 Limitations
text: Building on Erker et al. (2023) work we face similar limitations. In particular we address in this section the random splitting of our dataset in shortterm planning, the use of synthetic data from LLMs to generate candidates replies, the generalizability to other datasets/tasks and response selection in the era of LLMs.
Splitting data for the short-term planning experiments: Like in Erker et al. (2023), our shortterm planning results are limited by the fact that we split at fixed positions in the dialog, which might not necessarily be planable. While this suggests that the models perform slightly better if planning were always possible, it offers an unbiased comparison between the different models.
Usage of synthetic data in short-term planning experiments: Additionally, the candidates for this task are generated by a large language model (LLM) where two issues can arise: (1) An utterance might lead to a goal that is not very likely given the context (see Erker et al. (2023)) or (2) where the true utterance is out of distribution of the LLM candidates and this true utterance can only reach the goal.
Datasets: One further limitation of our work is that our models are only tested on three dialog datasets and only one story generation dataset.
Response selection in era of LLMs: While Large Language models are becoming more and more popular in response generation, they still suffer from hallucinations (Bouyamourn, 2023), which is why retrieval is still popular, especially in legal and medical domains (Louis and Spanakis, 2022; Shi et al., 2023).

heading: 9 Ethics
text: Like other work (Schramowski et al., 2022; Prakash and Lee, 2023), our models can have induced biases based on their training data. While we do not adress the concerns in this paper, all datasets that are used in our experiments are publicly available and do not include any sensitive information to the best of our knowledge.

heading: 10 Acknowledgement
text: This research work has been funded by the German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts within their joint
support of the National Research Center for Applied Cybersecurity ATHENE. Furthermore, this work has been funded by the LOEWE Distinguished Chair “Ubiquitous Knowledge Processing”, LOEWE initiative, Hesse, Germany (Grant Number: LOEWE/4a//519/05/00.002(0002)/81).

heading: A Maximum Similarity
text: In the maximum similarity-based approach we compute the batched matrix multiplication (BMM) as in the average similarity based version. Notably, our MaxSum Algorithm 1 expects the entire state (entire triangle), which can be concatenated with the BMM matrices from previous turns. For each candidate-context pair we sort the scores of
Algorithm 1 MaxSim for triple-encoders Require: bmm_result, index_tuples
for sample← 0 to bmm_result.shape[0]− 1 do for uf ← 0 to bmm_result.shape[1]− 1 do
utt_used← empty set bmm_sample← bmm_result[sample][uf ] i_sort← argsort(bmm_sample, desc) tuples_sort← [index_tuples[i] for i in i_sort] sum← 0, counter ← 0 for i in i_sort do
if tuples_sorted[i][0] ̸∈ utt_used or tuples_sorted[i][1] ̸∈ utt_used then
sum← sum+ bmm_sample[i] counter ← counter + 1 add tuples_sorted[i][0] to utt_used add tuples_sorted[i][1] to utt_used
end if end for bmm_result[sample][uf ]← sum/counter
end for end for
every pairwise contextualized representations in decreasing order. Similar to query representations of ColBERT, we then we add every score only if any of the utterances in the tuples was not yet part of the sum. In contrast to the simple average, the number of states can differ from candidate to candidate. Therefore, we have to average the result by dividing by the number of states.

heading: B Ablation Studies
text: Looking at the ablation study Table 4, we observe that pre-training a triple-encoder with (bi) curved contrastive learning (which has directional negatives) and then continuing with triplet loss (without directional negatives) yields the best performance. Followed by the triplet encoder trained from scratch and the triple-encoder with directional negatives. While all triple-encoders with the turn distance curvature (essence of CCL) yield better performance than bi-encoders, the ablation of utilizing triplet negatives but bi-positives already improves on simple CCL. Lastly, we compare our C3L triple-encoder to the triple-encoder without the curvature of scores, in other words, a triplet encoder only having hard positives. As the results show, it is 58.7% worse than curved triple-encoders, showing the fundamental necessity of the temporal curvature of curved contrastive learning for sequence modeling on relative/modular components.

heading: C Component Analysis of Triple-Encoder
text: We continue with the component analysis on the best triple-encoder from Table 5. The normal input of the triple-encoder yields the best results. Since
the mean operation of triple-encoders loses information of the original utterances, we added the normal bi-encoder cosine operation to the means, which reduced the performance. We note that the [B1] and [B2] tokens are essential, as the means between [B1] and [B1] as week as [B2] and [B2] reduce the performance drastically. Interestingly, [B2] is significantly better than [B1] in both means with itself as well as alone as a bi-encoder. This makes sense as the utterance closer to the current turn should have a higher impact on a candidate’s utterance than the ones further away. It’s especially noteworthy that direct neighbor contextualization, which only accounts for adjacent utterance pairs, performs competitively compared to the combined bi-encodings of [B1] and [B2]. This underscores the value of non-local neighbor contextualization, which improves performance by 18%.

heading: D Representations that Fire Together, Wire Together
text: We start the investigation of stronger additive properties of C3L over CCL in the bi-encoder setup by comparing the average similarity of sequences to the correct utterance and random sampled utterances. We use the same setup as in the sequence modeling evaluation on the test set. While the absolute similarity of CCL to correct utterances in the bi-encoder setup is greater than in C3L, Table 6 reveals that the similarity of C3L to random utterance is much closer to the target similarity of
0 for hard negatives, demonstrating stronger discriminative properties. Specifically, we find that similarity difference from random utterances to correct utterances is greater in C3L compared to CCL. However, to demonstrate the stronger additive properties, a stronger contribution of each context utterance within sequences to candidate utterances has to be shown. Hence, we measure for every context utterance in all contexts of size 8, the difference between the correct and the average random similarity. While for the bi-encoders each utterance is one representation, in the tripleencoder setup we have n − 1 mixtures of each utterance which we aggregate (mean) for each utterance respectively. Our results in Figure 8 show, that the additive properties over random utterances are significantly stronger over the entire history in C3L compared to CCL, thanks to our introduced co-occurrence learning objective. In general we observe that the latest utterance has the strongest contribution, while the influence of utterances from our dialog partner are in general more important shown by the fluctuation between odd and even turns. As each triple encoder utterance contains a mixture of all n − 1 context utterances, further away utterances decay less strongly as in the biencoder setup. For bi-encoders, the gap between C3L & CCL becomes closer over longer distances. While CCL looses a lot of information already after the last utterance, C3L bi-encoders have a much more steady decline as the information (wiring) of close utterances through its training objective is significantly better preserved.

heading: E Children Book Test
text: Apart from dialog we also experiment with text generation within the Children Book Test dataset (Hill et al., 2016).
E.1 Setup
The dataset is already split into a list of sentences for each story, which we treat similarly to utterances in our dialog setup. Apart from speaker tokens that are removed, we apply our method in the same way as for dialogs. We train a simple bi-encoder with CCL (Erker et al., 2023), triple encoders with C3L as well as its hard positive ablation. We evaluate the technique by ranking the next sentence.
E.2 Evaluation Similar to our dialog results 6, we observe that Triple Encoders are improving significantly over CCL with an increase of 39.85% in average rank (RQ1). This can also be observed for the triple as bi- encoder version that outperforms the hard positives ablation until a sequence length of 8 where contextualization seems to become more important than the relative distance objective of C3L (RQ2). We explore different settings for last l rows in Table 8. We find l = 1 performs best for sequences longer than 4 sentences. We believe that l = 2 is worse here as the speaker tokens are absent and therefore taking two over longer distances might lead to distortions.

heading: Future Work
text: For future work, we envision the exploration of triple-encoders for sequence modeling tasks other than dialog and story modeling. To encourage the community to contribute in this direction we release our model and open-source our code