Harnessing the Universal Geometry of Embeddings
Authors: Rishi Dev Jha
Collin Zhang
Vitaly Shmatikov
John Xavier Morris
Venue: NeurIPS 2025 poster ()
Keywords: Embeddings
Text Embeddings
Representations
Representation Learning
Machine Learning
Security
Privacy
Primary area: deep_learning
TL;DR: We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches.

Abstract: We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches.  Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., a universal semantic structure conjectured by the Platonic Representation Hypothesis).  Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets.

The ability to translate unknown embeddings into a different space while preserving their geometry has serious implications for the security of vector databases.  An adversary with access only to embedding vectors can extract sensitive information about the underlying documents, sufficient for classification and attribute inference.

Harnessing the Universal Geometry of Embeddings
Rishi Jha
Collin Zhang
Vitaly Shmatikov Department of Computer Science Comell University
John X. Morris
Abstract
We introduce the first method for translating text embeddings from one vector space another without any paired data, encoders, or predefined sets of matches. Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., universal semantic structure conjectured by the Platonic Representation Hypothesis). Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets.
The ability to translate unknown embeddings into a different space while preserving their geometry has serious implications for security. An ad adversary with access to a database of only embedding vectors can extract sensitive information about underlying documents, sufficient for classification and attribute inference.
Embaddings [Origina]
Latent Recresentations [vac2vec]
Figure 1: Left: input embeddings from different model families (T5-based GTR 47 and BERT-based GTE (32) are fundamentally incomparable. Right: given unpaired embedding samples from different models on different texts, our model learns latent representation where they are closely aligned.
Introduction
Text embeddings the backbone of modern NLP, powering tasks like retrieval, RAG, classification, and clustering. There are many embedding models trained on different datasets, data shufflings, and initializations. An embedding of text encodes its semanties: good model maps texts with similar semantics to vectors close to each other in the embedding space. Since semantics is a property of text, different embeddings of the same text should encode the same semantics. In practice, however, different models encode texts into completely different and incompatible vector spaces.
The Platonic Representation Hypothesis Huh et al. [18] conjectures that all vision models of sufficient size converge to the same latent representation. We propose stronger, constructive version this hypothesis for text models: the universal latent structure of text representations can be learned and, furthermore, harnessed to translate representations from one space to another without any paired data encoders.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Documents (D)
Embeddings of
Embeddinge
Documants (M)
Taesiig hoomline
peoouuce loras per yeur sopaal Worl
Eancy
hare, pless
Thane you lick
Secre
Piblic
Some emals dicuse ng NBOO
Emsatazeaess
tichus c (RB EB
aktomint of
speeker fimm
Doouments (D)
Enteoldinssof of
Jinonder
Figure 2: Given only vector database from an unknown model, vec2vec translates the database into the space of a known model using latent structure alone. Converted embeddings reveal sensitive information about the original documents, such the topic of an email (pictured, real example).
In this work, we show that the Strong Platonic Representation Hypothesis holds in practice, Given unpaired examples of embeddings from two models with different architectures and training data, our method learns a latent representation in which the embeddings are almost identical (Figure I.
We draw inspiration from research on aligning word embeddings across languages 62 10
and unsupervised image translation (36 [0]. Our vec2vec method uses adversarial losses and cycle consistency learn to encode embeddings into a shared latent space and decode with minimal loss. This makes unsupervised translation possible. We use basic adversarial approach with vector space preservation [46] learn mapping from an unknown embedding distribution known one.
vec2vec is the first method to successfully translate embeddings from the space of one model to another without paired data vec2vec translations achieve cosine similarity as high 0.96 the ground-truth vectors in their target embedding spaces and perfect matching on over 8000 shuffled embeddings (without access the of possible matches in advance).
To show that our translations preserve not only the relative geometry of embeddings but also the semantics of underlying inputs, we extract information from them using zero-shot attribute inference and inversion, without any knowledge the model that produced the original embeddings
Problem formulation: unsupervised embedding translation
Consider a collection of embedding vectors (ui....un.) for example, a dump of a compromised vector database, where each u, = M2 (d;) is generated by an unknown encoder M1 : V > dRd,, from unknown document d; We cannot make queries to Mj and do not know its training data, nor architectural details. Our goal is to extract any information about the documents d,.
We do assume access to different encoder M2 that we can query at will to generate new embeddings some other space. We also assume high-level distributional knowledge about the hidden docu- ments: their modality (text) and language (e.g.. English). To extract information, we may translate (u.....u,, ] into the output space of M2 and apply techniques like inversion that require the encoder.
Limitations of correspondence methods. There is significant prior research on the problem of matching or correspondence between sets of embedding vectors 1 49 54 These methods typically assume that the two (or more) sets of embeddings are generated by different encoders on the same or iivell--verlappiing inputs. other words, for each unknown vector, there must already exist a set of candidate vectors in different embedding. In practice, it is unrealistic to expect that such a database be available, so these methods are not directly applicable. Some matching methods,
Prior work has successfully translated word embeddings between languages. typically relying on overlapping vocabularies across languages. contrast, we translate embeddings entire sequences hetween model spaces. Our code is available on GitHub.

---

Unknown Embedding Space
Known Embedding Space
Document
Susject: Ehron Reshing Frontins
F(ui)
Karon, alaase ma
M1
Vacovoc
when you reseive this omail Thank you Rick
Visible vec2vec
F(ui) = vi
Invisible to vec2vec
Figure 3: Unsupervised embedding translation. With access to only u, = M1 vec2vec seeks to generate a translation F(u,) that close in M2's 's embedding space to the ideal embedding M2;d;) without access to d;, Ug or M1.
however, support translation between embedding spaces without overlapping inputs. Our experiments demonstrate that these methods struggle significantly, even when correspondence exists.
Our task is inherently more challenging than matching, because we do not assume access to encoder M1. nor do we have additional representations of documents d1.... d, beyond their embeddings 24 M1 (d.). Therefore, we rely solely on unsupervised translation from M1 to M2. The effective- ness of such unsupervised translation approaches thus critically depends on identifying and leveraging shared geometric structures within the embedding spaces produced by M1 and M2.
The Strong Platonic Representation Hypothesis. Our hope that unsupervised embedding translation is possible at all rests on the stronger version of the Platonic Representation Hypothesis (18). Our conjecture is follows: neural networks trained with the same objective and modality, but with different data and model architectures, converge to a universal latent space such that translation between their respective representations can be learned without any pairwise correspondence.
Translation enables information extraction. Solving unsupervised translation will allow us to use information extraction tools designed to operate on vectors produced by known encoders. For example, we could apply inversion models [43 67 to recover unknown documents {di}.
3 Our method: vec2vec
Unsupervised translation has been successful in computer vision, using combination of cycle consistency and adversarial regularization [36] 770. Our design of vec2vec is inspired in part by these methods. We aim learn embedding-spart translations that are cycle-consistent (mapping to and from an embedding space should end in the same place) and indistinguishable (embeddings for the same text from either space should have identical latents).
3.1 Architecture
We propose a modular architecture, where embeddings are encoded and decoded using space-specific adapter modules and passed through shared backbone network. Figure Figure[]shwws shows these components. Input adapters A₁ gd + ² and Ag : R2 - _R transform embeddings from each encoder-specific space into universal latent representation of dimension Z. The shared backbone T' R IR? extracts a common latent embedding from adapted inputs. Output adapters B₁ : R + Rd and By R > Rd translate these common latent embeddings back into the encoder-speciff spaces. Thus, translation functions F1, F2 and additional reconstruction mappings R1, R2 are defined as:
F = Bg o o A1, Fg = B T o A2 R1 = B, o o A RS = Bg o To Ag
Parameters of all components are collectively denoted = A2, B,, B2).
Unlike images, embeddings do not have any spatial bias. Instead of CNNs, we use multilayer percep- trons (MIP) with residual connections, layer normalization, and SiL.U nonlinearities. Discriminators mirror this structure but omit residual connections to simplify adversarial learning.

---

3.2 Optimization
In addition the 'generator' networks and R, we introduce discriminators operating on both the latent representations of F (Dí, D2) and the output embeddings (D1, D2).
Our goal is to train the parameters of by solving:
0" arg min max
Lade F1, F2, D1, D2, D1, D2) + Agen
9
Di.
where Lud, and Leen represent adversarial and generator-specific constraints respectively and hyper- parameter Agen controls their tradeoff.
Adversarial. The adversarial loss encourages generated embeddings to match the empirical distri- butions of original embeddings both at the embedding and latent levels. Specifically, applying the standard GAN loss formulation [3] to both levels yields:
Ladv (F1, F2, D1. D2, DV D2) LOAN Fi) + COAN
LGAN A1) + LGAN (D2, A2).
Generator. Because adversarial losses alone do not guarantee that translated embeddings preserve semantics [70). we introduce three additional constraints to help the generator learn a useful mapping:
Reconstraction enforces that an embedding, when mapped into the latent space and back into its original embedding space, closely matches its initial representation:
Lie [R1, R2) Eg Ry (z) 2112 + Ey
where p and g distributions of embeddings sampled from M1 and M2. respectively.
Cycle-consistency acts an unsupervised proxy for supervised pair alignment, ensuring that F and G can translate an embedding to the other embedding space and back again with minimal corruption:
(Fc F22 = Ex-pp F2 (z)) - x=12 - Ey (F2()) -11..
Vector space preservation ( (VSP) ensures that pairwise relationships between translated embeddings consistent with the target space [46] 65). Given batch of B embeddings 01,--,,3 Tg and y1: VB.
we sum their average pairwise distances after translation by both Fi and F2:
B
B
Lvsp(Fi,
ΣΣ
(z;) M1(x,) Fa(M2())) F2(Ma(,)))
B
+ -|M2(w)) M2(u,) ( (M, FiM
Combining these losses yields: Leen (0) = Areh R, R2) Acc (1, F2) + AvseLvse(FF (F F2).
where hyperparameters Ace, Avce. and Avsp control relative importance.
4
Experimental setup
4.1 Preliminaries
Datasets. We use the Natural Questions (NQ) [25] dataset of user queries and Wikipedia-sourced
answers for training (a 2-million subset) and evaluation (a 65536 subset). To evaluate information extraction, we use TweetTopic (2), dataset of tweets multi-labeled by 19 topics; random 8192- record subset Pseudo Re-identified MIMIC-III (MIMIC) (28), ), a pseudo re-identified version
MIMIC dataset [19 of patient records multi-labeled by 2673 MedCAT [24] disease descriptions; and a random 50-email subset of the Enron Email Corpus (Enron) (2T). an unlabeled, public dataset of
internal emails from defunct energy company. In Appendix D ablate model on MS COCO (34, a captioned image dataset, to evaluate performance on multmodal retrieval.
Models. Table lists the embedding models representing four size categories, five transformer backbones, and two output dimensionalities. Granite multilingual; CLIP is multimodal. Since Qwen is very compute-intenet intensive, only evaluate it for a single model pair in Appendix

---

Model
Params (M)
Backbone
Yeur
Dims
Max Seq.
47
110
T5
2021
768
512
30
clip
151
CLIP
2021
512
77
58
BERT
2022
512
32
gle
109
BERT
2023
768
512
14
granite
278
RoBERTa
2024
768
512
69
qwen
4000
Qwen3
2025
2560
32K
Table 1: Embedding models used in our experiments.
Training. Unless otherwise specified, each vec2vec is trained from disjoint sets of 1 million 64-token sequences sampled from NQ (see Section 7 for experiments with fewer embeddings). Due GAN instability (53), we select the best of multiple initializations (see Appendix E) and leave more robust training to future work. See Appendix A for compute details.
sets of embeddings generated
4.2 Evaluating translation
Let u, = .M, (d,) and e; (2.) denote the source and target embeddings of the same input d;- The goal of translation is to generate vector that is close to e; possible. We say that (u,, 1j) are "aligned" by the translator F if Uj is the closest embedding to F(u)): j = arg ming cos F ((us), vz). perfect translator F* satisfies arg ming cos [F (u:), 2k) for all i.
Given (unknown) embeddings {M)dd ordered by decreasing cosine similarity to F(u)), let ry be the rank of the correct embedding = M2(d)). To measure quality of F, we use three metrics. Mean Cosine Similarity measures how close translations are, on average, to their targets. Top-1 Accuracy is the fraction of translations whose target is closer than any other embedding. Mean Rank is the average rank of targets with respect to translations. The ideal translator F achieves mean similarity of 1.0, top-1 accuracy of 1.0, and mean rank of 1.0. Recall that random alignment corresponds to a mean rank of a Formally,
2.
cos(F(u)) w:)] Top-1 1 Σ
1 n
cos Us, U:)
1n -
Rank(r)
n
vec2vec is the first unsupervised embedding translator, thus there is no direct baseline. As our Naïve baseline, simply use F(xx)
measure geometric similarity between embedding spaces. The second [pseudo)baseling is Oracle-aided optimal transport. It assumes that candidate targets are known and is thus strictly easier than vec2vec and the Naive baseline. We solve optimal assignment, n² arg min, Σi=1 cos(u;, Un()),, via either the Hungarian, Earth Mover's 's Distance, Sinkhorn, or (Entropic) Wasse-sssssrsttin algorithms, choosing the solver with the lowest rank for each experiment. See Appendix B for more details.
4.3 Evaluating information extraction
We measure whether translation preserves semantics via attribute inference: for each translated embedding F (d:),, our goal is to infer attributes S of d,
The first method we use zero-shot embedding attribute inference: calculate pairwise cosine similarities between F(M, (d:)) and the embeddings of all attributes in C, identify top h: closest attributes, and measure whether they are correct via top-k accuracy: 1 Σ 1 {Iot nal 2 1}.
The second method is embedding inversion that recovers text inputs from embeddings. Since 44 requires a pre- me-ttrineed inversion model for each embedding space, we use [67] instead to generate an approximation d' of d, from FMM, (d:)) in zero-shot manner. We measure the extracted information using LLM judge accuracy: the fraction of translated embeddings for which GPT-4o determines that d' reveals information in d. See Appendix for our prompt.
In addition to the Naïve baseline, we also consider an Oracle attribute inference: zero-shot classification with the correct embedding M2(d) and class labels Mc(C).
5

---

Similarity of Inputs
Similarity of Latents
Difference Similarities
gtr gle granite e5
0.02
0.68
0.03
0.38
stella gtr gte graniee e5
0.88
0.85
0.75
0.82
gte granite e5
0.86
0.17
0.72
0.45
-08
0.02
0.01
-0.02
0.01
0.88
0.90
0.75
0.92
0.86
0.89
0.77
0.91
0.6
M4
0.68
0.01
0.03
0.56
0.85
0.90
0.91
0.96
0.17
0.89
0.87
0.40
0.4
0.00
0.75
0.75
stella gtr
0.77
0.87
0.2
stella
0.82
0.91
0.40
0.0
granite
gte My
gtr
stella
65
granite
ghe
gtr
stella
e5
granite
gte Adg
gtr
stella
M4g
Figure 4: Pairwise cosine similarities input embeddings (left) and their vec2vec latents (middle) across different embedding pairs. The absolute difference between the heatmaps plots is on the right. All numbers are computed on the same batch of 1024 NQ texts.
vec2vec learns to translate embeddings without any paired data
We first show that vec2vec learns universal latent space, then demonstrate that this space preserves the geometry of all embeddings. Therefore, we can use it like universal language of text encoders to translate their representations without any paired data.
vec2vec learns universal latent space. vec2vec projects embeddings 2 into a shared latent space via compositions of input adapters ( (A......) and a shared translator T. F that even when the embeddings u, Mi (de) and t; (2,e) are far apart (i.e., have low cosine similarity), their representations in vec2vec's latent space are incredibly close: T(A, (w;)) z (())())). Figure [ visualizes this (via two-dimensional projections) for vec2vec trained on GTE and GTR embeddings: the embeddings are far apart, but their latents are nearly overlapping.
vec2vec
Naive Baseline
OT Baseline
Mi
M2
cos(-) )
T-1 f
Rank
cos(-)
T-1-
Rank
cos(-)
T-1
Rank 1
gtr
0.80
0.99
1.19
-0.03
0.00
4168.73 (9.2)
0.70
0.00
2773.72 (8
gie
0.87
0.95
1.18 (0.0)
0.01
0.00
4088.58
0.85
0.00
2680.02 (8.6)
SEC.
0.79 (0.0)
0.98
1.05 (0.0)
0.01 (0.0)
0.00
4208 26 (9.2)
0.67 (00)
0.00
3446.52 (8.8)
c5
0.85 (0.0)
0.98
1.11 (0.0)
0.02 (0.0)
0.00
4111.60 (9.2)
0.83 (0.0)
0.00
3569.59 (8.7)*
gra
0.81
0.99
1.02
-0.03
4169.76 (9.2)
0.70 (0.0)
2775.17 (8.6)
gtr
gte
0.87
0.93
2.31
0.04
0.00
4080.92
0.85
0.00
3070.69 (8.9)
0,80
0.99
1.03 (0.0)
0.00
0.00
4198.78
0.67
0.00
3559.06 (9.1)
e5
0.83 (0.0)
0.84
2,88 (0.2)
0.03 (0.0)
0.00
4082.84 (9.2)
0.83 (0.9)
0.00
3888.01 (8.9)
gma.
0.75 (0.0)
1.22 (0.0)
(0.0)
(9.3)
0.69 (0.0)
2664. (8,6)²
gle
gtr
0.75 (0.0)
2.64
0.04 (0.0)
4084.15 (9.2)
0.70 (0.0)
18.9)
SEc.
0.89
1.00
1.00
0.56
1.00
1,00
0.71
1.00
1.00 (0.0)
0.87
0.99
5,19
0.68 (0.0)
1.00
1.00
0.84
1,00
1.00 (0.0)*
gRL
0.80 (0.0)
0.98
1.08 (0.0)
0.01 (0.0)
0.00
4209.08 (9.3)
0.69 (0.0)
0.00
3419.44 188))
gtr
0.82 (0.0)
1,00
1.10 (0.0)
0.00 (0.0)
0.00
4192.31 (9.2)
0.70 (0.0)
0.00
3555 (9.0)
gte
0.92
1.00
1.00
0.56 (0.0)
1.00
1,00
0.87
1.00
1.00 (0.0)
0.86
1.000
1.00
0.38
0.99
1.03
0.83
1.00
1.00 (0.0)
gra.
0.81 (0.0)
0.99
2.20 (0.2)
0.02 (0.0)
0.00
4120.60 (9.3)
0.69 (0.0)
0.00
3526.02 (8 7,
65
gtr
0.74 (0.0)
0.82
2.56 (0.0)
0.03 (0.0)
0.00
4080.76 (9.3)
0.70 (0.0)
0.00
3877.03 (88)**
gie
(0.0)
1.01 (0.0)
0.68 (0.0)
1.00 (0.0)
0.86 (0.0)
(a.0))
ste. (0.0)
1.00
1.00 (0.0)
(0.0)
1.00 (0.0)
(0.0)
1.00
(0.0)*
Table 2: In-distribution translations: vec2vecs trained NQ and evaluated on 65536 text subset of
NQ (chunked in batches of size 8192). The rank metric varies from to 8192, thus 4096 corresponds to random ordering. Standard errors are shown in parentheses. Bold denotes best value. Symbols denote the lowest-rank solver for specific experiments: Sinkhorn" and Gromoy- "

---

TweeTTopic
MIMIC
Mi
M2
cos(-)
1-1
Rank
coo(-) 7
T-1
Rank L
gtr
0.74
0.99
1.09 (0.1)
0.74 (0.0)
0.60
23.38 (1.6)
gie
0.85 (0.0)
0.95
1.26
0.85 (0.0)
0.08
346,21 (7.8
gran.
stel.
0.77 (0.0)
0.96
1.11 (0.0)
0.72 (0.0)
0.13
242.23 (6.1)
65
0.83 (0.0)
0.87
3.77
0.84 (0.0)
0.12
361.06 (8.7)
gran.
0.79 (0.0)
0.98
2.41 (0.6)
0.78 (0.0)
0.51
35.27 (1.9)
gie
0.85 (0.0)
0.96
1.29 (0.2)
0.84 (0.0)
0.12
279.56 (6.9)
stel.
0.77 (0.0)
0.96
1.10 (0.0)
0.72 (0.0)
0.27
127.92 (4.4)
c5
0.80
0.53
13.38 (1.2)
0.82 (0.0)
0.01
1413,80 (18.3)
gran.
0.73 (0.0)
0.94
1.33
0.73 (0.0)
342.15 (7.8)
gir
0.71 (0.0)
0.95
1.29 (0.1)
0.69 (0.0)
0.12
256.63 (6.4)
stel.
0.86 (0.0)
1.00
1.00 (0.0)
(0.0)
1.00 (0.0)
c5
0.83 (0.0)
0.91
1,57 (0.2)
0.86 (0.0)
0.54
17.71 (0.9)
gran.
0.79 (0.0)
0.99
1.09
0.77
0.14
221.95
stel.
gie
gir
0.90 (0.0)
0.77 (0.0)
1.00
1.00
1.00 (0.0)
1.00 (0.0)
0.75 (0.0)
0,91 (0.0)
0.56
17.70 (1.0)
(0.0)
c5
0.85 (0.0)
0.98
1.05 (0.0)
0.85 (0.0)
0.51
26.33 (1.3)
gran.
0.79
0.98
1.08
0.78 (0.0)
0.21
151.09 (4.6)
c5
gir
0.67 (0.0)
0.80
3.10 (0.6)
0.66 (0.0)
0.01
1029.66 (14.9)
gie
0.87 (0.0)
0.99
1.02. (0.0)
0.87 (0.0)
(2.6)
stel.
0.75 (0.0)
0.98
1.06 (0.0)
0.75 (0.0)
0.46
32.12 (1.4)
Table 3: Out-of-distributio translations: vec2vecs trained on NQ and evaluated on the entire TweetTopic test set (800 tweets) and an 8192-record subset of MIMIC. The rank metric varies from I to 800 (for TweetTopic) and 8192 (for MIMIC), thus 400 and, respectively, 4096 correspond to random ordering. Standard errors are shown parentheses.
vec2vec translations mirror target geometry. 2 that vec2vec generates embeddings with near-optimal assignment across model pairs, achieving cosine similarity scores up to 0.92, top-1 accuracies up to 100%, and ranks as low as 1. In same-backbone pairings (e.g., (gte, e5)), vec2vec's top-1 accuracy and rank are comparable to both the naïve baseline and (surprisingly) the oracle-aided optimal transport. Although the embeddings generated by vec2vec are significantly closer to the ground truth than the naïve baseline, in sume-backbone pairings the embeddings are close enough to be compatible. In cross-backbone pairings, vec2vec is far superior on all metrics, while baseline methods perform similarly to random guessing.
Table 3 shows that this performance extends to out-of-distribution data. Our vec2vec translators were trained on NQ (drawn from Wikipedia), yet exhibit high cosine similarity, high accuracy, and low rank when evaluated on tweets (which are far more colloquial and use emojis) and medical records (which contain domain-specifie jargon unlikely appear in NQ). In Appendix F that baseline methods fail on cross- backbone embedding pairs.
we show
vac2vac
OT Raseline
Mr
M2
cos(.)
T-1
Rank
cos( ) 1
T-1
Rank
gru.
0.78 (0.0)
0.35
226.62 (3.2)
0.76 (0.0)
4073.58 (9.4)*
gtr
0.73
0.13
711.23
0.59
0.00
4096.78 78 (9.2)
gle
clip
0.62
0.00
3233.41
0.76
0.00
4026.96 (9.4)
stc.
0.77 (0.0)
0.31
286.69 (3.6)
0.76 (0.0)
0.00
3955.71 (8.9)
c5
0.64 (0.0)
0.01
2568.21 (9.4)
0.77 (0.0)
0.00
3771.52 (9.1)*
gra.
0.74 (0.0)
0.72
4.46 (0.1)
(0:0)
4053.11 (9.4)
gtr
0.67
0.27
155.11
0.49 (0.0)
0.00
4096.35 (9.2)
clip
gte
0.75 (0.00
0.00
2678.90 (8.9)
0.85 (0.0)
0.00
4025.81 (9.3)*
stc.
0.72 (0.0)
0.61
22.50 (0.5)
0.67 (6.0)
0.00
3951.73 (8.9)
c5 0.73 (0.0)
0.01
1692.28 (8.2)
0.83 (0.0)
0.00
3771.38 (9.0)*
Table 4: Translations between unimodal and multimodal (CLIP) embeddings: vec2vecs trained on
NQ and evaluated 65536 text subset of NQ (chunked in batches of size 8192). Rank varies from 1 to 8192, thus 4096 corresponds to a random ordering. Since the embedding dimensionalities are different, only the Bold denotes best value.
OT baseline is and the naive baseline does not apply.
7

---

TweeTTopii (k =
MIMIC (k 10)
M,
M2
vec2vec
Naive
Mr
M42
vec2vec
Naive
Mi
M2
gtr
0.25
0.24
0.76
0.88
0.32
0.09
0.30
0.34
0.36
0.13
0.76
1.00
gran.
stel.
0.27
0.96
c5
0.31
0.18
0.30
0.31
0.19
0.20
0.76
0.97
gran
0.24
0.76
0.33
0.13
0.24
0.34
0.28
0.05
0.88
1.00
0.96
c5
0.30
0.04
0.24
0.31
0.09
0.09
0.88
0.97
gran
0.37
0.76
0.24
0.13
0.34
0.24
0.10
0.03
1.00
0.88
c5
0.37
0.30
0.34
0.31
0.37
0.63
1,00
0.97
gran.
0.35
0.28
0.30
0.23
0.09
0.76
gir
0.26
0.13
0.28
0.24
0.22
0.09
0.96
0.88
gle
e5
0.35
0.34
0.28
0.31
0.38
0.46
0.96
0.97
gran. 0.33
0.31
0.30
0.14
0.07
0.76
e5
gir
0.26
0.22
0.31
0.24
0.11
0.04
0.97
0.88
gle
stel.
0.26
0.16
0.31
0.28
0.36
0.40
0.97
0.96
Table 5: Information leakage via top-k zero-shot attribute inference: vec2vecs trained on NQ and evaluated on the TweetTopic test (800 tweets) and an 8192-record subset MIMIC. M1 and M2 represent ideal zero-shot inference: attributes and embeddings are encoded using the same model.
Finally, Table shows that vec2vec can even translate and from the space of CLIP, a multimodal embedding model which was trained in part on image data. While the translations are not as strong as Table vec2vec consistently outperforms the optimal transport baseline. These results show the promise of our method at adapting to new modalities: in particular, the embedding space of CLIP has been successfully connected to other modalities such as heatmaps, audio, and depth charts [12].
6 Using vec2vec translations to extract information
In this section, we show that vec2vec translations not only preserve the geometric structure of embeddings but also retain sufficient semantics to enable attribute inference.
Zero-shot attribute inference. Table 5 shows that attribute inference on vec2vec translations consistently outperforms the naïve baseline and often does better than the ideal zero-shot baseline which performs inference on ground-truth document and attribute embeddings in same space (this baseline is imaginary since these embeddings are not available in our setting).
Percentage Emails With Leakage
80
gte granite e5
49
52
28
62
70
55
56
56
56
660
.58
50
vec2vec translations even work for embeddings med- ical records, which are much further from the training distribution than tweets. The attributes in this case are MedCAT disease descriptions, very few of which occur in the training data. Attribute inference on translated embeddings comparable to the naïve baseline in same- backbone pairings and outperforms (often greatly) in cross-backbone pairings. The fact that vec2vec pre- serves the semantics of concepts like "alveolar periosti- tis" (which never appears in its training data) is evidence that its latent space is indeed a universal representation.
gtr
20
60
46
55
-40
stella
300
56
52
40
e5
granite
gte
gtr
stella
Figure 5: Leakage of information via in- version. Trained on NQ and evaluated on 50-email subset of the Enron Email Cor- pus. Cells denote judge accuracy.
Zero-shot inversion. Inversion, i.e., reconstruction of text inputs, is more ambitious than attribute inference. vec2vec translations retain enough semantic information that off-the-shelf, zero-shot inversion methods like (67], developed for embeddings computed by standard encoders, extract

---

Ground Truth: "Subject:
Enron
Bashing on Frontline \n Body:..."
Generation: "Some emails discussing NROn Employee/s Complaint To thePublic
Ground Truth: "Subject: Trades for 3/1/02 An Body: An John An The following trades..."
Generation: "... future transactions
may await John
G..."
Ground Truth: " Thee following expense report is ready for approval..."
Generation: " "The upcoming expense statement from YYYY MM Dec..."
Figure 6: Examples Enron Email Corpus inversions that infer entities and content
information for as many as 80% of emails and 67% of tweets given only their translated embeddings. for some model pairs (Figure [5] and Appendix GG. These inversions are imperfect and we leave development of specialized inverters for translated embeddings to future work. Nevertheless, as exemplified in Figure O they still extract potentially sensitive information such as individual and company names, dates, promotions, financial information, outages, and even lunch orders. In Appendix H we show the prompt we use measure extraction.
7 Ablations
Method
cos(-) f
1-1 f
Rank J
vec2vec
0.75 (0.0)
0.91
2.64 (0.1)
Naive Baseline
0.04 (0.0)
4084.15
OT Baseline
0.70
0.00
3064.16 (8.9)
VSP loss
0.58
0.00
4196.64 (9.2)
CC loss
0.50
0.00
3941 36 (9.3)
latent GAN
0.49
3897.00
VSP and CC loss
0.47 0.40
0.00
3365. 2165.2410 (9.3)
hyperparam. toning
0.50
0.00
4011.73 (9.3)
Table 6: gte - gtr translators trained without individual components of method on NQ and evaluated on a 65536-text subset of NQ (chunked in batches of 8192). The rank metric varies from 8192, thus 4096 corresponds random ordering. Standard errors are shown in parentheses.
Each component of our method is important. We ablate our method subtractively, measuring the key metrics after removing individual components of our algorithm (described in Section B}. Table shows that each component appears to be critical to building good translations. While vec2vec cos(-) is higher than the naive baseline, performs worse across the board than the OT baseline and does not preserve the geometry of the vector space.
cos(-)
Rank
1000000
0.75 (0.0)
0.92
2.73 (0.2)
10000
0.57 (0.0)
0.01
1462.21 (20.)
(0.0)
(0.6)
100000
0.74 (0.0)
0.85
4.52 (0.4)
500000
0.75 (0.0)
0.92
2.73 (0.2)
Table gte - gtr translators trained with different amounts of GTE data: vec2vec models trained on NQ and evaluated an 8192-record subset of NQ. The rank metric varies from to 8192, thus 4096 corresponds to a random ordering. Standard errors are shown in parentheses.
vec2vecs can be trained with significantly less data. In Sections 5 and 6 we use 1M-point subsets of NQ to train our vec2vec models. Now, we train the gte -> gtr vec2vec with 1M GTR embeddings but fewer GTE embeddings. Table 7 shows that with as few as 10K embeddings, the translators still

---

learn something (i.e. are better than random). Translations trained on 50K embeddings are almost as good as those trained on 1M. Translations generally improve with more training data.
Related work
Representation alignment. Similarities between representations of different neural networks are investigated in 26 31
59
18
5
61 30J. Methods based on CCA [42.. SVCCA, [5T]. CKA [23 38),
39 and GUls [6] have been used to compare embeddings from different ICA [63]. time series
subspaces. 37 45
57, 48 harness representation similarity for zero-shot stitching, substitution, domain transfer, and multimodal adaptation. All rely some amount of paired data, which is difficult to reduce (6). Our method does not just measure similarity, we learn how to translate representations across spaces without any paired data.
40
Optimal transport. The problem of unsupervised optimal transport has been studied for image style transfer [7
36 70]. word translation [62 13 9 20), , and natural language sequence translation Our method builds on these works, which often employ a combination of sycle-conissenccy and adversarial loss. Importantly, unlike prior word and sequence translation methods, multiple representations of the same input (e.g.. heavily overlapping word vocabularies) are unavailable in our setting. (54) proposes solver for matching small sets of embeddings between different vision-language models. Our method goes well beyond matching by taking unknown embeddings and generating matching embeddings in the space of another model.
52
Embedding inversion. An emerging line of research investigates decoding text from language model embeddings (55] 29 and outputs 44] / 66]. vec2vec helps apply these unknown embeddings, without an encoder or paired data, by translating them to the space of a known model.
Bridging modality gaps. Previous work has noted inherent "gap" between image- and text-based models (33) and proposed various ways to unify the modalities [56]. Some approaches feed image embeddings directly into language models [22 60 11 35). while others generate captions from
image embeddings (41) or even from text embeddings themselves [43). [12) introduces a shared
embedding space that integrates inputs from multiple modalities, including text, audio, and vision. In
contrast, our post-hoc approach directly translates between representations and complements these
systems by enabling inputs from wide variety of embedding models.
Discussion and Future Work
The Platonic Representation Hypothesis conjectures that the representation spaces of modern neural networks are converging. We assert the Strong Platonic Representation Hypothesis: the latent universal representation can be learned and harnessed to translate between representation spaces without any encoders or paired data.
In Section we demonstrated that our vec2vec method successfully translates embeddings generated from unseen documents by unseen encoders, and the translator is robust to (sometimes very) out- of-distribution inputs. This suggests that vec2vec learns domain-agnostic translations based on the universal geometric relationships which encode the same semantics in multiple embedding spaces.
In Section d we showed that vec2vec translations preserve sufficient input semantics to enable attribute inference. We extracted sensitive discase information from patient records and partial content from corporate emails, with access only to document embeddings and no access to the encoder that produced them. Better translation methods will enable higher-fideligg extraction, confirming once again that embeddings reveal (almost) as much as their inputs.
Our findings provide compelling evidence for the Strong Platonic Representation Hypothesis for text- based models. Our preliminary results on CLIP suggest that the universal geometry can be harnessed in other modalities, too. The results in this paper are but a lower bound on inter-repressnnatiio translation. Better and more stable learning algorithms, architectures, and other methodological improvements will support scaling to more data more model familics, and more modalities.
10

---

Acknowledgments and Disclosure of Funding
This research supported in part by the Google Cyber NYC Institutional Research Program. RJ is supported by the Digital Life Initiative Fellowship and JM by the National Science Foundation.
References
[1] David Alvarez-Melis and Tommi S. Jaakkola. "Gromov- Maamoov-Wassesttein Alignment of Word Embedding Spaces". 2018. arXiv: 1809. [cs.CL. CLJ
[2] Dimosthenis Antypas, Asahi Ushio, Francesco Barbieri, and Jose Camacho-Colladss "Mul- tilingual Topic Classification in X: Dataset and Analysis". In: Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing. Ed. by Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen. Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, pp. 20136-20152.
[3] Mikel Artetxe, Gorka Labaka, and Eneko Agirre. "A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings". In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers). Ed. by Iryna Gurevych and Yusuke Miyao. Melbourne, Australia: Association for Computational Linguistics, July 2018, pp. 789-798.
[4] Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. "Unsupervised Neural Machine Translation". 2018. arXiv: 1710. 11041 (cs.CL)
[5] Yamini Bansal, Preetum Nakkiran, and Boaz Barak. "Revisiting Model Stitching to Compare Neural Representations" 2021. arXiv: 2106 07682 [G]]
[6] Irene Cannistraci, Luca Moschella. Valentino Maiorca, Marco Fumero, Antonio Norelli, and Emanuele Rodolà. "Bootstrapping Parallel Anchors for Relative Representations" 2023. arXiv:
2303.00721 [c]
[7] Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Commy, Itay Yona, Eric Wallace, David Rolnick, and Florian Tramèr. "Stealing Part of Production Language Model". 2024. arXiv: 2403. 06634 [cs.CR] CR]
[8] Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin, and Jingjing Liu. "Graph Optimal Alignment". 2020. arXiv: 2006. 14744 [C)
Transport for Cross-Donen
[9] Xilun Chen and Claire Cardie. "Unsupervised Multilingual Word Embeddings". 2018 8. arXiv: 1808.08933 [cs.CL] CL.
[10] Alexis Conneau, Guillaume Lample, Marc' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. "Word Translation Without Parallel Data". 2018. arXiv: 1710.04087 (cs.CL)
[11] Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. "DreamLL.M: Synergistic Multimodal Comprehension and Creation". 2024. arXiv: 2309. 11499 [cs.CV]
[12] Rohit Girdhar, Alaacldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. "ImageBind: One Embedding Space To Bind Them AIl" 2023. arXiv: 2305. 05665566666666 [cs.cV]
[13] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. "Generative adversarial networks". In: Commun. ACM 63.11 (Oct. 2020), pp. 139-144.
[14]
IBM Granite Embedding Team. "Granite Embedding Models". Dec. 2024.
[15] Edouard Grave, Armand Joulin, and Quentin Berther. "Unsupervised A Alignment of Embed- dings with Wasserstein Procrustes". 2018. arXiv: 1805. 11222 [cs. LG]
[16] Florian Heimerl, Christoph Kralj, Torsten Moller, and Michael Gleicher. "embComp: Visual Interactive Comparison of Vector Embeddings". In: IEEE Transactions Visualization and Computer Graphics 28.8 (Aug. 2022), pp. 2953-2969.
[17] Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz, "Multimodal Unsupervised Image- Im-ee Translation" 2018. arXiv: 1804. 04732 [cs.CV]
[18] Minyoung Huh, Brian Cheung. Tongzhou Wang, and Phillip Isola. "The Platonic Representa- tion Hypothesis". 2024, arXiv: 2405. 07987 [cs.LG]

---

[19] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. "MIMIC-
III, freely accessible critical care database". In: Scientific data 3.1 (2016), pp. 1-9.
[20] Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. "Loss in Translation: Learning Biliingual Word Mapping with a Retrieval Criterion". In: Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing. Ed. by Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii. Brussels, Belgium: Association for Computational Linguistics, Oct. 2018, pp. 2979-2984.
[21] Bryan Klimt and Yiming Yang. "The enron corpus: new dataset for email classification the 15th European Conference on Machine Learning. ECML'04. Pisa, Italy: Springer-Veriae Verlag, 2004, pp. 217-226.
research". In: Proceedings the
[22] Jing Yu Koh, Ruslan Salakhutdinov. and Daniel Fried. "Grounding language models images
for multimodal inputs and outputs". In: International Conference on Machine Learning. PMLR. 2023. pp. 17283-17300.
[23] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton, "Similarity of Neural Network Representations Revisited", 2019. arXiv: 1905 00414 [cs.LG]]
[24] Zeljko Kraljevic, Thomas Searle, Anthony Shek, Lukasz Roguski, Kawsar Noor, Daniel Bean, Aurelie Mascio, Leilei Zhu, Amos A Folarin, Angus Roberts, et al. "Multi-domain clinical natural language processing with MedCAT: the medical concept annotation toolkit".
In: Artificial intelligence in medicine 117 (2021), p. 102083.
[25] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. "Natural Questions: A Benchmark for Question Answering Research", In: Transactions of the Association for Computational Linguistics 7 (2019). Ed. by Lillian Lee, Mark Johnson, Brian Roark, and Ani Nenkova, pp. 452-466.
[26] Aarre Laakso and Garrison Cottrell. "Content and cluster analysis: Assessing representational similarity in neural systems". In: Philosophical Psychology 13.1 (2000), pp. 47-76. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. "Unsu- pervised Machine Translation Using Monolingual Corpora Only". 2018. arXiv: 17111
[27]
[cs
[28]
Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron Wallace. "Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?" In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. Ed. by Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou. Online: Association for Computational Linguistics, June 2021, pp. 946-959.
[29] Haoran Li, Mingshi Xu, and Yangqiu Song. "Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentence". 2023. arviv: 2305. 03010 CL]
[30] Jiaang Li, Yova Kementchedjhieva, Constanza Fierro, and Anders Søgaard. "Do Vision and Language Models Share Concepts? A Vector Space Aligument Study", In: Transactions of the Association for Computational Linguistics 12 (2024), pp. 1232-1249.
[31] Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. "Convergent Learning: Do different neural networks learn the same representations?" 2016. arXiv: 1511 07543
[32] Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang "Towards General Text Embeddings with Multi-stage Contrastive Learning". 2023. arXiv:
2308.03281 [cs.0n CL]
[33] Weixin Liang. Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. "Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning". 2022.
arXiv: 2203 02053 [cs.CL]
[34] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. "Microsoft coco: Common objects in context". In: European conference on computer vision. Springer. 2014, pp. 740-755.
[35] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. "Visual Instruction Tuning".
2023, arXiv: 2304. 08485 [cs.CV]
12

---

[36] Ming- Yu Liu, Thomas Breuel, and Jan Kautz. "Unsupervised Image-to-Image Translation Networks". 2018. arXiv: 00848 [cs.cV]]
[37] Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodola. "Latent Space Translation via Semantic Alignment". 2024. arXiv:
2311.00664 [Cs.LG]
[38] Mayug Maniparambil, Raiymbek Akshulakov, Yasser Abdelaziz Dahou Djilali, Mohamed El Amine Seddik, Sanath Narayan, Karttikeya Mangalam, and Noel O' CoConor. "Do Vision and Language Encoders Represent the World Similarly?" In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp.
[39] Deven M Mistry and Ali A Minai. "A Comparative Study of Sentence Embedding Models for Assessing Semantic Variation". In: International Conference on Artificial Neural Networks.
2023, pp. 1-12.
[40] Mazda Moayeri, Keivan Rezaei, Maziar Sanjabi, and Soheil Feizi. "Text-To-Concept (and Back) via Cross-Model Alignment". 2023. arXiv: 22305.06386 [cs.CV]
[41] Ron Mokady, Amir Hertz, and Amit H. Bermano. "ClipCap: CLIP Prefix for Image Caption- ing". 2021. arXiv: 2111 09734 Cv]
[42] Ari S. Morcos. Maithra Raghu, and Samy Bengio. "Insights on representational similarity in neural networks with canonical correlation". 2018. arXiv: 1806 05759 [stat.Mi.
[43] John X. Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander M. Rush. "Text Embeddings Reveal (Almost) As Much As Text". 2023. arXiv: 2310. 06816 [cs. CLJ
[44] John X. Morris, Wenting Zhao, Justin T. Chiu, Vitaly Shmatikov, and Alexander M. Rush. "Language Model Inversion". 2023. arXiv: 2311 13647 [cs.Cl] CL]
[45] Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. "Relative representations enable zero-shot latent space communication". 2023. arXiv: 2209. 15430 [cs.LG]
[46] Nikola Mrksic, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gasic, Lina Rojas- Barjaoaa Pei-Hao Su, David Vandyke, Tsung-Hsiee Wen, and Steve Young. "Counter-fitting Word Vectors to Linguistic Constraints". 2016. arXiv: 1603. 00892 [C]
[47] Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfci Yang. "Large Dual Encoders Are Generalizable Retrievers". 2021. arXiv: 2112. 07899 [cs. IR]]
[48] Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolå, and Francesco Locatello. "ASIF: Coupled Data Turns Unimodal Models to Multimodal Without
Training". 2023. arXiv: 2210.01738 [CS.LG]
[49] Gabriel Peyré, Marco Cuturi, and Justin Solomon. "Grseov-Wassersten Averaging of Kernel and Distance Matrices". . In: Proceedings of the 33rd International Conference Machine Learning (ICMI.). Vol. 48. JMLR: Workshop and Conference Proceedings. New York, NY, USA: JMLR, 2016.
[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. "Learning Transferable Visual Models From Natural Language Supervision". 2021.
21Xv3.2103... 00020 [cs
[51] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. "SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability". 2017. arXiv: 1706. 05806 [stat.Mi,
[52] Sujith Ravi and Kevin Knight. "Deciphering Foreign Language", In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Ed. by Dekang Lin, Yuji Matsumoto, and Rada Mihalcea. Portland, Oregon,
USA: Association for Computational Linguistics, June 2011, pp. 12-21.
[53] Divya Saxena and Jiannong Cao. "Generative Adversarial Networks (GANs): Challenges, Solutions, and Future Directions". In: ACM Comput. Surv. 54.3 (May 2021).
[54] Dominik Schnaus, Nikita Araslanov, and Daniel Cremers. "It's a (Blind) Match! Towards Vision-Language Correspondence without Parallel Data". 2025. arXiv: 2503. 24129 [cs. cv]
[55] Congzheng Song and Ananth Raghunathan. "Information Leakage in Embedding Models". 2020. arxiv: 2004. 00053 [cs LG]
13

---

[56] Shezheng Song, Xiaopeng Li, Shasha Li. Shan Zhao, Jie Yu, Jun Ma, Xiaoguang Mao, and Weimin Zhang. "How to Bridge the Gap between Modalities: A Comprehensive Survey on Multimodal Large Language Model". 2023. arXiv: 2311 07594 [cs.CL]
[57] Yingtao Tian and Jesse Engel. "Latent translation: Crossing modalities by bridging generative models". In: arXiv preprint 1txXvv:992200222 (2019).
[58] Liang Wang, Nan Yang. Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder. and Furu Wei. "Text Embeddings by Weakly-Supervised Contrastive Pre-training".
2024, arviv: 2212. 03533 [cs.CL) CL]
[59] Liwei Wang. Lunjia Hu, Jiayuan Gu, Yue Wu, Zhiqiang Hu, Kun He, and John Hopcroft. "Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation" 2018 8. arXiv: 1810. 11750 [cs. LG]
[60] Wenhai Wang. Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. "VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks". 2023. arXiv: 2305 11175 [cs. [V]
[61] Christopher Wolfram and Aaron Schein. "Layers at similar depths generate similar activations across I1m architectures". In: arXiv preprint ar Xii:22504008777. (2025).
[62] Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. "Normalized Word Embedding and Orthog- onal Transform for Bilingual Word Translation". In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies. Ed. by Rada Mihalcea, Joyce Chai, and Anoop Sarkar. Denver, Colorado: Association for Computational Linguistics, May 2015, pp. 1006-1011.
[63] Hiroaki Yamagiwa, Momose Oyama, and Hidetoshi Shimodaira. "Discovering Universal Geometry in Embeddings with ICA". 2023. arxiv: 2305 13175 [ca. CL.]
[64]
Zhen Yang. Wei Chen, Feng Wang, and Bo Xu. "Unsupervised Neural Machine Translation with Weight Sharing". 2018. Xrv: 1804. 09057 [cs.CL)
[65]
Jinsung Yoon and Sercan O Arik. Brabbedding-Convereee A Unified Framework for Cross- Model Embedding Transformation" 2025.
[66] Collin Zhang, John X. Morris, and Vitaly Shmatikov. "Extracting Prompts by Inverting LLM Outputs". 2024. arXiv: 2405 15012 (cs.CL)
[67] Collin Zhang, John X. Morris, and Vitaly Shmatikov. "Universal Zero-shot Embedding Inver- sion". 2025. arXiv: 2504.00147 [C]
[68] Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. "Jasper and Stella: distillation of SOTA embedding models". 2025. arXiv: 2412. 19048 [cs. IR]
[69] Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang. Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. "Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models". In: arXiv preprint
arXiv:2506.05//1 (2025).
[70] Jun- Zhu, Taesung Park, Phillip Isola, and Alexei Efros. "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks". 2020. arXiv: 1703. 10593
[c8.CV]
14

---

A
Compute
Our training and evaluation were conducted using diverse compute environments, including both local and cloud GPU clusters. Experiments were done on NVIDIA 2080Ti, L4, A40, and A100 GPUs, listed in order of increasing computational capacity.
For our final results, we trained 25 vec2vec models fully and 30 models partially (see Appendix E The full models' training durations usually ranged from and model pair (which affected convergence rates). Partial convergence was stopped after days. Due to the size of Qwen, our (qwen, gte) ablation was trained for 20 days on an A 100. Taking conservative estimate of the average training time, this amounted to approximately 176 GPU days (24 models days/ model 30 models days / model + (qwen, gte) x 20 days model).
days, depending on the specific GPU
Evaluation procedures varied by model type:
The 10 main vec2vec models required ~1 hour each for NQ, TweetTopic, and MIMIC evaluation (across GPU types), plus 30 minutes for attribute extraction on TweetTopic and MIMIC, and 1.5 hours for inversion and downstream LLM evaluation on Enron and TweerTopic. Naive baselines required ~30 minutes each across all datasets.
The 15 additional fully-trained models required 30 minutes each for NQ evaluation, with an extra 30 minutes for MS COCO evaluation of (clip, granite).
Optimal transport baselines ran on CPU only, requiring ~ hour per dataset (three datasets for main models, one for others).
In total, our experiments consumed almost 176 GPU days for training and an additional 42 GPU hours for evaluation and analysis. An additional 45 CPU hours were required for optimal transport.
B Oracle-aided optimal transport baseline
Let u; M2 and v, M2(d,) denote embeddings of the same document d; from two different embedding models. In Section 5 we solve the optimal assignment problem:
arg min cos(ui, Ux(())
using four algorithms: Hungarian (linear sum assignment), Earth Mover's Distance (EMD), Sinkhorn, Wasnov--essrtt.inn For the Gromov- Wasserstein algorithm, we try both the entropic and non- entropic variants with multiple hyperparameter configurations and select the best figure. Note that the optimal transport (OT) baseline computes matchings and transports between embeddings derived from the same underlying lexts, strongly favoring OT methods. Nevertheless, OT still struggles when embeddings originate from different model backbones.
Since the Hungarian algorithm produces discrete matching, it is evaluated only using Top-1 Accuracy, while the other algorithms are evaluated across all metrics. For each experiment, the lowest-rank solver is reported in Table 2 and Table 4 (denoted by symbols in the final column). Evaluation metrics are defined as follows:
Top-1 Accuracy: Fraction of embeddings correctly identified as closest pairs, calculated by either selecting the maximum transported mass per embedding applying the Hungarian algorithm directly to the transport plan P. We report the higher accuracy between the two.
2. Mean Rank:
Average rank position of the correct embedding match vs when sorted by descending transported mass P.j from uy:
rank (v.) = position of e among sorted Psj.
3. Mean Cosine Similarity: Average cosine similarity between barycenters and true counter- parts:
j= PijUi
Similarity
cos(ol, v:).
ΣØ=1 P.j
15

---

C Translating to and from Qwen
Vec2vec
Baseline
M1
M2
cos(-) (-) f
T-I
Rank 4
ons(.) f
T-I
gte
0.50 (0.0)
0.92
2.28 (0.2)
0.38 (0.0)
0.00
425.28 (1.1²
gie
0.84 (0.0)
0.88
2.49 (0.3)
0.85 (9.0)
0.00
425.07 (1.2)*
Table 8: Translations between GTE and Qwen embeddings trained on NQ and evaluated on a 65536 text subset of NQ (chunked in batches of size 1024). Rank varies from 1 to 1024, thus 512 corresponds to random ordering. Since the embedding dimensionalities are different, only the Chomov-Wassersie OT baseline is run and the naive baseline does apply. Bold denotes best value.
As shown in Table vec2vec successfully translates between GTE and Qwen, significantly outper- forming the optimal transport baseline in all metrics except qwen gte cosine similarity, which we hypothesize may be due to the substantial performance gap between the models-indeed, Qwen differs from GTE in architecture (dense Qwen backbone), training methodology (unsupervised + model merging techniques), size (14x larger than the next largest model and 37 larger than GTE), context length, and recency. Given Qwen's 's size and computational cost, only evaluated this representative pair. We leave further evaluation to future work.
D Text-image retrieval on MS COCO
model
16
cos(.)
Rank
granite - clip
0.23
(0.0)
233.67
clip (haseline)
0.75
0.30 (0.0)
23.20 (08)
Table 9: Cross-model text-image retrieval on MS COCO: granite - clip vec2vec trained on NQ (unimodal) and evaluated on MS COCO's validation set. The rank metric varies from to 5000, thus 2500 corresponds to random ordering. Queries (captions) embedded with either Granite or CLIP. Documents (images) embedded with CLIP. Each caption has a unique image. Standard errors are shown in parentheses.
Our vec2vecs can "stitch" modalities onto unimodal models by translating to a multimodal model. To test this, we evaluated cross-modal text-image retrieval on MS COCO's validation set (5000 examples) (34), translating queries (captions) embedded with Granite to retrieve documents (images) embedded with CLIP using unimodal granite clip translator from section 4.3) Each caption has a unique image. We report Recall 16, cosine similarities, and Rank, with CLIP (for both documents and queries)
baseline.
As Table shows, translating Granite embeddings CLIP enables non-negligible cross-model multimodal retrieval with unimodal model for queries-aa despite zero multimodal training. Further evaluation of this paradigm with multimoda--ppecffee training a promising direction.
E
Initialization robustness by model backbone
GAN training is notoriously unstable to weight initialization (53). To measure our method's robust- trained fifteen e5 + gte (shared backbone) and e5 gtr (cross- (cross-backbone) vec2vecs on the
NQ dataset for a fixed 10 epochs.
For the translations between related models, vec2vec training was relatively stable across random seeds: 14 out 15 seeds achieved at least 80% top-1 accuracy within a fixed epoch budget, while the remaining run reached 72%. In contrast, translation between unrelated models proved significantly less stable, with only 3 out of 15 runs achieving convergence (80% top-1 accuracy). We leave improving the seed stability of our training regime future, valuable work.
16

---

F Full out-of-distribution translation results
We provide baseline numbers for the experiments shown in Table by dataset.
vec2vec
Naive Baseline
OT Baseline
E,
E2
cos(-) f
T-1
Rank 4
coe(-)
T-1
Rank 4
cos(-) f
1-1
Rank
gtr
0.74 (0UD)
0.99
1.09 (0.1)
-0.04 (0.0)
0.00
415.61 (8.2)
0.71 (0.0)
0.01
220.93 (2.1)
gra.
gie
ste.
0.85 (0.0)
0.77 (0.)
0.95
0.96
1.11 (0.))
1.26 (0.1)
0.00 (0.0)
0.00 (0.0)
0.00
0.00
406.77 76 (8.2)
417.27 (8.2)
0.87 (0.0)
0.74 (0.)
0.00
0.01
201 488 (6.6)*
239.36 (6:7)
e5
(000)
(0.7)
0.02 (0.0)
0.00
405.53 (8.1)
0.87 (00)
244.94 (7.4)²
0.79 (0.0)
0.98
2.41 (0:6)
-0.04 (0.0)
0.00
411,53 (8.3)
0.57 (0.0)
0.01
398.29 (8.2)
0.85 (0.0)
0.96
1.29 (0.2)
0.04 (0.0)
0.00
392.01 (8.2)
0.86 (0.D)
0.01
259.47 (7.4)
gtr
stc.
0.77 (0.0)
0.96
1.10 (0.0)
0.00 (0.0)
0.00
394.69 (8.3)
0.74 (0.0)
0.00
294.58 (7.0)²
e5
0.80 (0U0)
0.53
13.38 (1.2)
0.03 (0.0)
0.00
400.85 (8.2)
0.87 (0.0)
0.01
266.04 (3.7)*
0.73 (0.0)
0.94
1.33 (0.5)
0.00 (0.0)
0.00
408 81 (8.3)
0.56 (0.D)
0.01
398.16 (8.2)"
gle
gtr
0.71 (0.0)
0.95
1.29 (0.1)
0.04 (0.0)
0.00
386.58 (8.3)
0.71 (0.)
0.01
254.74 (7. (7.3)²
0.86 (0.0)
(0.0)
0.58 (0.0)
1,00
(0.0)
(0.0)
(0.0)"
e5
0.83 (OUD)
0.91
1.57 (0.2)
0.68 (0.0)
1.00
1.00 (0.0)
1.00 (0.D)
1,00
1,00 (0.0)
gra.
0.79 (0.0)
(0.1)
0.00 (0.0)
0.00
418. (8.4)
0.57 (0.0)
399.56 06 (8.2))
gtr
0.77 (0.0)
1.00 (0.0)
0.00 (0.0)
0.00
393.07 (8.1)
0.71 (0.0)
0.00
294.65 (7.4)
gte
0.90 (0.0)
1,00
1.00 (0.0)
0.58 (0.0)
1,00
1.00 (0.0)
1.00 (0.0)
1.00
1.00 (0.0)"
e5
0.85 (0.0)
(0.0)
0.37 (0.0)
0.89
(0.1)
(0.0)
1.00
1.00 (0.0)"
0.79 (0.0)
0.98
1.08 (0.0)
0,02 (0.0)
0,00
405.75 (8.3)
0.57 (0.0)
0.01
398.34 (8.2)
e5
gtr
0.67 (0.0)
0,80
3.10 (0.6)
0.03 (0.0)
0.00
401.16 (8.4)
0.71 (0.0)
0.00
268 28 (7.6)*
giz
(000)
(0.0)
0.68 (0.0)
(0.0)
(0.))
(0.01"
0.75
0.98
1.06 (0.0)
0.37 (0.0)
1,00
1,00 (0.0)
1.00
1,00
1,00 (0.0)"
Table 10: Out-of-distribution translations on ToeetTopic (with baselines): vec2vec models trained on NQ and evaluated on the entire TweetTopic test set (800 tweets). The rank metric varies from to 800, thus 400 corresponds to a random ordering. Standard errors are shown in parentheses. Symbols denote the lowest-rank solver: Earth Mover's 's Distance* and Gromov-Wasserster Wasserstein
vec2vec
Naive Baseline
OT Baseline
E1
E2
co6(-)
T-1
Rank
coa(-) 1
T-1
Rank
cos(-) f
T-1
Rank
gtr
0.74 (0.0)
0.60
23.38 (1.6)
-0.02 (0.0)
0.00
4010.00 (25.8)
0.82 (0.0)
0.00
3962.83 (26.1)
gra.
gle
0.85 (0.0)
0.08
346.21 (7.)
0.01 (0.0)
0.00
3978.35 (26.1) 1)
0.92 (0.0)
0.00
3808. (25.9))
stc.
0.72 (0.0)
0.13
242.23 (6.1)
-0.01 (0.0)
0.00
3900. (26.2)
0.86 (0.0)
0.02
3780.44 (26.0)
c5
0.84
0.12
361,06 (8.7)
0.02
0.00
4024,92 (26.1)
0.93
0.00
3937.63 (26.2)
gra.
0.78 (0.0)
0.51
35:27 (1.9)
.0.02 (0.0)
0.00
4023.67 (26.1)
0.87 (0.0)
0.00
3964.83 (26.1)
gtr
gle
0.84 (0.0)
0.12
279.56 (6:9)
0.08 (0.0)
0.00
4180.47 (26.2)
0.87 (0.0)
0.00
4088.97 (26.2)
sle.
0,72 (0.0)
0.27
127.92 (4.4)
0.00 (0.0)
0.00
4296.04 (26.1)
0.76 (0.0)
4095. (26.1)
0.82 (0.0)
0.01
1413,80 (18.3)
0.09 (0.0)
0.00
4064.47 (26.2)
0.93 (0.0)
0.00
4010.13 (26.1)"
gra.
0.73 (0.0)
0.09
342.15 (7.8)
0.01 (0.0)
0.00
3946.19
(25.8)
0.87 (0.0)
0.00
3802, 92 (25.9)"
gte
gtr
0.69 (0.0)
0.12
256.63 (6.4)
0.08 (0.0)
0.00
4229. 90 (26.2)
0.69 (0.0)
0.00
4094 02 (26. D)
stc.
0.85 (0.0)
1.00
1.00 (0.0)
0.56 (0.0)
1.00
1.00 (0.0)
(0.0)
1.00
1.00 (0.0)"
<5
0,86 (0.0)
0.54
17.71 (0:9)
0.69 (0.0)
0.98
1.04 (0.0)
1.00 (0:0)
1.00
1.00 (000)"
gra.
0.77 (0.00
0.14
221.95 (5:9)
.0.01 (0.0)
0.00
3951.42
(25.9)
0.87 (0.0)
0.01
3776.52 (26.0)"
gtr
0.75 (0.0)
0.56
17.70 (1.0)
0.00 (0.0)
0.00
4339.83 (26.2)
0.70 (0.0)
0.00
4093.61 (26.1)
gle
0,91 (0.0)
1.00
1.00 (0.0)
0,56 (0.00
1.00
1.00 (0.0)
1.00 (0:0)
1.00
1.00 (0.D)"
e5
0.85 (0.0)
0.51
26.33 (1.2)
0.35 (0.0)
0.59
12.68 (0.6)
0.93 (0.0)
1.00
1.00 (0.0)"
gra.
0.78 (0.0)
0.21
151.09
0.02
0.00
4008, 10 (25.9)
0.87
0.00
3932.58 (26.2)
gir
0.66
0.01
1029.64 (14.9)
0.09
0.00
4032.85 (26.2)
0.82 (0.0)
0.00
4010.06 (26.1)
gle
0.87 (0.00
0.60
32.59 (2.6)
0.69 (0.0)
0.98
1.09 (0.0)
1.00 (0:0)
1.00
1.00 (0.0)*
stc.
0.75 (0.0)
0.46
32.12 (1.4)
0.35 (0.0)
0.86
2.49 (0.1)
0.86 (0.0)
1.00
1.01 (0.0)"
Table 11: Out-of-distribttion translations on MIMIC (with baselines): vec2vec models trained on NQ and evaluated on an 8192-record subset of MIMIC. The rank metric varies from I to 8192, thus 4096 corresponds to a random ordering. Standard errors are shown in parentheses. Symbols denote the lowest-rank solver: Earth Mover's Distance", Sinkhorn and Gromov- Sasoov-ssssrssenn
17

---

G Zero-shot inversion on TweetTopic
Percentage of Tweets With Leakage
65
gte granite e5
43
42
27
47
60
51
52
34
52
55
50
M,
46
38
39
62
45
stella gtr
39
48
50
54
35
57
42
67
47
30
e5
granite
gte
gtr
stella
M2
Figure 7: Leakage of information via inversion. Trained on NQ and evaluated on a 50-tweet subset of the TweetTopic dataset. Cells denote judge accuracy.
We replicate Figure 5 on the TweetTopic dataset in Figure 7 extracting information given only translated embeddings tweets. We achieve non-negligible leakage for all model pairs.
H Prompt for measuring information extraction
Figures 5 and 7 utilize an LLM judge to determine whether the approximate inversion (using 667 of a translated email embedding leaks information about the original email. Figure shows our prompt to the GPT-4o judge:
Original email:
(ground |
Reconstructed email: [generation]. Does the reconstructed email leak any information about the original email? Answer with only 'yes' or 'no'.
Figure 8: The prompt given to the LLM judge.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claim is that we are (1) able to translate embeddings with high cosine similarity across model pairs with different architectures, parameter counts, and training datasets and (2) these translations can be used for attribute inference. These are the two experimental sections exactly.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The key limitation of our method is the training stability of GANs. Downstream of this limitation is long computation times and limited (while still effective) translation quality. This is discussed at length in the discussion.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not show any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Along with our releasing our code (used for each of the experiments present), we describe in depth the loss functions and architecture choices we made. We also ablate these choices in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We link to well-documented code that will help reproduce any of our experi- ments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
mublic//gudess/Cdeettbmisss oow.aaammmsiiiooooooooooooeee for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips /nipsscc/puut cttp:/scccpbbblcccuicceeeecc guides http:///lwlllliiiiee/oo//uuaaaaa
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All details necessary to reconstruct our experiments are contained in our code. On top of that we describe our method length in the body of paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: Wherever applicable we show error bars: the rank and cosine metrics, namely. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We include compute information in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpts:/t///
Answer: [Yes]
Justification: All of our experiments are on aublicly-avallable datasets that are either fully anonymous or pseudonymized.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: On the positive side, we provide evidence for a popular hypothesis in the ML community. On the negative side, we highlight, discuss, and provide remediation ideas for emerging threat. As with many security-flaooe papers, we hope increase visibility of emerging threats.
22

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., denloymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Although we have none planned, future releases of model weights would be of translators between known, open-source embedding spaces. These would not be very useful in exfiltrating data from unknown embeddings (a concern we highlight).
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All data is publicly available, cited, and used as dictated by the licenses and terms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
23

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our public code release will include documentation on how exactly to recon- struct our experiments.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We did not use crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did use crowdsourcing human subjects.
Guidelines:
24

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our paper's methodology does involve LLMs as any important, original, or non-standard components,
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy hbte:eee//eerrrsscc////offeerrre for what should or should not be described.
25


Reviews:
[{"review_id": "ujkKcVh86o", "reviewer": "Reviewer_5c82", "summary": "The authors point out that the embedding spaces of current text encoders tend to converge. Based on this assumption, they propose a novel method called vec2vec, which transforms the text embeddings from one model to another. This is a completely new unsupervised approach that does not require paired data—only the vector embeddings from the source model and access to the target model are needed. The authors decompose the embedding translation task into three components: adapter, shared backbone, and adapter. And they employ the idea of cycle-GAN for training. The proposed method achieves promising results on the embedding translation task.", "questions": "1.\tThe authors need to improve the zero-shot inversion component and provide a reasonable evaluation of the reconstruction percentage on datasets such as TweetTopic and MIMIC.\n2.\tThe authors should conduct experimental validation on a broader and more advanced set of language models, such as GPT-2 or LLaMA3, to support their hypothesis.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 1, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.\tThis paper introduces a novel scenario based on the assumption that the embedding spaces of text encoders tend to converge. Under this assumption, it enables the transformation of embeddings from one model into the feature space of another.\n2.\tThe authors propose the vec2vec framework to realize this scenario and demonstrate its effectiveness to some extent.\nWeaknesses:\n1.\tThe Strong Platonic Representation Hypothesis proposed by the authors for text models lacks extensive empirical validation. The authors only selected six models for embedding translation, which provides insufficient evidence to support the general applicability of this hypothesis across modern text encoders of varying sizes, architectures, and training data. In particular, there is a lack of evaluation on the GPT series, which are more widely used in the language domain.\n2.\tThe core application of this work should be zero-shot inversion—recovering the original text represented by the source model’s embeddings. However, the results in this area are not very strong.", "was_revised": true, "final_justification": "This paper offers a degree of novelty and contribution. However, because the zero-shot inversion demonstration is not compelling, the work has notable limitations and insufficient practical utility in real-world scenarios. I therefore maintain my original score unchanged.", "rebuttal": "Thank you for the helpful review. We attempted to address each of your comments below, broken down by weakness and question.\n\n---\n**(W2, Q1):** Thank you for the suggestion, we are evaluating inversions on TweetTopic (to complement the existing evaluations on MIMIC) for the next version of our paper. While our zero-shot inversion numbers are not as strong as methods with access to the ground-truth encoder or other auxiliary information (e.g., [1, 2]), we emphasize that our setting is different and harder.\n\nPrior to our work, **it was not known whether anything *at all* can be extracted from embedding vectors without prior information** (i.e., with no supervision, no paired data, and no knowledge of the encoder). That zero-shot inversion is even possible in this setting is a significant result.\n\nEven though we use zero-shot inverters designed for non-noisy embeddings, they still extract sensitive information from our noisy translations. Figures 5 and 6 show that inverted embeddings leak names, entities, and IP. Our experiments show that as many as 80% of inversions reveal confidential information. \n\nFurthermore, sensitive information can also be revealed using zero-shot attribute extraction directly on translated embedding vectors. As we show in Table 5, our translations reveal nearly as much sensitive information about medical records and corporate emails as the ground-truth embeddings. We expect that future work on better inversion methods in the unsupervised, unpaired translation setting will demonstrate even more and finer-grained information leakage from embeddings.\n\nWe will reinforce the above points in the introduction and discussion sections.\n\n[1] Morris, et al. “Text Embeddings Reveal (Almost) As Much As Text”. 2023.\n\n[2] Zhang, et al. “Universal Zero-shot Embedding Inversion”. 2025.\n\n---\n**(W1, Q2):** The six models in our evaluation cover a broad range of realistic embedding models:\n* Six different producers / organizations / distinct training datasets and regimes\n* Three standard encoder sizes: ~100M, ~150M, and ~300M\n* Four different architectures: T5, BERT, RoBERTa, CLIP\n* Unimodal and multimodal (CLIP) training\n* English-only and multilingual (Granite) training\n* Four “vintages”: 2021-2024 (eg, Granite is more recent than Llama3).\n\nFour of the models in our evaluation (E5, GTE, Stella, and Granite) actually **outperform** Llama3 on both Multilingual and English MTEB.  GTR performs comparably to Llama3. CLIP and GPT2 are not measured. Each of our models was near-SotA at the time of their release. \n\nDue to computational constraints (see Appendix A for details), we could not perform a comprehensive evaluation on 7B models. We agree that it is interesting to evaluate our methods on bigger models and are currently training vec2vec on 4B Qwen embeddings (which are newer and more commonly used than GPT2 and Llama3)."}, {"review_id": "itAzqYKC8q", "reviewer": "Reviewer_vYwp", "summary": "This work proposes a GAN-based unsupervised approach to translate embedding space of different text encoders to an aligned latent embedding space. The method does not rely on parallel data points, and only relies on embeddings encoded by different text embedding models. Similar to the Platonic Representation Hypothesis paper, the paper reveals the similar semantic structures that different text encoders learn to model. The paper also briefly shows that embedding inversion models can zero-shot apply on translated embeddings to inverse texts from embeddings, which show certain safety-related implications.", "questions": "See above.", "limitations": "See above.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Different from approaches that require parallel data points to align embedding space, such as those in multilingual and translation fields, etc., the proposed method does not require parallel data points.\n2. The behavior of the method is quite comprehensively evaluated, on different tasks (including whether the translated embedding can retrieve the paired data point encoded by the target encoder; and on attribute inference and embedding to text inversion) and on texts of OOD domains.\n\nWeaknesses:\n1. From the current narrative of the abstract and introduction, it is not immediately clear what's the motivation of the proposed method, and whether the focus of the work is to reveal universal semantic structures learned by different encoders (``The strong Platonic Representation Hypothesis'') or proposing a method. If the former, the proposed hypothesis seems under-theorized to distinguish from the original Platonic Representation Hypothesis, and if the latter, the motivation for the method is a bit vague (see point 2).\n2. It is not very well-justified why expecting embeddings of parallel data is unrealistic (line 47) and what is \"our setting\" - it seems like not much relevant description is provided before this section. For most cases, it is quite realistic to encode a few hundred texts with two encoders and learn even a simple linear projection to align them, even if one or both of them are API models. It might help to justify the setting more from the safety aspect, e.g., cases of vector base breach that risks text inversion, and put more emphasis on the text inversion experiments in the paper.\n3. Experiment details (model details, hyperparameters, compute requirements etc) are not at all described in 4.1 other than briefly mentioning that GAN is not stable.\n4. Certain overclaims exist in the paper including the framing of the method even though it's in spirit very similar to methods like CycleGAN.", "was_revised": true, "final_justification": "As mentioned in the original review comments, there are weaknesses in the current version regarding unclear narrative (whether the focus of the paper is the hypothesis or the method), unclear motivation (why assuming access to paired data points from different encoders are unrealistic), certain overclaims (claimed novelty despite similar to CycleGAN), and unclear experiment settings in 4.1 other than briefly mentioning GAN is unstable. In the rebuttal, the authors partly addressed these concerns and promised to clarify these unclear descriptions (main narrative/motivation, experiment settings, contextualize the overclaims) in the next version. The score has been raised to 4 to reflect authors' efforts in the rebuttal.", "rebuttal": "Thank you for the constructive review. We address it point by point, below.\n\n---\n**(W1):** Thank you for pointing out that the abstract and introduction are confusing.  We will clarify in the next version that we are proposing **both a hypothesis and a method.**\n\nThe original Platonic Representation Hypothesis (PRH) conjectures that all image models of sufficient size converge to the same latent representations. This version of PRH is not constructive, it does not yield a working method for **pure representation translation**: given a representation of one model (and no other information!), translate it into a semantically equivalent representation of another model.\n\nOur work extends the PRH constructively to text models.  We propose and establish empirically not only that universal latent representations exist, but also that they can be explicitly characterized. We give the first method that operates exclusively on the embedding vectors of large text sequences and translates them from one vector space to another, while preserving a significant amount of semantic information about the underlying inputs.\n\n---\n\n**(W2):** Thank you for pointing out that our wording is unclear.  To clarify, the existence of a universal latent representation and the constructive ability to translate embeddings to and from this representation without any additional information and no access to the encoder are important from both the fundamental perspective and the security perspective. \\\n \\\nFundamental perspective: we aim to answer a basic question, **how much information about a text input is preserved by its embedding representation?** In other words, how much information is contained in the vector itself, with no access to the encoder, no paired data, no candidate inputs, no parallels in another vector space, and no frequency statistics?  We believe that this is a fundamentally interesting and important question regardless of the application setting. \\\n \\\nSecurity perspective: when a vector database is compromised or leaked, often the adversary gains access only to the raw contents of the database, i.e., embedding vectors.  While in some settings auxiliary information may be available, this is not guaranteed.  Our results establish a basic lower bound: any vector database compromise will leak at least as much information as can be inferred by translation into a known embedding space + zero-shot inversion.  As we show, this already includes a serious amount of sensitive information. \\\n \\\nWe will clarify this in the next version of the paper.\n\n---\n\n**(W3):** Sorry for the confusion. Model details are provided at the end of Section 3.1. Compute requirements are described in detail in Appendix A. We adapt CycleGAN to function on embeddings instead of images by replacing CNNs with residual MLPs. We additionally utilize layer normalization and SiLU nonlinearities. A single vec2vec training run takes anywhere from 1 to 7 days to train on NVIDIA 2080Tis, L4s, A40s, and A100s.\n\nAll of our code is published with data, hyperparameters, and instructions on how to recreate each experiment. We will clarify this in the main text and add additional hyperparameters to the Appendix.\n\nWe will also add analysis on the stability of our method. Based on reviewer feedback, we analyzed the impact of initialization on convergence and found that translations between related models (e.g., BERT-backbone) are stable across random seeds. When translating embeddings from E5 to GTE, 14 out of 15 seeds converged, achieving at least 80% top-1 accuracy within a fixed epoch budget. The remaining run reached >70%.\n\nIn contrast, translation between unrelated models, such as from E5 to GTR (cross-backbone), proved significantly less stable, with only 3 out of 15 runs achieving convergence. We are currently working to make this regime more stable, and in future work plan to revisit the problem. In the meantime, we argue that showing the first evidence that these solutions exist is an important contribution.\n\n---\n\n**(W4--Part 1):** Thank you for the comment.  We are committed to making sure each of our claims is backed up. To that end, we would be happy to contextualize any specific claims.\n\nIn our own review of the submitted version of the paper, we identified this claim “vec2vec is the first method to successfully translate embeddings from the space of one model to another without any paired data” as potentially too broad, and modified to provide the following context: “Prior work has successfully translated word embeddings between languages, typically relying on overlapping vocabularies across languages. In contrast, we translate embeddings of entire sequences between model spaces.“\n\n---\n**(W4--Part 2):** While based on CycleGAN, our work is the first to use this method for pure representation translation, i.e., to learn a universal representation for translating embeddings of entire text sequences between model spaces **without encoder access or paired data**.\n\nPreviously, CycleGAN was used for style transfer in images [1] and standard GANs were used to translate word embeddings between languages, typically relying on overlapping vocabularies across languages [2].  As we demonstrate in Appendix C, CycleGAN does not work out of the box. To create vec2vec, we apply an additional GAN on the latent space and introduce an extra loss term (VSP) for the first time. Our ablations in the appendix show that both are critical for pure representation translation. When translating from GTE to GTR on NQ, vec2vec achieves an average rank of 2.64. If any one component of our method is removed (VSP loss, CC loss, latent GAN loss, or hyperparameter tuning), performance drops close to random guessing, with ranks of 4196.64, 3941.36, 3897.09, and 4011.73, respectively. Furthermore, cosine similarities plummet from 0.75 to ~0.5, and accuracies drop from 91% to 0%. \n\n[1] Huang, et al. “Multimodal Unsupervised Image-to-Image Translation”. 2018. \n\n[2] Conneau, et al. “Word Translation Without Parallel Data”. 2018."}, {"review_id": "urlcT0P67q", "reviewer": "Reviewer_LNxu", "summary": "The main contribution of the paper is a method (named \"vec2vec\") for learning model-agnostic vector representations of text documents. This leans on a representation hypothesis which states that given sufficiently many data and model capacity, there should be a common latent representation. Further, the authors conjecture that for text this is also possible without enforced alignment of known pairs of data from different representations. Using mappings that connect vector representations learned by different models, the authors leverage those model-agnostic representations as a \"bridge\" or \"universal representation\". This is achieved assuming only access to a learned set of vector representations of some unknown text data and then a known data set.", "questions": "1. Given how the chosen \"oracle-aided optimal assignment\" performed in the experiments sometimes abysmally, it made me wonder why that was the case. It seems the paper did not address that, at least per experiment. Could you clarify this?\n2. Following (1), could that be an indicator that this is not an appropriate baseline or another \"upper bound\" baseline could be adequate?\n3. Could the authors clarify how often does a training run need to be restarted to achieve a converged optimisation procedure?", "limitations": "The major limitation I see is caused by the lack of stable training. I appreciate that the authors also discussed direct societal impacts of their work, in particular to data leakage in the form of vector representations, being more serious than previously thought.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nOverall, the paper is clearly written, properly motivated, and the authors place it well amongst existing literature. Technically, the methodological build-up is easy to follow and intuitive. Importantly, the authors highlight the possibility of information extraction from leaked vector databases, which is critical from an information security standpoint.\n\n**Weaknesses**\n\nWhile intuitive, the methodology adopted is quite pragmatic, having a number of hyperparameters in the loss and architectural possibilities. As a concrete example of the pragmatism adopted, the authors forego investigation of instability in their proposed methodology, instead opting for investing more compute resources to achieve the desired outcomes. While I think that dedicating too much of the paper on making training stable could distract from the main point of the paper, it still passes an impression of being preliminary work. As I see it, it becomes one of the weakest points of the adopted methodology: the unstable training is rooted in the choices made that simplify the treatment of the problem as a more intuitive adversarial objective.\n\nAs a side note, I believe the chosen title is somewhat misleading, as there is no geometric treatment of the problem setting addressed in the paper. I would recommend the authors either clarify why this terminology is chosen at certain points (e.g., l. 56) in the paper or move away from it. As a suggestion, arguing in this direction using cosine similarities as evidence, you could consider using \"representation\" or \"vector alignment\" instead.", "was_revised": true, "final_justification": "Despite its pragmatic nature, the paper is experimentally thorough and shows itself to be a good first attempt at solving a tricky problem.\n\nWhile initially the authors wrote the paper in an optimistic tone, if they adopt the changes suggested by me and other reviewers, including clarifying stability (or lack thereof) and compute requirements, I am satisfied with the methodology adopted and recommend acceptance.", "rebuttal": "Thanks for your review–we’ll respond point-by-point below:\n\n---\n**(W1--Part 1):** While we agree that our method and framing are complex, our ablations (in Appendix C) show that **each part of our methodology appears necessary to learn translations** (that work better than random guessing)**.**  When translating from GTE to GTR on NQ, vec2vec achieves an average rank of 2.64. If any one component of our method is removed (VSP loss, CC loss, latent GAN loss, or hyperparameter tuning), performance drops close to random guessing, with ranks of 4196.64, 3941.36, 3897.09, and 4011.73, respectively. Furthermore, cosine similarities plummet from 0.75 to ~0.5, and accuracies drop from 91% to 0%.\n\nAs demonstrated by the failure of oracle-aided optimal assignment baselines, the problem is hard.  Prior to our work, it was not known or even expected that a black-box, unsupervised, no-encoder translation between text embeddings can be learned at all.  Our work provides the first constructive proof that such a method exists, and we hope that future research will improve it.\n\n---\n**(W1--Part 2):** We found that the adversarial setup was necessary to bridge the distance between the source and target embedding distributions in an unsupervised way. This formulation is standard in the literature [1, 2]. We experimented with other methods (such as iterative diffusion and normalizing flows) but they did not succeed at learning translations.\n\n[1] Conneau, et al. “Word Translation Without Parallel Data”. 2018.\n\n[2] Huang, et al. “Multimodal Unsupervised Image-to-Image Translation”. 2018.\n\n---\n**(W2):** Thank you for the note. Our use of the word “geometry” comes from the fact that semantic similarity in embedding spaces is typically measured via geometric angles between embedding vectors.  In linguistics, “geometry” is used to describe the shape or structure of semantics, i.e., how words and phrases are related). \n\n---\n**(Q1, Q2):** We will add the following discussion to our paper. During our evaluation, we were also surprised that oracle-aided optimal assignment did not work well for all model choices. A few things to note: \n\n1. The baselines do work when the underlying encoder models are from the same family (in our experiments, on any pairing of e5, gte, and stella models).\n2. Even though these baselines support unsupervised assignment of **single-word** embeddings, they are not designed for non-overlapping sets of longer text sequences.\n\n   When each embedding represents a single word, support sets for any large set of embedding vectors contain mostly the same semantic information (in different languages). For example, when translating from English to Spanish, the embeddings for ‘cat’ and ‘gato’ appear in the source and target distributions, respectively.\n\n   Our setting is different and significantly harder.  We work with embeddings of relatively long text sequences.  Given an embedding vector of an unknown sequence in the source embedding space, it is not guaranteed that the training set contains an embedding of the same or even closely related sequence in the target space.  This makes the assignment problem much more difficult.  Whereas for most words, the set of embeddings from the target space already contains an assignment target, this is not the case for text sequences.\n\n3. Better baselines: our work is the first to show that black-box translation is possible for text sequence embeddings in the non-overlapping, unsupervised regime.  Since there is no prior work for this setting, there are no natural baselines. \n\n   Instead, we use methods from the word translation literature as baselines. We include them in the Optimal Transport / Assignment baseline. We have added a section to the Appendix detailing these optimal transport (OT) experiments, covering the methods used, evaluation metrics, and experimental setup. Additionally, as illustrated below for the NQ dataset, we now denote the best-performing OT model using symbols (Earth Mover’s Distance: ∗, Sinkhorn: †, and Gromov-Wasserstein: ‡), and report the average cosine similarity between barycenters and their true counterparts.\n\n   We will further update this section once our evaluation of another baseline, entropic Gromov-Wasserstein, concludes this weekend (results below). \n\n---\n\n**(Q3):** Based on reviewer feedback, we analyzed the impact of initialization on convergence and found that translations between related models (e.g., BERT-backbone) are stable across random seeds. When translating embeddings from E5 to GTE, 14 out of 15 seeds ‘converged,’ achieving at least 80% top-1 accuracy within a fixed epoch budget; the remaining run reached >70%. In contrast, translation between unrelated models, such as from E5 to GTR (cross-backbone), proved significantly less stable, with only 3 out of 15 runs achieving convergence. We are currently trying to make this regime more stable, and in future work we will certainly revisit the problem. In the meantime, we argue that showing the first evidence that these solutions exist is an important contribution. This analysis will be added to the appendix of our paper.\n\n---\n### Updated Tables\n```\n+--------+--------+----------+--------+--------------+\n|             Updated OT Baseline Table              |\n+--------+--------+----------+--------+--------------+\n|   M₁   |   M₂   | cos(·) ↑ | T-1 ↑  | Rank ↓       |\n+--------+--------+----------+--------+--------------+\n| gra.   | gtr    | 0.50(0.0)| 0.00   |4094.22(9.2)‡ |\n|        | gte    | 0.85(0.0)| 0.00   |4069.91(9.3)‡ |\n|        | ste.   | 0.45(0.0)| 0.00   |4096.35(9.2)‡ |\n|        | e5     | 0.68(0.0)| 0.00   |4096.17(9.2)‡ |\n+--------+--------+----------+--------+--------------+\n| gtr    | gra.   | 0.50(0.0)| 0.00   |4093.55(9.2)‡ |\n|        | gte    | 0.85(0.0)| 0.00   |4079.92(9.2)‡ |\n|        | ste.   | 0.46(0.0)| 0.00   |4093.85(9.0)‡ |\n|        | e5     | 0.83(0.0)| 0.00   |4066.42(9.2)‡ |\n+--------+--------+----------+--------+--------------+\n| gte    | gra.   | 0.69(0.0)| 0.00   |4069.23(9.2)‡ |\n|        | gtr    | 0.70(0.0)| 0.00   |4078.45(9.0)‡ |\n|        | ste.   | 0.69(0.0)| 1.00   |1.00(0.0)†    |\n|        | e5     | 0.83(0.0)| 1.00   |1.00(0.0)†    |\n+--------+--------+----------+--------+--------------+\n| ste.   | gra.   | 0.48(0.0)| 0.00   |4096.38(9.2)‡ |\n|        | gtr    | 0.50(0.0)| 0.00   |4095.46(9.3)‡ |\n|        | gte    | 0.86(0.0)| 1.00   |1.00(0.0)†    |\n|        | e5     | 0.83(0.0)| 1.00   |1.00(0.0)†    |\n+--------+--------+----------+--------+--------------+\n| e5     | gra.   | 0.69(0.0)| 0.00   |4096.12(9.2)‡ |\n|        | gtr    | 0.70(0.0)| 0.00   |4065.74(9.2)‡ |\n|        | gte    | 0.86(0.0)| 1.00   |1.00(0.0)†    |\n|        | ste.   | 0.68(0.0)| 1.00   |1.00(0.0)†    |\n+--------+--------+----------+--------+--------------+\n```\n\n```\n+--------+--------+----------+--------+--------------+\n|            Entropic Gromov-Wasserstein             |\n+--------+--------+----------+--------+--------------+\n|   M₁   |   M₂   | cos(·) ↑ | T-1 ↑  | Rank ↓       |\n+--------+--------+----------+--------+--------------+\n| gte    | gra.   | 0.69(0.0)| 0.00   |2664.38(8.6)‡ |\n|        | gtr    | 0.70(0.0)| 0.00   |3064.16(8.9)‡ |\n|        | ste.   | 0.67(0.0)| 0.00   |3457.99(9.0)‡ |\n|        | e5     | 0.83(0.0)| 0.00   |3096.15(8.5)‡ |\n+--------+--------+----------+--------+--------------+\n| gtr    | gra.   | 0.70(0.0)| 0.00   |2775.17(8.6)‡ |\n|        | gte    | 0.85(0.0)| 0.00   |3070.69(8.9)‡ |\n|        | ste.   | 0.67(0.0)| 0.00   |3559.06(9.1)‡ |\n|        | e5     | 0.83(0.0)| 0.00   |3888.01(8.9)‡ |\n+--------+--------+----------+--------+--------------+\n```"}, {"review_id": "RLVuwFLIVh", "reviewer": "Reviewer_qFGw", "summary": "This paper learns how to build a mapping from a distribution of data encoded in one embedded space into another space (of the same dimension) where the encoding is know.  It is assumed that the distributions of the two datasets are known, and then by aligning the distributions with GAN-style loss function one can understand the meaning of embedded items in the first data set.  \n\nEmpirically, the paper shows that if the architectures have a lot of commonalities (same architecture, embedded elements random from same data set), then the method works quite well.  If not, the performance degrades some, but is still pretty good.", "questions": "I would like to understand how well this particular method worked against baselines.  Or why we could not compare against those.", "limitations": "yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n  + innovative approach, and provides some new evidence for the Plantonic Representation Hypothesis.  \n\n  + I was surprised to see how well it worked on datasets that shared a common backbone (although naive methods already work well), or hidden test was randomly split from test set.  \n\n  + I like the experimental design.  It tested a variety of things, on several architectures, evaluations/data sets, and types of data (text/image).  \n\n  + The paper is well written.  \n\n\nWeaknesses:\n  - There seems nothing specific to LLMs here, as opposed to work from 5-10 years ago that applied translation to non-contextual language models (e.g., word2vec, GloVe) for different languages. These ideas (which seem natural) could have been applied there.  This decreases the novelty to me.   \n\n  - I would have liked to see a direct comparison against translation of language models (e.g., Conneau etal 2018) as a better baseline.  What is gained with the multi-term GAN loss function in this paper.  \n\n  - The paper does not explain how it does alignment in full detail.  It says \"via either the Hungarian, Earth Mover’s Distance, Sinkhorn, or Gromov-Wasserstein algorithms, choosing the best-performing solver per experiment\" which is quite un-satisfying.", "was_revised": true, "final_justification": "I am convinced by the author's rebuttal and new experiments.  I think this is an important paper, that provides interesting insights into what is captured in language models, and will certainly spawn more work trying to replicated and improve upon their results.", "rebuttal": "Thank you for your response.  We respond point by point.\n\n---\n\n**(W1):** Methods for unsupervised transport on word embeddings do not solve our problem because they rely on heavily-overlapping word vocabularies between languages. For example, when translating from English to Spanish, ‘cat’ and ‘gato’ appear in both the source and target training data, respectively. \n\nOur setting is different and significantly harder.  We work with embeddings of relatively long text sequences.  Given the embedding of an unknown sequence in the source space, it is not guaranteed that the training set contains an embedding of the same or even a related sequence in the target space.  This makes our assignment problem–that of mapping entire documents, not just individual words–much more difficult.\n\nTables 2, 4, 6, 7 show that even when we use optimal assignment methods to compute translations between embeddings derived from **the same underlying texts** (we call this “oracle-aided”), they struggle to produce good assignments for embeddings that originate from different model backbones.\n\nWe have rewritten the introduction to clarify this distinction.\n\n---\n\n**(W2, Q1):** Conneau et al’s method is similar at a very high level (they also define the problem as adversarial learning) but it does not apply directly in our setting for the following reasons:\n\n1. Our non-overlapping setting (i.e., the inputs for the source and target embeddings during GAN training are not the same) breaks the key assumption behind the Procrustes step in their method.\n2. Since we don’t have access to the embeddings’ underlying text, refinement methods based on frequency proposed by Conneau et al cannot be used.\n\nWe do compare our approach with word translation methods that potentially apply in our setting.  For example, [1] requires neither overlapping supports, nor frequency-based analysis.\n\nTables 2, 4, 6, 7 in our paper measure the performance of OT methods, including [1].  Even with perfectly overlapping supports (an unrealistic assumption which favors OT), **OT works only for a few encoder pairs and fails when the encoders are from different model families.** For example, when translating from GTE to Stella on NQ (both BERT-based), the best OT method (in this case Sinkhorn) achieves a perfect rank and accuracy of 1.0 and 1.0. However, when translating from GTR to Stella, [1] (the best performing method for this pair) achieves only a near-random rank of 4095.46 and accuracy of 0.0. These numbers improve slightly (rank of 3559.06 and accuracy of 0.0) by introducing entropic regularization to [1], but they still fail for embedding translation and significantly underperform vec2vec.\n\nIn the next version of the paper, we will clarify the above and expand our comparison with entropic Gromov-Wasserstein once it concludes this weekend (results below).\n\n[1]  Alvarez-Melis, et al. “Gromov-Wasserstein Alignment of Word Embedding Spaces”. 2018.\n\n---\n\n**(W2):** For our GAN implementation, Appendix C demonstrates that **each term is critical.** When translating from GTE to GTR on NQ, vec2vec achieves an average rank of 2.64. If any one component of our method is removed (VSP loss, CC loss, latent GAN loss, or hyperparameter tuning), performance drops close to random guessing, with ranks of 4196.64, 3941.36, 3897.09, and 4011.73, respectively. Furthermore, cosine similarities plummet from 0.75 to ~0.5, and accuracies drop from 91% to 0%. \n\nWe will highlight this discussion in the paper.\t\n\n---\n\n**(W3):** Sorry about this. We have added a section to the Appendix detailing these optimal transport (OT) experiments, covering the methods used, evaluation metrics, and experimental setup. Additionally, as illustrated below for the NQ dataset, we now denote the best-performing OT model using symbols (Earth Mover’s Distance: ∗, Sinkhorn: †, and Gromov-Wasserstein: ‡), and report the average cosine similarity between barycenters and their true counterparts.\n\nWe will further update this section once our evaluation of another baseline, entropic Gromov-Wasserstein, concludes this weekend.\n\n---\n\n### Updated Tables\n```\n+--------+--------+----------+--------+--------------+\n|             Updated OT Baseline Table              |\n+--------+--------+----------+--------+--------------+\n|   M₁   |   M₂   | cos(·) ↑ | T-1 ↑  | Rank ↓       |\n+--------+--------+----------+--------+--------------+\n| gra.   | gtr    | 0.50(0.0)| 0.00   |4094.22(9.2)‡ |\n|        | gte    | 0.85(0.0)| 0.00   |4069.91(9.3)‡ |\n|        | ste.   | 0.45(0.0)| 0.00   |4096.35(9.2)‡ |\n|        | e5     | 0.68(0.0)| 0.00   |4096.17(9.2)‡ |\n+--------+--------+----------+--------+--------------+\n| gtr    | gra.   | 0.50(0.0)| 0.00   |4093.55(9.2)‡ |\n|        | gte    | 0.85(0.0)| 0.00   |4079.92(9.2)‡ |\n|        | ste.   | 0.46(0.0)| 0.00   |4093.85(9.0)‡ |\n|        | e5     | 0.83(0.0)| 0.00   |4066.42(9.2)‡ |\n+--------+--------+----------+--------+--------------+\n| gte    | gra.   | 0.69(0.0)| 0.00   |4069.23(9.2)‡ |\n|        | gtr    | 0.70(0.0)| 0.00   |4078.45(9.0)‡ |\n|        | ste.   | 0.69(0.0)| 1.00   |1.00(0.0)†    |\n|        | e5     | 0.83(0.0)| 1.00   |1.00(0.0)†    |\n+--------+--------+----------+--------+--------------+\n| ste.   | gra.   | 0.48(0.0)| 0.00   |4096.38(9.2)‡ |\n|        | gtr    | 0.50(0.0)| 0.00   |4095.46(9.3)‡ |\n|        | gte    | 0.86(0.0)| 1.00   |1.00(0.0)†    |\n|        | e5     | 0.83(0.0)| 1.00   |1.00(0.0)†    |\n+--------+--------+----------+--------+--------------+\n| e5     | gra.   | 0.69(0.0)| 0.00   |4096.12(9.2)‡ |\n|        | gtr    | 0.70(0.0)| 0.00   |4065.74(9.2)‡ |\n|        | gte    | 0.86(0.0)| 1.00   |1.00(0.0)†    |\n|        | ste.   | 0.68(0.0)| 1.00   |1.00(0.0)†    |\n+--------+--------+----------+--------+--------------+\n```\n\n```\n+--------+--------+----------+--------+--------------+\n|            Entropic Gromov-Wasserstein             |\n+--------+--------+----------+--------+--------------+\n|   M₁   |   M₂   | cos(·) ↑ | T-1 ↑  | Rank ↓       |\n+--------+--------+----------+--------+--------------+\n| gte    | gra.   | 0.69(0.0)| 0.00   |2664.38(8.6)‡ |\n|        | gtr    | 0.70(0.0)| 0.00   |3064.16(8.9)‡ |\n|        | ste.   | 0.67(0.0)| 0.00   |3457.99(9.0)‡ |\n|        | e5     | 0.83(0.0)| 0.00   |3096.15(8.5)‡ |\n+--------+--------+----------+--------+--------------+\n| gtr    | gra.   | 0.70(0.0)| 0.00   |2775.17(8.6)‡ |\n|        | gte    | 0.85(0.0)| 0.00   |3070.69(8.9)‡ |\n|        | ste.   | 0.67(0.0)| 0.00   |3559.06(9.1)‡ |\n|        | e5     | 0.83(0.0)| 0.00   |3888.01(8.9)‡ |\n+--------+--------+----------+--------+--------------+\n```"}]

Author rebuttal:

Meta-review: This paper considers a novel setting where we translate text embeddings between model spaces with no paired data or encoder access. The proposed vec2vec architecture (adapters + shared backbone with adversarial, cycle‑consistency, and vector‑space‑preservation losses) is a thoughtful adaptation of unpaired translation to the representation domain, with ablations indicating each term matters. Empirically, results are strong across multiple backbone families (BERT/T5/RoBERTa) and hold reasonably OOD. Main reservations are training instability and compute, some lingering over‑claiming tone vs. CycleGAN/word‑alignment precedents, and initially sparse details on OT comparisons (partly addressed during discussion).
Decision: Accept (poster)