Title: Harnessing the Universal Geometry of Embeddings
Venue: NeurIPS 2025
Accepted: yes
Keywords: Embeddings

Text Embeddings

Representations

Representation Learning

Machine Learning

Security

Privacy
Abstract: We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches.  Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., a universal semantic structure conjectured by the Platonic Representation Hypothesis).  Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets.

The ability to translate unknown embeddings into a different space while preserving their geometry has serious implications for the security of vector databases.  An adversary with access only to embedding vectors can extract sensitive information about the underlying documents, sufficient for classification and attribute inference.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper learns how to build a mapping from a distribution of data encoded in one embedded space into another space (of the same dimension) where the encoding is know.  It is assumed that the distributions of the two datasets are known, and then by aligning the distributions with GAN-style loss function one can understand the meaning of embedded items in the first data set.  \n\nEmpirically, the paper shows that if the architectures have a lot of commonalities (same architecture, embedded elements random from same data set), then the method works quite well.  If not, the performance degrades some, but is still pretty good.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n  + innovative approach, and provides some new evidence for the Plantonic Representation Hypothesis.  \n\n  + I was surprised to see how well it worked on datasets that shared a common backbone (although naive methods already work well), or hidden test was randomly split from test set.  \n\n  + I like the experimental design.  It tested a variety of things, on several architectures, evaluations/data sets, and types of data (text/image).  \n\n  + The paper is well written.  \n\n\nWeaknesses:\n  - There seems nothing specific to LLMs here, as opposed to work from 5-10 years ago that applied translation to non-contextual language models (e.g., word2vec, GloVe) for different languages. These ideas (which seem natural) could have been applied there.  This decreases the novelty to me.   \n\n  - I would have liked to see a direct comparison against translation of language models (e.g., Conneau etal 2018) as a better baseline.  What is gained with the multi-term GAN loss function in this paper.  \n\n  - The paper does not explain how it does alignment in full detail.  It says \"via either the Hungarian, Earth Mover’s Distance, Sinkhorn, or Gromov-Wasserstein algorithms, choosing the best-performing solver per experiment\" which is quite un-satisfying.\n\nQUESTIONS:\nI would like to understand how well this particular method worked against baselines.  Or why we could not compare against those.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main contribution of the paper is a method (named \"vec2vec\") for learning model-agnostic vector representations of text documents. This leans on a representation hypothesis which states that given sufficiently many data and model capacity, there should be a common latent representation. Further, the authors conjecture that for text this is also possible without enforced alignment of known pairs of data from different representations. Using mappings that connect vector representations learned by different models, the authors leverage those model-agnostic representations as a \"bridge\" or \"universal representation\". This is achieved assuming only access to a learned set of vector representations of some unknown text data and then a known data set.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nOverall, the paper is clearly written, properly motivated, and the authors place it well amongst existing literature. Technically, the methodological build-up is easy to follow and intuitive. Importantly, the authors highlight the possibility of information extraction from leaked vector databases, which is critical from an information security standpoint.\n\n**Weaknesses**\n\nWhile intuitive, the methodology adopted is quite pragmatic, having a number of hyperparameters in the loss and architectural possibilities. As a concrete example of the pragmatism adopted, the authors forego investigation of instability in their proposed methodology, instead opting for investing more compute resources to achieve the desired outcomes. While I think that dedicating too much of the paper on making training stable could distract from the main point of the paper, it still passes an impression of being preliminary work. As I see it, it becomes one of the weakest points of the adopted methodology: the unstable training is rooted in the choices made that simplify the treatment of the problem as a more intuitive adversarial objective.\n\nAs a side note, I believe the chosen title is somewhat misleading, as there is no geometric treatment of the problem setting addressed in the paper. I would recommend the authors either clarify why this terminology is chosen at certain points (e.g., l. 56) in the paper or move away from it. As a suggestion, arguing in this direction using cosine similarities as evidence, you could consider using \"representation\" or \"vector alignment\" instead.\n\nQUESTIONS:\n1. Given how the chosen \"oracle-aided optimal assignment\" performed in the experiments sometimes abysmally, it made me wonder why that was the case. It seems the paper did not address that, at least per experiment. Could you clarify this?\n2. Following (1), could that be an indicator that this is not an appropriate baseline or another \"upper bound\" baseline could be adequate?\n3. Could the authors clarify how often does a training run need to be restarted to achieve a converged optimisation procedure?\n\nLIMITATIONS:\nThe major limitation I see is caused by the lack of stable training. I appreciate that the authors also discussed direct societal impacts of their work, in particular to data leakage in the form of vector representations, being more serious than previously thought.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a GAN-based unsupervised approach to translate embedding space of different text encoders to an aligned latent embedding space. The method does not rely on parallel data points, and only relies on embeddings encoded by different text embedding models. Similar to the Platonic Representation Hypothesis paper, the paper reveals the similar semantic structures that different text encoders learn to model. The paper also briefly shows that embedding inversion models can zero-shot apply on translated embeddings to inverse texts from embeddings, which show certain safety-related implications.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Different from approaches that require parallel data points to align embedding space, such as those in multilingual and translation fields, etc., the proposed method does not require parallel data points.\n2. The behavior of the method is quite comprehensively evaluated, on different tasks (including whether the translated embedding can retrieve the paired data point encoded by the target encoder; and on attribute inference and embedding to text inversion) and on texts of OOD domains.\n\nWeaknesses:\n1. From the current narrative of the abstract and introduction, it is not immediately clear what's the motivation of the proposed method, and whether the focus of the work is to reveal universal semantic structures learned by different encoders (``The strong Platonic Representation Hypothesis'') or proposing a method. If the former, the proposed hypothesis seems under-theorized to distinguish from the original Platonic Representation Hypothesis, and if the latter, the motivation for the method is a bit vague (see point 2).\n2. It is not very well-justified why expecting embeddings of parallel data is unrealistic (line 47) and what is \"our setting\" - it seems like not much relevant description is provided before this section. For most cases, it is quite realistic to encode a few hundred texts with two encoders and learn even a simple linear projection to align them, even if one or both of them are API models. It might help to justify the setting more from the safety aspect, e.g., cases of vector base breach that risks text inversion, and put more emphasis on the text inversion experiments in the paper.\n3. Experiment details (model details, hyperparameters, compute requirements etc) are not at all described in 4.1 other than briefly mentioning that GAN is not stable.\n4. Certain overclaims exist in the paper including the framing of the method even though it's in spirit very similar to methods like CycleGAN.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors point out that the embedding spaces of current text encoders tend to converge. Based on this assumption, they propose a novel method called vec2vec, which transforms the text embeddings from one model to another. This is a completely new unsupervised approach that does not require paired data—only the vector embeddings from the source model and access to the target model are needed. The authors decompose the embedding translation task into three components: adapter, shared backbone, and adapter. And they employ the idea of cycle-GAN for training. The proposed method achieves promising results on the embedding translation task.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.\tThis paper introduces a novel scenario based on the assumption that the embedding spaces of text encoders tend to converge. Under this assumption, it enables the transformation of embeddings from one model into the feature space of another.\n2.\tThe authors propose the vec2vec framework to realize this scenario and demonstrate its effectiveness to some extent.\nWeaknesses:\n1.\tThe Strong Platonic Representation Hypothesis proposed by the authors for text models lacks extensive empirical validation. The authors only selected six models for embedding translation, which provides insufficient evidence to support the general applicability of this hypothesis across modern text encoders of varying sizes, architectures, and training data. In particular, there is a lack of evaluation on the GPT series, which are more widely used in the language domain.\n2.\tThe core application of this work should be zero-shot inversion—recovering the original text represented by the source model’s embeddings. However, the results in this area are not very strong.\n\nQUESTIONS:\n1.\tThe authors need to improve the zero-shot inversion component and provide a reasonable evaluation of the reconstruction percentage on datasets such as TweetTopic and MIMIC.\n2.\tThe authors should conduct experimental validation on a broader and more advanced set of language models, such as GPT-2 or LLaMA3, to support their hypothesis.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-07-05"}