Title: SENSE: $\underline{\text{SEN}}$sing Similarity $\underline{\text{SE}}$eing Structure
Venue: NeurIPS 2025
Accepted: no
Keywords: Privacy Preserving

Distributed Learning

Federated Learning

Contrastive Neighbor Embedding

Hyperbolic Embedding

Matrix Completion
Abstract: Low-dimensional embeddings are central to analyzing and visualizing high-dimensional data. However, widely adopted NE methods assume centralized access to all data an unrealistic constraint in privacy-sensitive, decentralized environments. We propose \textbf{SENSE}, a geometry-aware, privacy-preserving framework for global neighbor embedding without raw data exchange. SENSE reconstructs global structure using local distance measurements and structured matrix completion, enabling embeddings that preserve both local and global geometry in Euclidean and hyperbolic spaces. It further integrates contrastive learning by deriving cross-client positive and negative pairs from estimated similarities, effectively generalizing negative sampling under structural constraints. Experiments across diverse real-world datasets show that SENSE achieves embedding quality on par with centralized baselines, while offering strong privacy guarantees. Theoretical analysis provides formal bounds on reconstruction fidelity and privacy, establishing conditions under which structure and confidentiality are jointly preserved. Our code is publicly available on this link: https://anonymous.4open.science/r/SENSE-SEnsing-Similarity-SEeing-Structure-2CB4/


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes SENSE, a  privacy-preserving framework for neighbor embedding and dimension reduction. It extends the SMACOF method used in classical MDS to handle recovery of all pairwise distances when only partial observations are available, and it uses existing methods for low-dimensional embedding such as t-SNE, UMAP and PHATE.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\nWhile neighbor embedding and dimensionality reduction have both been studied for long, decentralized neighbor embedding in privacy-constrained regimes may not be a well-explored sub-area. The strategy of employing anchors and decomposing the problem into a first matrix completion/reconstruction stage and a second low-dimensional embedding stage is creative and interesting. It is also good that some theory pertaining to the impossibility of reconstructing the data points themselves is presented. \n\nWeaknesses: \n\n- The paper has several claims that are either exaggerated or not well substantiated. For example, it is claimed that \"Theoretical\nanalysis provides formal bounds on reconstruction fidelity and privacy\" but I only found theoretical results pertaining to the impossibility of reconstructing the data in general. I could not find theory pertaining to fidelity. In that direction, it would be good to have positive statements proved for theoretical guarantees/error bounds the quality of reconstruction using projection-aware SMACOF.\n\n- Similarly, in the introduction, the paper mentions that SENSE overcomes the limitations of previous works in terms of cryptographic overhead, high communication and privacy costs etc. However, it doesn't demonstrate the framework's scalability to datasets bigger than different versions of MNIST and a runtime analysis of the SENSE pipeline (including the two stages involved) is missing.\n\n- There are some instances where the writing is somewhat sloppy. For example, NE in the abstract is used before being defined. In eq(2), $b$ seems to be a typo. In eq(5), it should be $\\hat{D}=\\mathcal{D}(\\hat{X})$, where $\\hat{X}=\\arg\\min...$. Also, the dimension of $\\hat{X}$ (equivalently the constraint $X'$) needs to be clarified. In section 2, $X$ denotes the original dataset, in section 3.1 it denotes the embedding.\n\nQUESTIONS:\n- Is non-exact/non-unique recovery enough for privacy in the real applications mentioned in this paper? If yes, then an explanation is needed, in order to emphasize the significance of theorems and connect them more closely to the applications. \n- As mentioned in the weaknesses section of this review, it would also be good to have  theoretical guarantees/error bounds on  the quality of reconstruction using projection-aware SMACOF (in order to justify the claims in the abstract and intro on the method's fidelity).\n- In the theorems, what happens if it is known (say, by an adversary) that the data is structured in some way. For example, say it lives on a low dimensional subspace of dimension $r$, with $r<K<d_h$, or if the data is approximately sparse? Wouldn't one be able to solve the linear system ($A_j x_j = b$) by using (say) regularization or constraining the solution to the structure set? \n\n- In the experimental set up, the paper states that  \"A subset of $10\\%$ of the total data points is designated as anchors\" but then states \"The total number of anchors (global+local) is fixed at $d_h-1$\". It seems from later discussion that $d_h-1$ is the criteria being used in practice? On a somewhat related note, is there a more disciplined way to determine the number of anchors and select the anchors beyond random choice?\n\n- I am curious: Can the two stages be merged (i.e., is there a fundamental reason why one has to do distance recovery first, as opposed to attempting to solve the whole problem in one go?).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This paper proposes privacy preserving SENSE framework, which utilizes local distance between anchor and non-anchor points to reconstruct global geometry using structured matrix completion.\n- The paper provides theoretic proofs on privacy guarantee, and demonstrates the framework’s application in Euclidean space, hyperbolic space, and evolving environment with new data entry. The framework has good compute efficiency. \n- Experiments showcased that the privacy preserving results are almost on par with centralized approaches on MNIST and German credit datasets, even when number of anchor points are low.\n\nSTRENGTHS AND WEAKNESSES:\nStrength \n- The authors provide solid theoretical foundations, problem formulation and framework. The proofs are thorough and rigorous.\n- The paper’s proposal is potentially useful in lots of downstream applications, where privacy preservation is important. The paper also demonstrates the uses in alternative embedding space (e.g. hyperbolic), and iterative experiment context.\n- The suggested method is efficient in computation compared to other privacy preserving mechanisms.   \n\nWeakness\n- While theorem 3.2 proves that the inverse map from anchor distances to features is non-unique, the distance itself may still reveal some information about the feature. \n- The experiments use simple datasets like MNIST and German credit. The performance is unclear on more sophisticated datasets. \n- The paper may benefit from better ablation of anchor selection criteria (e.g. random vs in distribution), on top of the increasing anchor ablations provided. \n- Nit: It would be helpful to introduce an anchor based method as part of the background. Also figure 1 needs to be clearer for people to grasp the concept of the paper at one go.\n\nQUESTIONS:\n- Does the non-anchor to anchor distance sent to the central server reveal any private information? How are the proposed framework robust to inversion attacks and membership inference attacks?\n- Qualitatively and theoretically, how does anchor selection strategy affect privacy robustness?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides techniques for obtaining embeddings from the NE family of dimensionality reduction methods, specifically in the setting where the data is *both* distributed across multiple servers and is sensitive (requiring privacy). This is done by using anchors which are consistent across the data chunks and estimating pairwise distances to the points using MDS which leverages the anchors. Since dimensionality reduction methods only need these pairwise distances, the authors can then produce an embedding from these estimated pairwise distances.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThis paper's main strength is in the thoroughness with which it approaches the methodology. The number of datasets is great, the chosen NE methods are natural and the experiments ablate appropriately across the relevant parameters. In addition to this, the related work is presented carefully and the methodology seems sound. It is nice that using the anchors mitigates the amount of data one must pass between sites while also accounting for integrating information between them effectively. Additionally, Section 3 is quite clear and easy to follow.\n\n## Weaknesses\n\nUnfortunately, the weaknesses significantly outweigh the strengths of the paper in my eyes.\n\n### Weakness 1: Definition of privacy\n\nThe paper repeatedly uses terms like \"privacy-preserving\", even suggesting in the last sentence of Theorem 1 that the outcome is a \"privacy guarantee\". However, the notion of privacy is... never defined?? I spent the entire paper trying to piece together what the authors might mean by \"privacy preserving\" and what is and isn't sufficient for privacy preservation. It appears that simply sending the data between sites is not privacy preserving. Similarly, the authors seem to be suggesting that sending the *correct* distances between points is also not privacy preserving (since their theorems require that we have less than $d_h$ anchors). However... using $d_h - 1$ anchors *is* privacy-preserving somehow? This is quite confusing to me -- Table 3 explicitly shows that there is $>0.99$ agreement between the \"original\" distances and the extrapolated ones. Table 6 similarly shows a $0.95$ similarity in many cases. So utilizing the original distances is not privacy-preserving but getting to within an arbitrarily good precision of the original distances is okay? Similarly, the MNIST plot in Figure 3a shows that we're getting essentially 100% accuracy on the normalized metric score using $d_h - 1$ anchors. Additionally, somehow the pointwise setting allows 10% of the points to be used as anchors... so some points can be shared between sites while other points cannot?\n\nThese apparent contradictions lead me to extrapolate that the authors are using the following definition for a \"privacy-preserving algorithm\": an algorithm is privacy-preserving if it does not send exact point positions or exact inter-point distances, but it *is* privacy preserving if it sends arbitrarily close-to-exact inter-point distances. Furthermore, a privacy-preserving algorithm is nonetheless allowed to share some percentage of the points between sites in an un-private manner.\n\nTo me, the above definition is consistent with the authors' experiments and presentation. However, I'm not sure that this is not a *useful* definition of an algorithm being privacy-preserving. Moreover, if one wished to use this as the definition, why not simply add a tiny amount of noise to the points and then compute their distances directly?\n\nIn short, the authors should provide a clear definition of what it means to be privacy-preserving and explain how their results exist within this definition.\n\n### Weakness 2: Strange presentation\n\nIn general, the text has moments where the presentation is quite strange. The entire introduction is quite difficult to parse (and I come from the field of NE embeddings). The first paragraph tries to set up (a) what NE methods are, (b) the difficulty of getting embeddings in distributed settings and (c) the difficulty of also getting embeddings in privacy-sensitive settings. Luckily, I know how these methods work, so I am not lost here yet, but I am confused by the following statements:\n- MDS is not an NE method (line 18)\n- Why would this be particularly problematic for t-SNE and UMAP (line 23-24)? All DR methods require distance information across all pairs of points.\n- Why does relating NE methods to contrastive learning further emphasize the importance of accurate pairwise similarities? (Line 25)\n\nAs a result, the first paragraph of the introduction does not adequately introduce what the actual challenges *are* for accomplishing the tasks the authors have set out to solve. Consequently, when the related work is presented in the second paragraph, it is completely unclear what we are trying to do and why these other methods fail at doing it. For example, I come away with 0 understanding of what SMAP is or why it doesn't work (it's unclear what the authors mean by \"its cryptographic overhead making [things] impractical\"). Similarly, I don't understand why FedNE lacks privacy guarantees or what dSNE and FdSNE are or why they diverge from standard FL protocols (or, indeed, what FL protocols even are). Then, the related work section paragraph concludes by introducing the authors' algorithm?\n\nI am then completely lost during the introduction's third paragraph. What are anchors in this setting? What is anchor-sharing across servers? Why would it be a constraint? Why would they not be a \"core architectural component\" and why is it important to argue that they are? \n\nIn general, it seems the authors make many assertions they are doing smart things, but I have no idea what these smart things are or, indeed, why they are smart. The bullet points in the introduction suffer from similar issues. Similarly, upon reading the introduction, I had absolutely 0 idea what Figure 1 was saying (it only made sense after Section 3).\n\nI am picking on the introduction in particular since it is where this issue is the most apparent, but similar points percolate throughout the paper:\n- CNE does not use an encoder, I believe (Line 90)\n- What are the metrics being used? I am intimately familiar with the NE space and have not heard of trustworthiness, continuity, steadiness or cohesiveness. Usually, people report knn recall or other local similarity preservation measures, since NE methods emphasize preserving local neighborhoods. Perhaps trustworthiness and continuity do this, but I am unsure since they are not defined in the paper.\n- I'm a bit confused by the theorems, but this is perhaps due to my personal misunderstanding. As I understand it, the authors are only calculating distances to anchor points and, therefore, between the 'private' points. If this is the case, then wouldn't it generally be impossible to exactly recover all of the $x_i$ points? Distances are rotation-invariant. So one could get all the pairwise distances and still not know the positions of the points, since they could be placed an infinite number of ways while still maintaining all the pairwise distances (due to the number of rotations being continuous and therefore infinite).\n\nQUESTIONS:\n1. What is the definition of \"privacy\" and \"privacy preservation\" that this paper uses?\n2. Why is it sufficient to have $d_h - 1$ anchors but suddenly $d_h$ of them fails? What if $d_h \\rightarrow \\infty$? Is the suggestion that something is privacy preserving if it has any amount of uncertainty at all?\n3. Why would it not be privacy preserving to simply add a tiny bit of noise to the data points and share their noisy distances directly? Could you then use $d_h$ anchors?\n4. What are the metrics you are using and why are these the natural choices rather than other, more standard ones?\n\nLIMITATIONS:\nThe paper does not have a formal limitations section. There are no societal impacts of the present work. However, a few limitations that I can think of off the top of my head are:\n- There is not a reference suggesting that somebody is eager to solve the problem the authors have set out to solve. I do believe it is interesting and potentially relevant, but it would be nice to see outside confirmation of that fact.\n- It's unclear how one would translate these techniques to other definitions of privacy.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-22"}