Title: Isometry pursuit
Venue: NeurIPS 2025
Accepted: no
Keywords: Manifold learning

interpretability

sparse coding
Abstract: Isometry pursuit is a convex algorithm for identifying orthonormal column-submatrices of wide matrices.
It consists of a vector normalization followed by multitask basis pursuit.
Applied to Jacobians of putative coordinate functions, it helps identify locally isometric embeddings from within interpretable dictionaries.
We provide theoretical and experimental results justifying this method, including a proof with realistic assumptions that such isometric submatrices, should they exist, are contained within the obtained support.
For problems involving coordinate selection and diversification, it offers a synergistic alternative to greedy and brute force search.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Isometry Pursuit which is a convex approach to identify near-orthonormal subsets of columns from a wide matrix using normalization and multitask basis pursuit (MBP). The key theoretical contribution is a guarantee that if a true isometric (orthonormal) submatrix exists, then it is contained in the support of the relaxed MBP solution. The method is compared to a greedy baseline and evaluated on some datasets (such as Wine and Iris).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n+ The paper has a clear theoretical goal. In particular, it defines a clean and interpretable optimization problem: select D columns from a P-column matrix to approximate an orthonormal basis. \n+ The relaxation via multitask basis pursuit is standard but applied here in a targeted way. The main theorem (Proposition 4) states that if an orthonormal submatrix exists, it is in the support of the MBP solution. This is a good result under the assumption that such a submatrix exists.\n+ The experimental comparisons with greedy show modest gains. \n\nWeaknesses\n- Limited novelty: The method largely builds on known tools. In particular, the formulation is a direct application of group Lasso/multitask BP to a normalized design matrix. Also, the main theoretical guarantee is similar in spirit to known results in sparse recovery, especially in group sparsity contexts. The idea of trimming using convex relaxation before brute-force or greedy is standard in Lasso literature and feature selection. The paper feels more of a repackaging of known techniques and not a fundamentally new algorithmic or theoretical insight.\n- I found the motivation to be underdeveloped. The ties of the method proposed to interpretability, diversification, and embedding tasks are loose and not rigorously explored. For example, why is orthonormality the right or sufficient condition for interpretability or coordinate selection? How does this help in practice for downstream tasks (e.g., learning, generalization)? The introduction and discussion handwave at these points but do not really deliver on them, which weakens the practical relevance of the contribution.\n- The theory only applies when a true isometry exists. but no bounds are given on how well the method performs when no exact orthonormal subset exists. This is a major limitation. For instance, is the solution approximately orthogonal? Is there a bound on the deviation from orthogonality? Without such results, the utility of the algorithm in the general case is unclear.\n- The experiments are too simplistic. The datasets (Wine, Iris, Ethanol) are small and toy-like. There is no compelling large-scale use case shown. Also, comparisons are limited to simple greedy, but no comparisons are made to more modern subset selection methods. The runtime analysis is limited and does not rigorously show why the proposed method is superior.\n- There are several points where definitions or symbols are introduced abruptly or used vaguely. For example, the differential definitions (Def. 1 and 2) are not tightly integrated into the rest of the math. The normalization function q is defined in a confusing way. \n- Even if correct and moderately effective, the method does not offer a breakthrough for sure. The main theoretical result only applies to a very narrow case (existence of exact isometry), and the empirical gains are modest. The authors suggest applications to word embeddings, but these are not explored deeply.\n\nQUESTIONS:\n1) Motivation: Why is orthonormality the right or sufficient condition for interpretability or coordinate selection? How does this help in practice for downstream tasks (e.g., learning, generalization)?\n2) Theory: Is the solution approximately orthogonal in the approximate case where no isometry exists? Is there a bound on the deviation from unitarity? Can you derive such results? \n3) Experiments: Comparisons to other subset selection methods\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the problem of column selection from a large-scale matrix with the goal of having the select submatrix orthogonal (i.e., an isometry). Using a formulation via a particular norm minimization in Equation 23, they propose their algorithm in page 6, called isometry pursuit. They prove in Proposition 3 that it is rotationally invariant, and  the optimal solution is always feasible (Proposition 4). The paper is concluded with experiments.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n- motivating problem setting\n\n\nCons:\n\n- paper's writing can be substantially improved; some equations can be made look nicer, more aligned; the paper looks a bit unpolished\n\n- there is a minor issue with the paper formatting (and by no means I'm asking for rejecting for this--so this is just a heads up for the future modifcation). Line numbers are missing.\n\nQUESTIONS:\nThis is an interesting paper. Unfortunately, I found the paper less polished and needs major changed to be prepared for publication. The novelty of the results are not well explained, and more evidence should be given for applications of the method, too. \n\n### Major Comments/Question:\n\n\n- Section 3.1 is not clear. Before it, everything is clear and easy to follow, but in this section, it is not clear why you need a ground truth objective. The role of parameter c is not well discussed, and why you pick such exponential (cosh?) function. Why the loss function is maximized for orthogonal matrices? Why it is invariant under inversion? There is also a typo in notations there: Please either use $l$ or $\\ell$ (later is preferred in academic writing but it is important to keep consistency in the paper)\n\n- What does Definition 3 want to say, in words? I know that you provided explanations there but I still have trouble to parse it. That part is difficult to follow, though it is essential for understanding the rest of the paper. Also, after Equation 8 you mentioned that q is convex, but this violated Definition 3. \n\n- Algorithms are not labeled. \n\n- It is a bit hard to understand what the contribution of the paper really is. The paper is trying to present the results via a story (which is great) but unfortunately the challenges and the original ideas used to resolve the problem are not well discussed and explained. The theory is more or less not deep enough, and the contributions look incremental (based on the current writing of the paper--otherwise I believe the paper has non-trivial interesting contributions that are probably not well phrased). \n \n### Other Comments:\n\n\n- In Equation 3, please remind the reader the definition of loss. Also, it would be great if you can have a couple of examples about different loss functions.\n\n\n- Suggestion: for equation 3, use $\\operatorname{argmin}$; this makes things look nicer\n\n- After Equation 9, you say: \"Regardless ... non-convex.\" What do you mean there, because the problem is combinatorial and convexity is not even well-defined there. Please clarify. \n\n - Equation 23: What does w mean? Please include explanation in the paper\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new convex method called Isometry Pursuit for selecting approximately orthonormal submatrices from a wide matrix. The idea is to first normalize the column vectors in a specific way and then apply multitask basis pursuit to identify candidate subsets. The method is evaluated on synthetic and real datasets like Iris, Wine, Ethanol, and a small word embedding dataset. The authors provide both theoretical justification and empirical evidence that their method works well, especially when combined with a two-stage approach.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The method is backed by solid theoretical guarantees. In particular, Proposition 4 shows that the convex relaxation will retain any exactly orthonormal subset, if it exists.\n\n2. The experiments show that the proposed two-stage method can outperform greedy baselines in terms of a ground truth isometry loss.\n\n---\n\n**Weaknesses:**\n\n1. The time complexity of the two-stage method is not clearly stated, whereas the complexity of the brute-force algorithm is given as $O(C_l P^D)$.\n\n2. Apart from efficiency, the scalability of the proposed algorithms is not discussed. In Section 5, one potential application mentioned is in recommendation systems, which typically involve large-scale data where scalability is critical.\n\n3. It would be helpful for new readers if the paper included more examples to illustrate the importance of the target problem, i.e., selecting orthonormal column submatrices from wide matrices.\n\n4. In Section 5, the paper refers to potential applications in recommendation systems and RAG. However, it is not clear how the proposed method could be concretely applied in such scenarios.\n\nQUESTIONS:\n1. Are there any more realistic examples to better illustrate the necessity or significance of the targeted problem? The example, “Say you are hosting an elegant dinner party, and wish to select a balanced set of wines for drinking and flowers for decoration,” may be too brief or abstract to fully convey the scenario.\n\n2. Isometry Pursuit is a convex algorithm. Why can it produce non-unique solutions?\n\n3. Could you analyze the time complexity of the proposed method compared with that of the greedy algorithm? It would also be helpful to provide some empirical runtime results.\n\n4. Could you discuss the scalability of the proposed method, and how it could be applied in recommendation systems or RAG?\n\nLIMITATIONS:\nThe primary limitation appears to be that the proposed method may not always find the optimal solution.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-16"}