Title: On the rankability of visual embeddings
Venue: NeurIPS 2025
Accepted: yes
Keywords: ranking

visual representations

structured representations

ordinal regression
Abstract: We study whether visual embedding models capture continuous, ordinal attributes along linear directions, which we term _rank axes_. We define a model as _rankable_ for an attribute if projecting embeddings onto such an axis preserves the attribute's order. Across 7 popular encoders and 9 datasets with attributes like age, crowd count, head pose, aesthetics, and recency, we find that many embeddings are inherently rankable. Surprisingly, a small number of samples, or even just two extreme examples, often suffice to recover meaningful rank axes, without full-scale supervision. These findings open up new use cases for image ranking in vector databases and motivate further study into the structure and learning of rankable embeddings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes whether existing visual encoders have rankable embeddings. The authors define that an embedding space is rankable, if there exists a rank axis such that the projection onto it preserves the ordering of a continuous attribute. They claim that embeddings of modern visual encoders are highly rankable compared to lower bound metrics provided by the untrained-encoder baseline, and upper bound metrics provided by nonlinear regression and finetuned encoders. The paper also finds that a small number of extreme samples often suffice to recover meaningful rank axes.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The proposed algorithm in this paper demonstrates a reasonable formulation for analyzing whether existing encoders capture ordinal information. \n- The paper supports its claims through extensive evaluations over a diverse range of datasets and model architectures. \n- It also covers few-shot and zero-shot learning scenarios, showing that rank axes can be effectively recovered using only a small number of extreme samples. \n\n**Weaknesses**\n- Some important details are missing, such as how the nonlinear and fine-tuned methods were trained. \n- The paper lacks sufficient references to existing literature on methods for constructing ordinal embeddings. Are the performance gaps between linear regression over trained embeddings still small when models are trained using the loss functions proposed in [1], [2]?\n- For an embedding to be practically useful in real-world ranking tasks, it must have a high degree of ordinality. Given that there are existing encoders specifically trained to produce strong ordinal embeddings, it is unclear whether analyzing or relying on the ordinality of a generic encoder serves a meaningful purpose. \n- The paper lacks in-depth analysis on why certain attributes yield lower SRCCs. It instead relies on general, unsupported hypotheses(L183-188).\n- The number of datasets per attribute was limited to a maximum of 2~3. Given the abundance of datasets available for attributes such as age and aesthetics, a more extensive evaluation across a broader range of datasets would have further substantiated the generalization capability and rank axis transferability proposed in this paper. \n\n\n[1] Seon-Ho Lee, Nyeong-Ho Shin, and Chang-Su Kim. Geometric order learning for rank estimation. In Proc. NeurIPS, 2022.\n\n[2] Kaiwen Zha, Peng Cao, Jeany Son, Yuzhe Yang, and Dina Katabi. Rank-N-Contrast: Learning continuous representations for regression. In Proc. NeurIPS, 2023.\n\nQUESTIONS:\n- For the upper-bound settings, are the MLPs and encoders trained using just a regular MAE or MSE loss?\n\n- It seems that the few-shot performance would vary significantly depending on how the instances were sampled. If random sampling was used, how many trials were run to obtain the reported results?\n\n- Attributes like age have a large number of available datasets, such as MORPH II[1], CLAP2015[2], AgeDB[3], and CACD[4]. Would it be possible to conduct experiments on several of these additional datasets to further verify the rankability and rank axis transferability across a broader range of age datasets?\n\n[1] Karl Ricanek and Tamirat Tesafaye. MORPH: A longitudinal image database of normal adult age-progression. In Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition, 2006.\n\n[2] Sergio Escalera, Junior Fabian, Pablo Pardo, Xavier Baró, Jordi Gonzàlez, Hugo J. Escalante, Dusan Misevic, Ulrich Steiner, and Isabelle Guyon. ChaLearn looking at people 2015: Apparent age and cultural event recognition datasets and results. In Proc. ICCV Workshops, 2015. \n\n[3] Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. AgeDB: The first manually collected, in-the-wild age database. In Proc. CVPR, 2017.\n\n[4] Bor-Chun Chen, Chu-Song Chen, and Winston H Hsu. Face recognition and retrieval using cross-age reference coding with cross-age celebrity dataset. IEEE Trans. Multimedia, 2015.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows that image embeddings from common vision encoders can be ranked for specific attributes; e.g., for person counting or image quality. The method proposes different ways of learning such a rankable embedding by relying on linear or non-linear models to be trained on a small percentage of the dataset. Evaluation on a variety of tasks and attributes reveal that such rankable properties exist and that some attributes are easier to rank than others.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n*S1. The idea of learning rankable embeddings in common vision encoders is simple and effective.*\n* This reveals that common encoders have this information and it can be learned easily with a low percentage of the training set.\n\n*S2. The paper does not require fancy components, is applicable to a variety of vision encoders and tasks.*\n* It is appreciated that the paper assess a variety of vision encoders and tasks, showing its wide applicability.\n\n**Weaknesses**\n\n*W1. The claim of linear embedding is unverified*\n* The evaluation only focuses on measuring the Spearman rank correlation coefficient. What about the Pearson linear correlation coefficient?\n* It would have been interesting to also provide qualitative or quantitative analysis on algorithmic operations in the embedding space. Does moving by the same magnitude and direction yield the same effect? (e.g., the number of people in the image always increases by N with this vector)\n\n*W2. Few-shot claims are over-stated*\n* While it is interesting to find attribute embeddings from existing models, the method still requires a thousand of examples! It seems like 5-10% of the dataset is still needed to learn the attribute embeddings. This is in contrast with the few-shot literature where few-shot usually means less than 10 examples. I would suggest to rephrase \"few-shot\", or mentions of \"a small number of samples\" to something more accurate.\n\n*W3. Behavior of out-of-scope values is unclear*\n* The method requires to provide extreme values to learn how to rank. Let say we present examples ranging from 0 to 100. What happens if at inference we present an image where the value is 200?\n\n*W4. Comparisons are missing.*\n* The evaluation only focuses on the selected vision encoders on various ranking tasks. But, what about the SOTA methods in each of these respective tasks? It would have been interesting to know what is the upper-band.\n* Furthermore, the idea of ranking attributes had been recently explored in image quality assessment, where for example related work (e.g., [55], among many others) find a linear embedding between what is a \"bad photo\" and a \"good photo\". Direct comparisons with such methods is possible and would benefit the paper.\n\nQUESTIONS:\nI would be happy to increase my score following discussions on the linearity of the embedding, few-shot phrasing, and comparisons with very similar methods in the field of image quality.\n\nLIMITATIONS:\nThe paper has not discussed limitations or potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates whether visual encoders (like CLIP model) inherently encode ordinal attributes along linear directions in the embedding space, an ability which the authors name as rankability. Through extensive evaluation on multiple models across multiple attribute datasets, the authors state that existing visual embedding spaces are indeed rankable, and often times the rank axis of ordinal attribute can be estimated using only 2 extreme samples (one sample possessing the least of the attribute, one sample possessing the most). In addition, the authors also propose few-shot and zero-shot learning methods to estimate this rank axis, as well as analyze how well a rank axis is transferrable across different datasets (within the same attribute). In overall, this paper shows a different research direction into understanding the structure of visual embedding space (in addition to existing work that study hierarchies, disentanglement, separability of embedding space).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Good and clear writing.\n- Estimating rankability is a clearly defined, new problem for understanding embedding space that has not been explored.\n- Evaluation is done thoroughly across multiple models and attribute datasets, although the selected attributes are still a bit not diverse yet.\n- It’s interesting to learn that discovering rank axis may sometimes require only 2 extreme samples, or even as few as 10% of the training data. Using only text prompts from VLMs to discover rank axis is also an interesting idea.\n\nWeaknesses:\n1/ Limited attributes and domain diversity: attributes are face-related (age) or low-level (yaw, pitch, roll, aesthetics), so generalizability is still questionable. It would be better to also study this on more difficult attributes (could be as a qualitative study only) such as color, size, or semantic attributes such as “luxurious”. To generate datasets for more diverse attributes, one way is to prompt state-of-the-art text-to-image generators to generate object images with increasing order of attributes.\n\n2/ L187, the authors state one hypothesis on rankabiliy could be dependent on attribute-wise variety in the training data, but did not provide more details. This is interesting and is also worth studying in the paper, at least for the attributes with low rankability such as yaw and roll. In addition, is there any insight into why DINO performs so much better on yaw and roll than CLIP-based models? What do you think will be the attributes that CLIP-based model performs much better than DINO?\n\n3/ Applicability: one potential application of the proposed approach is fast ranking in image database with respect to an attribute, but the authors did not discuss anything else. Apart from that, is there any other application? If we learn that a particular visual embedding space has high rankability, is that a good thing? Should we train visual encoders to have good rankability? If model A has better rankability than model B in terms of attribute X, would A perform better than B on classification task of attribute X? Is it necessary to study rankability?\n\nQUESTIONS:\nQuestions presented in weaknesses section above, which are mostly around:\n- Expand attribute diversity.\n- Downstream applications of rankability. Is rankable a good thing?\n- Should we train visual encoders to have good rankability in its embedding space? How can we train for it?\n\nLIMITATIONS:\n- Yes, the authors did discuss the limitation of the work.\n- The authors state that their work does not have societal impacts. However, I believe it does to some extent. Discovering rankability of embedding space, as well as proposing zero-shot / few-shot to discover rank axis can make it become a lot easier to develop ranking feature for image database where we can rank images in terms of attributes that discriminate against certain groups of people.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an intriguing experimental study that delves into the rankability of visual embeddings in great detail. It begins with concise mathematical definitions, followed by extensive experiments conducted from multiple perspectives: the input side (using different datasets), the output side (considering various ranking attributes), and the research subjects (different model types and specific models). The study finds that, under the paper's settings, visual features often exhibit good linear rankability, and CLIP-based models generally outperform non-CLIP models. Additionally, the paper explores efficient methods for obtaining projection directions under different settings.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. **Clarity of Writing:** The paper is written with a highly logical and smooth flow, making it very accessible to readers. The conclusions are presented clearly and straightforwardly.​  \n2. **Comprehensive Experiments:** The paper conducts cross-research on multiple datasets, a wide range of attributes, and various models. The experimental results are thus relatively reliable.​ \n3. **Rigorousness in Writing:** The authors clearly define the applicable boundaries of the experimental conclusions in each section and analyze the results meticulously and rigorously.\n\n## Weaknesses​\n1. **Theoretical Analysis:** Although the paper provides extensive experimental research and analysis, it would be beneficial if the authors could offer some theoretical explanations. For instance, why do these visual embeddings possess good linear rankability? And why do CLIP models typically perform better in this regard? Such analyses would help summarize more universal and generalizable fundamental principles.​\n2. **Experimental Results:**\n    -  **A. Exceptional Cases**:​ In Table 2, we observe that in some scenarios, such as Crowd Count (ST-A) and Crowd Count (ST-B), the finetuned results are even worse than those of the Nonlinear method. Does this indicate potential optimization difficulties for these two methods? Or are there other underlying reasons?​\n    - **B. Conclusion Summarization**: The authors mainly summarize the differences in rankability between different model types (CLIP and non-CLIP). It would be valuable if they could further explore and summarize how different dataset types, different attribute types, and even the loss functions used in model training affect rankability.​\n3. **Method Design:** In Section 4, the paper analyzes methods for efficiently obtaining projection directions. However, in Figure 2, only datasets with good rankability performance are presented. We are curious whether the proposed efficient methods for obtaining projection directions remain effective on more challenging datasets, such as the Kinect dataset shown in Table 2.\n\nQUESTIONS:\n1. Regarding Weakness 1, we strongly recommend that the authors add relevant theoretical analysis to enhance the theoretical depth of the paper.​\n2. Regarding Weakness 2.A, an in-depth exploration of the reasons for the poor performance of the Finetuned method compared to the Nonlinear method in specific scenarios is expected.​\n3. Regarding Weakness 2.B, a more comprehensive summary of the impact of different factors on rankability is desired.​\n4. Regarding Weakness 3, experiments on challenging datasets to verify the effectiveness of the method for obtaining projection directions are suggested.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}