On the rankability of visual embeddings
Authors: Ankit Sonthalia
Arnas Uselis
Seong Joon Oh
Venue: NeurIPS 2025 poster ()
Keywords: ranking
visual representations
structured representations
ordinal regression
Primary area: deep_learning

Abstract: We study whether visual embedding models capture continuous, ordinal attributes along linear directions, which we term _rank axes_. We define a model as _rankable_ for an attribute if projecting embeddings onto such an axis preserves the attribute's order. Across 7 popular encoders and 9 datasets with attributes like age, crowd count, head pose, aesthetics, and recency, we find that many embeddings are inherently rankable. Surprisingly, a small number of samples, or even just two extreme examples, often suffice to recover meaningful rank axes, without full-scale supervision. These findings open up new use cases for image ranking in vector databases and motivate further study into the structure and learning of rankable embeddings.

On the rankability of visual embeddings
Ankit Sonthalia
Arnas Uselis University of Tübingen arnas wewellissutittteingeee
University of Tübingen
ankit. centtt.sohhhlliaieo iantthaali@@uiii -tuebingenncnc. de
Scong Joon Oh University of Tübingen
Abstract
We study whether visual embedding models capture continuous, ordinal attributes along linear directions, which term rank axes. We define model as rankable for an attribute projecting embeddings onto such an axis preserves the attribute's s order. Across 7 popular encoders and datasets with attributes like age, crowd count, head pose, aesthetics, and recency, we find that many embeddings are inherently rankable. Surprisingly, small number of samples, extreme examples, often suffice to recover meaningful rank axes, without full-seale supervision. These findings open up new use cases for image ranking in vector databases and motivate further study into the structure and learning of rankable embeddings. Our code is available at https//gghhb..cmmkksonnhaaaaaaa vision embeddinb
even just two
Introduction
Visual embeddings are widely used for image retrieval. This relies on embeddings forming metric space, where similar images are placed nearby. Modern visual encoders generally satisfy this property, and many systems depend on it in the form of vector databases.
Ranking is another core operation in databases. allows users to navigate large collections by sorting, paginating, and filtering results. For instance, e-commerce platforms like Amazon benefit from ranking product images by visual quality or certain product-specific attributes (e.g., ranking shoes by how formal they look).
In this work, we ask: are visual embeddings also rankable? Retrieval relics local similar- ity around a query. Ranking requires a global or- dering along attribute. Prior work has largely addressed the former. The global rankability of embeddings remains underexplored.
Local similaity
Rank axis
Query
define rankability as follows: given an em- bedding function f and continuous attribute A (e.g., "age"), we say f is rankable with respect to A if there exists rank axis V such that the projection v UJ fx) preserves the correct order the target attribute A(x) over a dataset. For instance, if A denotes "age". this projection would sort face images from youngest oldest.
Retrieval
Ranking
We examine two questions: (1) Are visual embeddings rankable? (2) How easily can we recover the rank axis for given attribute?
39th Conference Neural Information Processing Systems (NeurIPS 2025).

---

To address (I), we evaluate 7 modern visual encoders, from ResNet to CLIP, across 9 datasets with 7 attributes: age, crowd count, 3 head pose angles (pitch, roll. yaw), image aesthetics, and recency. We find that many embedding spaces are indeed rankable (Section
To address (2), estimate the rank axis V with minimal supervision. The structure of the embedding space makes full-dataset regression unnecessary. In many cases, handful of annotated samples and, in some cases, a pair of samples 2 (low) and 2h (high) already recover non-trivial ranking performance. For the latter case, we define the rank axis as V = (f(z) (=1))/llf(z>) This opens up the possibility for fast ordering of new images by arbitrary attributes. For example, photo app lets users sort selfies by age appearance. It uses CLIP embeddings and two reference images: one of a child and one of an elderly person. The app computes Vage without training. Users scroll from youngest-looking to oldest-looking faces in their album (Section D.
f(z)]]2
Contributions:
1. We define and motivate rankability as a property of visual embeddings, distinct from retrieval.
2. We study rankability across modern encoders and real-valued attributes; results show that current embeddings are rankable.
3. We show that rank axes can sometimes be recovered using only two or a handful of labelled samples.
2 Related work
Embeddings for retrieval. Visual encoders are commonly used to index images in vector databases, enabling nearest neighbour search for retrieval tasks (42 52
33 57]. This setup, known as deep metric
learning 6 42). predates vision-language models like CLIP (2). CLIP and related models shifted
the focus to modal similarity modelling, where vision and language share joint embedding
space used for classification [52], retrieval []. and captioning [39 26]. While the majority of
work in visual encoders is devoted to the understanding of the local similarity structure, we study how visual embeddings support global ranking instead of just local retrieval.
Improving embedding geometry and structure. Prior work has explored ways to improve the geometry the embedding space. Order embeddings and hyperbolic representations have been used to model hierarchies [65] 21 8 [1]. Training disentangled representation (71) is considered critical for compositionality. where attributes are assigned certain linear subspaces [60]3). Others have defined concepts like uniformity and separability of the representations [70). In this work, we focus on the analysis wide range of visual encoders, rather than introducing recipes for improvements.
Analysing embedding geometry and structure. A large body of work has examined the geometry and structure of CLIP's learned embedding space. CLIP and its derivatives have been studied ex- tensively [4] 533 32 (7). Several works have reported modality gaps between vision and language
embeddings 12 Some studies point to the absence of certain structures and capabilities in CLIP or the association of attributes to corresponding instances. Others argue that much information is already present in CLIP representations, including -frts-of-speech and linguistic structure (44), attribute-object bindings (24), and compositional at-
representations: attribute-object bindings 31 80 25), (5),
tributes [62] 63). The platonic representation hypothesis further suggests that models converge to similar internal structures (19]. In this work, we analyse the embedding geometry and structure for modern visual embeddings from the novel perspective of rankability.
Linearly probing an embedding. Linear probing is a fast and widely used method to test for the presence of concepts in visual embeddings (23) 18 64). It measures the accuracy of linear classifier trained on intermediate-layy
features, effectively testing whether hyperplane can separate embeddings containing a concept from those that do not. This technique has been used to study the geometry of CLIP's embeddings (30) and to probe for specific information such as attribute-object bindings and compositionality [24] 62). While effective for binary separability, linear probes are limited in capturing non-binary, ordinal, or relational structures (I). Prior work has not directly analysed how continuous attributes are laid out in the embedding space. Our work extends this line of research by moving beyond concept detection to characterise the internal structure of embeddings along ordinal axes, introducing rankability as a new property not captured by prior probing methods.

---

Ordinal information in embeddings. Early works on relative and ordinal attributes explored how
continuous visual attributes could be inferred and ranked [50 36 84 [4 76.. However, these efforts
were limited to smaller models and datasets. Recent studies have begun to examine ordinal signals in
large pre-trained models. These include aligning CLIPP with ordinal supervision [69], and applying
CLIP or general VL.Ms specific tasks such as aesthetics [74] 67], object counting [48 20)], crowd
counting (35). and difference detection [56 22] Several works have adapted CLIP for ranking through
prompt tuning (34), adapter-based methods [79] or regression-based fine-tuning I0, 68 Despite
these efforts, most focus on lask-specific performance or modifying the embedding space, rather than understanding its internal ordinal structure in the embeddings themselves. Our work fills this gap by systematically analysing the rankable structure of existing visual encoders and revealing the presence of ordinal directions in their embedding spaces.
3 Are vision embeddings rankable?
We set out to answer the question. For this, we first formally define rankability and strategies to measure it (Section 3.T). We introduce the models and data used for our experiments in Section 3.2 We present results in Section 3.3
Notation. Our experiments use RGB image datasets with real-valued attribute labels. We denote an image dataset as X C Rx3 HxW and define an ordinal attribute A as function A X Y C R where the range of possible labels and A(x) is the ground-truth label for given image æ. An image encoder is function f X + Rd where is the dimensionality of the embedding space. We occasionally use the term "representation" to refer to an image encoder.
3.1 Rankability
We aim characterize the linear structure of the ordinal information present in visual embedding spaces. Our definition of rankability then naturally emerges as follows.
Definition 1 (Rankability). A representation f X + Rd rankable for an ordinal attribute A over an image dataset X if there exists rank axis UA E Sa-1 d 1 dimensional unit sphere, such that for any Iı, Z2 E X with A(œ1) 2 A(z2), it follows that v f(1)) 2 f22))
The above definition requires a rank axis va to exactly preserve the ordering provided by the attribute A over the dataset X. In practice, we measure rankability using the generalisation performance of the rank axis U learned training split Xurain and tested on disjoint split X In this section, we obtain VA via linear regression on the labelled samples: {f(.). a.) where E IR is the ground-truth continuous attribute label for each x,, In Section 4 we consider approaches that do not require access to the attribute labels a..
Spearman's 's rank correlation coefficient (SRCC), denoted p serves our primary metric quantifying the monotonicity of the relationship between the true ordinality of the attribute A and the predicted ranking along VA. SRCC is widely used in related contexts (78).
We provide three reference points for the obtained rankability to contextualise the SRCC values:
(1) No-train ("lower bound"). Even for untrained visual encoders, the embeddings do not result in null rank correlation (p=0). In order to correctly capture the no-information baseline, we consider the performances of randomly initialized versions of each encoder considered [61]. The optimal rank axis V in this space serves as lower bound for the rankability of the pretrained encoder.
(2) Nonlinear (upper bound for embedding). To estimate the total ordinal information in the given embedding, we use two-layer multilayer perceptron (MLP), known to be universal approximator [16]. Comparing against a non-linear regressor lets us estimate the proportion of ordinal information in embedding that can be extracted linearly with a rank axis.
(3) Finetuning (upper bound for encoder architecture). To measure a broader upper bound indicating the capacity of the encoder architecture and the learnability of the attribute, we finetune the encoder. This conceptually envelops the nonlinear regression upper bound.
3

---

3.2 Experimental details
We provide further details on the list of attributes, datasets, encoder architectures, and the model selection protocol.
3.2.1 Attributes and datasets
In total, we use datasets, covering 7 attributes. We provide detailed breakdown in Table !
Table I: Datasets and attributes. Summary of datasets used for evaluating the rankability of visual representations.
Attribute
Dataset
#Train-val
#Test
Label Range
Split
UTKFace
83
13,146
3,287
21-60
From 27
Age
Adience
~44
44k
age groups
Official S-fold
UCT-QNRE
1,201
334
49-12.865
Official
Crowd count
ShanghaiTech-
82
300
182
33 3,139
Official
Shanghai
82
400
316
9-578
Official
Pitch
BIWI Kinect 113
10,493
4,531
+60º
6 seqs.
Yaw
BIWI Kinect
10,493
4,531
175
seqs.
Roll
BIWI Kinect
10,493
4,531
1445
test seqs.
AVA 41
230,686
4,692
Ratings (1-10)
From [78
Aesthetics
KonIQ-TOk 17
8,058
2,015
Ratings (001
Official
Image modernness
Historical Color Images 49
1,060
265
decades
From 78
3.2.2 Architectures
We test representative image-only and CLIP-based encoders. Among image-only encoders, the ResNet50 (15). ViT-B/32@ 224px [9], and ConvNeXtv2-L (73) architectures. Likewise, among CLIP encoders, we test the ResNet50, ViT-B/32, and ConvNeXt-Lee 320px variants. We also test DINOv2 [46] (ViT-B/14 variant). See Table for more information.
Table Summary of vision encoders used in our study. "Acronym" refers to how each model is denoted in our main results tables.
Acronym
Architecture
#Dims
Year
#Params
Type
Input Size
RN50
ResNet50
2048
2015
25.6M
ConvNet
224x224
VITB32
ViT-B/32
768
2020
88.2M
Transformer
224x224
CNX
ConvNeXrV2
1536
2023
198M
ConvNet
224x224
DINO-B14
DINOv2 (ViT-B/14)
768
2023
86.6M
Transformer
518x518
CLIP-RN50
CLIP ResNet50
1024
2021
38.3M
ConvNet
224x224
VIP-VITB32
CLIP ViT-B/32
512
2021
87.8M
Transformer
224x224
CLIP-CNX
CLIP ConvNeXtV2
768
2023
199.8M
ConvNet
320x320
3.2.3 Hyperparameter tuning and model selection
Hyperparameters for reporting final performances on the test set are selected based on the best validation SRCC, using either official validation splits holding them out from the corresponding training splits.
Linear and nonlinear regression. We test 30 random hyperparameter configurations per dataset- model pair. The initial learning rate is sampled from a log-uniform distribution over [10-6, 10-1] and decayed over cosine schedule to zero, while the weight decay is sampled from log- uniuurmform distribution over [10-7, 10-41, *1. Data augmentation (horizontal flipping) is also toggled on or off randomly for given run. We use 100 epochs and batch size of 128 throughout. For nonlinear regression, we use 2-layer MLP with 128 hidden dimensions and ReLU non- non-linearity.

---

Finetuning, For image-only ResNet50 and ConvNeXt encoders, we conduct grid search over two en- coder learning rates (10 10 and weight-decay rates (0, 10 For ViT-B/32, DINOv2 and CLIP models, we conduct larger grid search over three encoder learning rates (10-? 10-6 10-9) and four weight-decay values (0, 10-6, 10-1 10-3). The downstream model always uses learning rate of 0.1. We use 20 epochs for all runs. Batch size is fixed at 128, except for ConvNeXt-v2,, DINOv2 and CLIP-ConvNeXt--2, where batch size reduced to 16 because of memory constraints. We use horizontal-flij augmentation in all finetuning runs to mitigate overfitting.
3.2.4 Compute resources
All experiments were conducted on a single NVIDIA A100 GPU with 40GB of memory. All primary experiments using frozen embeddings completed within 1-2 minutes, we cache the embeddings. Finetuning experiments took between and hours each, depending on the dataset. In total, our experiments amounted to approximately 150 compute-daye We used the "stuned" Python library [55 for managing the experiments.
3.3
Results
Setup. The SRCC for the linear regressor measures the rankability of the underlying vision encoder, while the baselines (no-encoder, nonlinear regression and finetuning) provide reference points. We report our main results in Table 3 and Table 4 In Table 3 we average metrics across all architectures. Then in Table we zoom into individual architectures while retaining only the rankability metric and averaging across all datasets that contain the same attribute. We also report qualitative results in Figure emerge.
!
Our observations vary across individual attributes and architectures, but some patterns
(young) tth
(old) 100th
25th
50th
t5th
(looking down) 0th
25th
50th
th
(looking up) 100
23
26
27
28
54
-10
74
Vage
Vpitch
(not crowded) tth
(crowded)
(bad) 0th
(good)
t5th
50th
75th
100
25th
50th
75th
100th
129
233
751
944
1408
25
58
78
76
Verowd count
Vaesthetics
Figure Visualisation of rank axes. We show th percentile samples along the rank axes found using linear regression over ELIP-VIT-B/3 embeddings from each respective dataset.
Average rankability of vision embeddings is non-trivially high. We observe in Table 3 that with respect to age on the Adience dataset, the average rankability of 0.861 much higher than the no-train lower bound of 0.266, while the nonlinear and finetuned upper bounds (0.878 and 0.910) are only slightly higher in comparison. fact, for all attributes except for yaw and roll, the average rankability is closer to the two upper bounds than to the lower bound.
Linear regression often at par with dedicated SOTA efforts. Comparison with SOTA methods is outside of our main scope. We nevertheless find it interesting that often, linear regression over pretrained unmodified embeddings performs at par with dedicated SOTA efforts that require changing the underlying encoder embeddings, applying complex downstream models specialised loss functions for the ranking task. For instance, as noted in Appendix E on age estimation over UTKFace, linear regression achieves MAE of 4.25, only slightly underpertorming (78 (MAE 3.83) and at par with 27 (MAE= 4.23). Our results suggest that the rankability of visual embeddings provides a
5

---

Table Vision embeddings are generally rankable. Spearman rank correlation p between the true ranking of data samples and the predicted ranks. Higher is better. Results are averaged across all 7 architectures. No-train: Find d on the embeddings of a randomly-initialised encoder. Rankability: How well does VA encode ordinal information? Nonlinear: Maximal non-linear ordinal information contained in embeddings. Finetuned: How learnable the target attribute? For more information,
see Section 3.1
Attribute (Dataset)
No-train lower bound
Rankability main
Nonlinear upper bound
Finetuned upper bound
Age (UTKFace)
0.199
0.766
0.776
0.799
Age (Adienes)
0.266
0.861
0.878
0.910
Crowd (UCF-QNRE)
0.220
0.843
0.854
0.886
Crowd Count (ST-A)
0.120
0.734
0.749
0.689
Crowd Count (ST-B)
0.135
0.869
0.887
0.840
Pitch (Kinect)
0.405
0,803
0,811
0.975
Yaw (Kineet)
0.078
0.434
0.597
0.967
Roll (Kinect)
0.151
0.218
0.326
0.859
Aesthetics (AVA)
0.156
0.653
0.692
0.693
Aesthetics (KealQ-10) 10k)
0.435
0.761
0.793
0.901
Recency (TCI)
0.324
0.680
0.688
0.722
Table Rankability across datasets and models. Spearman's rank correlation p between the true ranking of data samples and the predicted ranks. Higher is better. See Table 2 for model details.
Model
DINO- B14
CLIP- RN50
CLIP- VITB32
CLIP-
Attribute (Dataset)
RN50
VITB32
CNX
CNX
Age (UTKFFee)
0.633
0.739
0.772
0.770
0.820
0.810
0.820
Age (Adience)
0. 723
(l.288
0.871
0.853
0.898
0.924
0.928
Crowd (UCF-QNRF)
0,864
0.837
0.810
0.788
0,870
0.870
0.860
Crowd (ST-A)
0.799
0.700
0.695
0.653
0.760
0.750
0.780
Crowd (ST-B)
0.879
0.878
0.867
0.821
0.890
0.860
0.890
Pitch (Kineet)
0.663
0.673
0.909
0.716
0.860
0.920
0.880
Yaw (Kineet)
0.624
0.305
0.384
0.804
0.120
0.360
0-440
Roll (Kineet)
0.352
0.196
0.298
0.512
0.090
0.020
0.060
Aestheties (AVA)
0,589
0.609
0.644
0.566
0.700
0.710
0.750
Aesthetics (KontQ-10s)
0. 0.739
0.713
0.744
0.681
0.800
0.790
0.860
Recency (HCI)
0.600
0.592
0.631
0.571
0.780
0.770
0.820
strong, simple baseline that should considered before applying such dedicated regression efforts. See Appendix E for detailed SOTA comparisons.
rankable than non-CLIP embeddings. We observe in Table 4 that age, aesthetics, recency and pitch, the best CLIP encoder (e.g., CLIP-ComNNeXi with an SRCC of 0.928 on Adience) wins out against the best non- oon-CLIP encoder (e.g., vanilla ConvNeXt with an SRCC 0.871 on Adience). On crowd count, the best CLIP encoders are largely tied with the best non-CLIP encoders (e.g., CLIP-RN50 0.870 vanilla RN50 0.864 for UCF-QNRF). On yaw and roll, DINO massively outperforms CLIP encoders (e.g., 0.804 vs 0.440 for yaw). conclusion, apart from isolated but interesting exceptions, CLIP encoders generally outperform or match non-CLIP encoders.
CLIP embeddings

---

Some attributes are better ranked than others. For example, the average SRCC over the two age datasets is 0.8 (see T)ble Similar average SRCCs are observed for crowd count and pitch angle. Image aesthetics and recency are less well-ranked with average SRCCs between 0.65 and 0.75. Even within the same dataset (BIWI Kinect), yaw and roll angles with SRCCs of 0.434 and 0.218 are quite poorly ranked in comparison to pitch (0.803). We hypothesize that attribute-wiss rankabilities are directly proportional to attribute-wise variety present in the training data. See also Appendix [A]
Caveats. The current results are empirical, and our claims are based on the set of attributes considered in our study (we partially address this with wider set of attributes in Appendix C. Despite following choices established in the literature, we may sometimes not uncover the most optimal finetuned upper bounds. A broader study involving theoretical support for rankability, using more ordinal attributes and possibly stronger upper bounds would be promising direction for future work.
Takeaway from $3. In general, visual embeddings are highly rankable compared to both the lower bound and upper bound baselines, although there exist variations across attributes (e.g., most encoders struggle rank based yaw and roll angles) and encoders (e.g., CLIP emheddings are, in general, more rankable than non-CIIP embeddings).
How to (efficiently) find the rank axis?
wc showed that visual embeddings are generally rankable. However, the rank axes VA. determining the direction along which the continuous attribute A is sorted accordingly (Section 331] were Iearned using a lot of training data with continuous-attriuue annotations that are typically expensive to collect. We examine whether rank axes can also be discovered in more sample- and label-efficient manners.
In Section 3
We organise the section by first tackling an easier setting with abundant data and then more challenging settings with less data available. We begin with the setting where we have fraction of the dataset with continuous attribute labels (Section I. Then, we examine the possibility to compute the rank axes with a few "extreme" points that require no cumbersome continuous attribute annotation (Section 4.2). For further practicality, domain shifts (Section 4.3). Finally, we discuss potential strategies to compute the rank axis in a zero-shot manner with text encoders in vision-language models (Section 4.4)
examine whether the obtained rank axes resilient to
4.1 Learning using a fraction of the dataset with continuous attribute labels
In this section, we investigate the learnability of the rank axis FΛ for an attribute of interest A, when fewer samples with continuous attribute annotations are available. We report the results in Figure 2 We choose the datasets UTKEaace, Adience and AVA because of their relatively large sizes compared to the other datasets used in our experiments.
Only a fraction of training data is sufficient. For age on the Adience dataset, only ~ 1k training samples out of over 11k are sufficient for covering as much as 95% of the gap between the SRCCs of the no-train baseline and full-dataset linear regression. Similarly. for image aesthetics on AVA (CLIP- viT), only ~ 16k (CLIP-ViT) ViT) or even ~ 8k (CLIP-ConvNed data points out of ~ 230k are sufficient. is evident that learning the rank axis often requires only a fraction of the original training dataset.
4.2 Learning using a small number of extreme pairs
We consider the practical scenario where a user wishes to sort their vector database using an arbi- trary attribute A. They do not have access to continuous attribute annotations (as was assumed in Section 4.I. but can readily obtain few samples at the "extremes" of the desired rank axis. We test the effectiveness of this approach.
Setup. Given training and test splits Xiain and X respectively, we sample sets of images S and S, from the lower and upper extremes of Xuain respectively. This simulates the scenario where user may obtain such "extreme" images using a readily available source like Web search engine. Next,
7

---

1.0
1.0
1.0
SRCC
0.5
Extr
Few
Uninformation
SRCC
0.5
SRCC
0.5
0.0
95% of full perf.
0.0
0.0
2
24
2
2
23
2
2
2
2
2
2
2
2
210
2
2
#samples
#samples
#samples
(a) Adience (CLIP-ViT)
(b) UTKFaae (CLIP-VIT)
(c) AVA (CLIP-ViT) -VT))
1.0
1.0
1.0
SRCC
0.5
SRCC
0.5
SRCC
0.5
0.0
0.0
0.0
2
2
2'
2
2
2
2
2
2
2
2
2
2
21
2
2
#samples
#samples
#samples
(d) Adience (C)I
(e) UTKFaae CoPyNoXo)X
(f) AVA (CLIP- No)
Figure 2: Learning using a small number of samples with continuous labels vs extreme samples without continuous labels: extreme samples win out in the small-train-set regime. Extreme refers training using samples from the extreme ends of the ranking axis (without continuous labels), while Few refers learning on continuously labelled samples. Uninformative refers to no-train baseline while Finetuned refers to the finetuned upper bound. Dashed vertical lines labeled 95 % of full perf. indicate coverage of 95% of the gap between Uninformative and Finetuned. Full-dataset linear regression performance is denoted by marker (*)).
calculate the "lower extreme cluster" z mSan(Si) and "upper extreme cluster" Tu = (S.). Finally, the vector UA-
gives the rank axis (akin to a steering vector [4
59).
Observations. We report the results in Figure comparing with the scenario where GT continuous attribute labels are available. We observe that when the size of the training dataset is extremely small (upto 1k), the extreme pains method performs better than or at par with regular training using an equal number of continuously labelled samples. This effect can be seen most prominently in the Adience dataset where a rank axis obtained from just two extreme samples achieves an SRCC of ~ 0.75 on average, while the vanilla training case results in an average SRCC close to O. As the training dataset continues to grow size, regular training with labelled data catches up and finally surpasses the extreme-pairs method. It is nevertheless striking that extreme pairs perform so well at lower sizes of the training dataset.
Takeaway. If one has an extremely small number of samples (1k or less), obtaining extreme samples from both ends of the rank axis (with no additional labeling) may constitute a better expenditure of resources than obtaining continuous attribute labels.
4.3 Robustness of rank axis
We now consider the case where there is no access samples from the target distribution. Assuming that a rank axis was previously learned using some source distribution, how transferable is it to other distributions? We investigate this using three attributes: age, crowd count, and image aesthetics.
Setup. In our main experiments, we use two age datasets (UTKFace, Adience), three crowd count datasets (UCF-QNRF, ShanghaiTech--A and ShanghaiTech-B) and two aesthetics datasets (AVA, KonIQ-1Ok). Within each set of datasets containing the same attribute, we then test how well a rank axis learned from one dataset transfers to another, and vice versa. We employ SRCC on the target dataset as our transfcrability metric in Table 5 Further, we also report cosine similarities between each pair of rank axes in Table 6 As reference points, we also report inter-attribute observations (e.g.. transfrability from an age dataset to an aesthetics dataset).
Transfer is non-trivial and asymmetric. For example, the rank axis learned from Adience has an SRCC of 0.680 on UTKFace, which is significantly higher than the SRCCs of the same rank axis on other datasets. Given that the two datasets have different labeling systems (especially with Adience labels being much more coarse than those UTKFace), this is not immediately intuitive, or trivial,
8

---

Table 5: Rank axis transferability. Spearman rank correlation coefficients when a rank axis is
trained on dataset i (rows) and evaluated on dataset j (columns).
Evaluated
Age (UTKFaee
Age (Adience)
Crowd (UCF QNRI)
Crowd (ST-))
Crowd (STB)
Aesthetics
Aesthetics
(ANA)
(Konlg10k)
Age (UTKFuce)
+0.81
+0.55
1212
00.1
-0.15
-0.07
+0.05
Trained on
Age (Adieses)
+0.68
+0.91
-0.13
-0.10
-0.21
+0.01
-0.08
Crowd (UCF-QNRF)
+0.16
-0.39
+0.87
+0.82
+0.66
+0.01
+0.07
Crowd (ST-A)
+0.05
-0.14
+0.73
+0.75
+0.72
+0.13
+0.07
Crowd (ST-B)
+0.31
+0.17
+0.41
+0.39
+0.86
+0.11
+0.05
Aesthetics (AVA)
0.12
-0.11
+0.22
+0.00
+0.49
+0.70
+0.45
Aesthetics (Koa1Q10k)
+0.06
-0.08
+0.03
+0.18
+0.05
+0.28
+0.79
and indicates the presence of some (albeit imperfect) "age" axis in the embedding space. At the same time, transfer in the opposite direction is not necessarily equally good (albeit still non-trivial), i.e., rank axis trained on UTKFace achieves an SRCC of only 0.55 on Adience.
Rank directions are non-trivially correlated. For example, the cosine similarity between age trained on UTKFaae and Adience is 0.360. While this similarity is much lower than 1.0, still significant given the high dimensionality of the embedding space. This observation again indicates the presence of "universal" age axis which was (albeit not perfectly) captured by regressors trained on both datasets. Notably, some unexpected correlations also emerge, e.g., age (UTKFace) and aesthetics (KonIQ- 10kk) rank axes have a cosine similarity of 0.220. This suggests the presence of unintended correlations in the training/ test data.
Table Cosine similarity of rank axes. compute geometric alignment of rank axes trained on datasct (rows) and dataset j (columns).
Dataset j
Age (UTKFuoe
Age (Adience)
Crowd (UCF-QNRF)
Crowd (ST-A)
Crowd (ST-)
Aesthetics (AVA)
Aesthetics (KuntQ)0k)
Age (UTKPace)
+1.00
+0.36
+0.14
+0.08
+0.06
-0.04
+0.22
Age (Adiene)
+0.36
+1.00
+0.02
+0.04
+0.03
+0.03
+0.05
Dataset 1
Crowd (UCF-QNRF)
+0.14
+0.02
+1.00
+0.54
+0.26
+0.00
+0.21
Crowd (ST-A)
+0.08
+0.04
+0.54
+1.00
+0.31
+0.01
+0.14
Crowd (ST
+0.06
+0.03
+0.26
+0.31
+1.00
+0.08
+0.07
Aesthetics (AVA)
0.04
+0.03
+0.00
+0.01
+0.08
+1.00
+0.29
Aesthetics (KoaIQ10k)
+0.22
+0.05
+0.21
+0.11
+0.07
+0.29
+1.00
4.4 Zero-shot setting
For VLMs, language is potentially data-free approach to finding a rank axis. In principle, a text prompt could correspond to rank axis in the embedding space. The SRCC of our linear regressors trained in Scction then sets upper bound to the SRCC of any rank axis recovered via prompting. In this section, we investigate the gap between this linear regression upper bound and zero-shot prompt search.
Setup. We consider two zero-shot settings. In the single-prompt setting, the direction is defined by the embedding of a text prompt (e.g., "a picture of an old person")). In the text-difference setting, the direction is defined by the difference between the embeddings of two text prompts, each describing one extreme of the given attribute (e.g., "a picture of empty room" and "a picture of crowded room"). The latter setting is similar to the one used (66). We perform a prompt search over 500

---

prompts generated by GPT-4o [45] for the single- propepppppppt setting, and 100 prompt pairs generated similarly for the text-difference setting, We report our results in Table
Observations and takeaway. Overall, zero-shot methods suboptimal. For in- stance, even the best zero-shot result cor- responds to p 0.782 on Adience vs. the corresponding linear model at = 0.917.
Table 7: Zero-shot results.
Zero-shot One prompt
Zero-shot Difference
Linear
Attribute (Dataset)
upper bound
Age (UTSRRas)
0.577
0.600
0.817
Age (Adimao
0.670
0.782
0,917
The gap is more pronounced for some tributes (e.g..
Crowd Count (UCT-QNRP)
0.523
0.315
0.867
0.449 vs. = 0.790 for image recency). While more optimal prompts may have been overlooked dur- ing the search, this also reflects a realis- tic setting where one exhausts all intuitive prompt choices. Currently, is evident that language-based prompting lags consider- ably behind linear regression using image data, although the text-difference method improves upon vanilla prompting.
Crowd Count (ST-A)
0.242
0.763
Crowd Count (SE4)
0.590
0.535
0.880
Pitch (Kinet))
0.520
0.617
0,887
Yaw (Knect)
0.117
0.060
0.307
Roll (Kinee)
-0.070
-0.010
0.057
Aesthetics (AMA)
0.367
0.410
0.720
Aesthetics (Koatt- IIk)
0.103
0.547
0.817
Recency
0.190
0.449
0.790
Takeaway from $4. One may often efficiently discover the (almost) optimal rank axis using only a fraction of the total labelled data, or even pairs of extremes without continuous labels. Learned rank axes are quite transferable across different datasets, suggesting the presence of "universal" rank axes, although they may be somewhat spuriously entangled with the datasets used obtain them.
Conclusion and future work
In this work, we investigate visual encoders for the presence of ordinal information. Extensive experiments reveal not only that such information is present, indicated by the high Spearman rank correlation of nonlinear regressors, but also that most of the available ordinality is linearly encoded, as indicated by the small gap between the performances of linear and MLP regressors. The embedding space indeed possesses a "rankable" structure. This is unexpected and practically useful.
These findings provoke further questions. Most notably, the linearity of ordinal information suggests that one could potentially also characterise embeddings as interpretable collections of latent ordinal subspaces. This would involve discovering far bigger set of ordinal attributes: we leave this exciting direction to future work.
Broader impact statement. Our work is largely a foundational analysis into the structure of em- bedding spaces and, as such, has direct negative societal impacts. However, we highlight a few potential, indirect impacts. First, the ability to rank images containing personal or sensitive infor- mation (e.g. faces) with respect to arbitrary attributes may risk exposing certain individuals that are at the extreme ends of a spectrum (e.g. income level). Second, ordering individuals along a single attribute axis, such as gender, may reinforce existing stereotypes and offend and marginalize certain demographic groups.
Acknowledgements. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 011S 1809AA The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Ankit Sonthalia and Arnas Uselis. The authors are also grateful to Wei-Hsiang Yu, the first author of (78), for helpful insights.
References
[1] Guillaume Alain and Yoshua Bengio. "Understanding intermediate layers using linear classifier probes". In: arXiv preprint 1rXiv:1610.0164 (2016))
10

---

[2] Alberto Baldrati et al. "Conditioned and composed image retrieval combining and partially fine-tuning CLiP-based features". In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2022), pp. 4964. URL: https : /api
semanticscholar. org/CorpusId 251034454
[3]
Davide Berasi et al. "Not Only Text: Exploring Compositionality of Visual Representa- tions in Vision-Language Models" In: ArXiv abs/2503.17142 (2025). URL: https: semanticscholest ar
277244112
[4]
Ting Chen et al. "A Simple Framework for Contrastive Learning of Visual Representations" " In: Proceedings of the 37th International Conference on Machine Learning. International Conference on Machine Learning. PMLR, Nov. 21, 2020, pp. 1597-1607. URL: https
//proceedings... mlr.
wtt.ss.w.....s.ssel.eeeee wwwpppwwsssseeepeeepepp.p chep/0hennnnccc .html
[5] Wei Chen et al. "Deep Learning for Instance Retrieval: A Survey". In: IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (2021), pp. 7270-7292, URL: https://api
comanticschhl..
org/ //rrp/rpsssc Id 245837930
[6] Wei Chen et al. "Deep learning for instance retrieval: A survey". . In: IEEE Transactions on Pattern Analysis and Machine Intelligence 45.6 (2022). pp. 7270-7292.
[7]
"Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowds".
In: Haroon Idrees et al. Lecture Notes in Computer Science. Cham: Springer International Publishing, 2018, pp. 544-559,
ISBN: 978-3-030-01215- 978-3-030-01216-88 DOL: 10. 1007/ URL: https: http:::///innmn...innneeee com 10. 1007 030
978- 030 01200-1211111 8_ 33,
01216-8 8333
[8]
Karan Desai et al. "Hyperbolic Image-Text Representations", In: ArXiv abs/2304.09172 (2023).
URL: httph:t/aaaaiiiii wwtp:dff///////oo/// /2344 09172.pdf.
[9] Alexey Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image Recognition
at Scale. arXiv:2010. 11929, June 2021. DOT: 10. 48550/arXi: 2010 11929
URT. http:
11929
[10] Yao Du et al. Teach CLIP to Develop a Number Sense for Ordinal Regression. Aug. 7, 2024.
DO1: 10. 48550 / arXiv 2408 03574 arXiv: 2408 03574 [ca]
URL: http: /arxiv
org/ absa /aas///0.... 03574 Pre-published.
[11] Eran Eidinger, Roee Enbar, and Tal Hassner. "Age and Gender Estimation of Unfiltered Faces". In: IEEE Transactions on Information Forensics and Security 9.12 (Dec. 2014),
pp. 2170-2179. ISSN: 1556-6021. DOI: 10. 1109 / TIFS 2014 2359646 URL: https
/ //ieeexplore.ce ieee. www.eeeeee..reeeeeeeeeeeeeeeeeee
wttg/ddccumen//690222
[12] Abrar Fahim, Alex Murphy, and Alona Fyshe. "Its Not a Modality Gap: Characterizing and Addressing the Contrastive Gap". In: ArXiv abs/2405.1855 (2024). URL: https:
semanticscholar wtatoolrrrrrroooooooooooooo lttg/Corpussdd 270095104
[13] Gabriele Fanelli et al. "Real Time Head Pose Estimation from Consumer Depth Cameras". In: Pattern Recognition. Ed. by Rudolf Mester and Michael Felsberg, Red. by David Hutchison et al. Vol. 6835. Berlin, Heidelberg: Springer Berlin Heidelberg, 201 pp. 101-110. ISBN:
978-3-642-231222 978-3-642-231123 DOL: 10. 1007 9988- 0--64-2222 23123- 0__000__11 URL.:
http://lime wwinnk..priinrrrr. com/ 10. 1007 00007978-000 3- 642 23123-
[14] Hu Han et al. "Heterogeneous face attribute estimation: deep multi-task learning approach". In: IEEE transactions on pattern analysis and machine intelligence 40.11 (2017), pp. 2597-
2609.
[15] Kaiming He al. "Deep Residual Learning for Image Recognition". In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, June 2016. pp. 770-778. -36777778885111.1 DoI: 10. 1109 CVPR. 2016. 90 URL: httt:://ieexplae
ISBN:
ieee. Sheeeorgg/dccumet///8a0
[16] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. "Multilayer feedforward networks are universal approximators" In: Neural networks 2.5 (1989), pp. 359-366
[17] Vlad Hosu et al. "KonIQ-10s An Ecologically Valid Database for Deep Learning of Blind Image Quality Assessment". In: IEEE Transactions on Image Processing 29 (2020), pp. 4041- 4056. ISSN: 1057-7149, 1941-0042 DOI 1109/T11 2020 2967829 arXiv: 1910 06180
[cs] URL:
org

---

[18] Yunshi Huang et al. "Lpt+: A surprisingly strong linear probe for few-shot clip". In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 23773-23782.
[19] Minyoung Huh et al. "The platonic representation hypothesis". In: arXiv preprint arXiv:2405.07900 (2024).
[20] Ruixiang Jiang, Lingbo Liu, and Changwen Chen, "CLIP-Count: Towards Text-Guided Zero- Shot Object Counting". In: Proceedings of the 31 ACM International Conference on Multime-
dia. Oct. 26, 2023. pp. 4535-4545, Dor: 10. 1145/3581788 3611788888618889
arXiv: 2305, 2305.07304
[cs]] URL: http: / /awxxvxivvva www.ii.oog/.ii.... http//bb//33055 07304
[21]
Valentin Khrulkov et al. "Hyperbolic Image Embeddings". In: 2020 JEEE/CVF Conference
Computer Vision and Pattern Recognition ( (CVPR) (2019), pp. 6417-6427,
URL: http:
//ieeexplore. leee. org/ stamp/ httttaapp/map...//// jasp:ppp--sckaammmbberrre
[22]
Jihyung Kil et al. CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs.
July 23, 2024. DOE: 10 48550 / arXiv 2407 16837 arXiv: 2407 16837 [cs] URL:
http://arxi: org abs /407 16837, Pre-published.
[23] Been Kim et al. "Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)". In: International conference on machine learning. PMLR. 2018, pp. 2668-2677.
[24]
Darina Koishigarina, Arnas Uselis, and Seong Joon Oh. CLIP Behaves like Bag-of- Words Model Cross-modally but Not Uni-modally. Feb. 8, 2025. DOL: 10 48550 / arXiv. 2502 03566 arXiv: 2502. 03566 [cs] published.
URL:
http: // arxiv org/ / abs / 2502 03566
Pre-
[25]
Darina Koishigarina, Arna Uselis, and Seong Joon Oh. "CLIP Behaves like Bag-of- Words Model Cross-modally but not Uni-modally". In: arXiv preprint tcXiv:2502.03560 (2025). Chia-Wen Kuo and Z. Kira. "Beyond a Pre- ree-Trined Object Detector: Cross-Modal Textual and Visual Context for Image Captioning". In: 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) (2022), pp. 17948-17958.
[26]
URL: https : api
semanticscholar org/ Corpus Id: 248572429
[27]
Maksim Kuprashevich and Irina Tolstykh. MiVOLO: Multi-input Transformer for Age and Gender Estimation. Sept. 22, 2023. DOI: 10. 48550 / arXiv 2307 04616 arXiv: 2307 04616 [cs]
URL: http: /ttt::://axiivv org/ abs 2307 04616 Pre-published.
[28] Alina Kuznetsova et al. "The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale". In: International Journal of Computer
[cs], pp. 1956-1981 ISSN: 0920-5691, 1573-1405 URL.: http: / //tr///avvvv///////a//eeae/aawaa org/ abs 1811 00982 [29] Seon- Lee, Nyeong- Shin, and Chang Su Kim. "Geometric Order Learning for Rank Estimation" In: (2022).
Vision 128.7 (July 2020). Xiv: 11000992
DOI: 10. 1 s 12611263-000 020- 01316-2
[30] Meir Yossef Levi and Guy Gilboa. The Double-Ellipsoid Geometry of CLIP. Nov. 21, 2024.
DO1: 10. 48550 arXiv 2411 14517 arXiv: 2411 14517
[ce]
URL: http: : // /rxxiv
org/ htggaabs/4411. 14517 Pre-published.
[31] Martha Lewis et al. "Does clip bind concepts? probing compositionality in large image models".
In: arXiv preprint arXiv:22/2.10533 (2022).
[32] Junnan Li et al. "Align before fuse: Vision and language representation learning with mo-
mentum distillation". In: Advances in Neural Information Processing Systems. Vol. 34. 2021, pp. 9694-9705.
[33] Junnan Li et al. "BLIP: Bootstrapping Larguuage-Image Pre-training for Unified Vision- Language Understanding and Generation". In: Proceedings of the 39th International Confer- ence Machine Learning. International Conference on Machine Learning. PMLR, June 28, 2022, pp. 12888-12900. URL: http::tttts::prrceeeeiiee .ww.iiinnnnmm......mmmmmmmemmmre mlr www.pprrsssvvee. 161//122.nn...... httpp/1122nnhmmmmll11m.
[34] Wanhua Li et al. OrdinalCLIP. Learning Rank Prompts for Singuagg-Guiieed Ordinal Regres- sion. Oct 1, 2022. DOL: 10. 48550/ arXiv. 2206. 02338 arXiv: 2206. 02338 [cs]] URL:
http:///rxivv org/ abss/206..a 02338 Pre-published.
[35] Dingkang Liang et al. CrowdCLIP: Unsupervised Crowd Counting via Vision VasguagLanguage
Model. Apr. 9, 2023. DOL: 48550 48550/ arxiv 2204.30444444 04231 arXiv: 2304. 04231 [cs] URL:
http://arxiv. org/ ab/// 232222234444224131 04231
Pre-published.
12

---

[36] Lucy Liang and Kristen Grauman. "Beyond comparing image pairs: Setwise active learning for relative attributes". In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2014, pp. 208-215.
[37] Yiming Ma, Victor Sanchez, and Tanaya Guha. CLIP-EBC: CLIP Can Count Accurately through Enhanced Blockwise Classification. Version 3. Mar. 25, 2025. DOL: 10. 48550/ 38850/arXiis arviv
2403. 09281 arXiv: 2403. 09281 [cs]
URL: http: /ttp/a/aavvv///////av//aaa/aaaaa org/ /tss//////// 2403 09281
Pre-published.
[38] Loic Matthey et al. dSprites: Disentanglemmnt
testing
Sprites
dataset.
http:///iiub..oo//eeem////////// -atsattteee--tsse...... dataset/. 2017.
[39]
Ron Mokady, Amir Hertz, and Amit Bermano. "Clipcap: Clip prefix for image captioning". In: arXiv preprint 1XXv:2111.0977:: (2021).
[40] Stylianos Moschoglou et al. " "AgeDB: The First Manually Collected, In-the-Wild Age Database". " en. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Honolulu, HI, USA: IEEE, July 2017, pp. 1997-2005. ISBN: 978-1 URL: http://ieeexploxt// ieee org/
5386-0733-6. DO1: 10. 1109/CVPRM 2017 250 @ocument/8014984//
[41] N. Murray, L. Marchesotti, and F. Perronnin. "AVA: A Large-Scale Database for Aesthetic Visual Analysis". In: 2012 IEEE Conference Computer Vision and Puttern Recognition. 2012 IEBE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI: IEEE, June 2012. DOL: 10. 1109/cvpp
62479544
URL: http://ieeexplrrrr/ ieee
oxg/do /t/////mmtttt//9/99
[42] Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. "A metric learning reality check", In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16. Springer. 2020, pp. 681-699.
[43] Kartik Narayan et al. FaceXFormer: A Unified Transformer for Facial Analysis. arXiv:2403. 12960 [cs]. Mar. 2025. DOI: 10 48550 / arXiv 2403. 12960. URL.: http:
httpl/v.rrg/bbb////eeee //arxiv.org wwexiioorr//e abs 2403. 12960
[44] James Oldfield et al. "Parts of Speech- Subspaces in Vision-Language Models". In:
ArXiv abs/2305. 1405 (2023). URL: https: postps://ant semanticscholaattetttt. .rg//Crppsso
258841517
[45] OpenAl et al. GPT-4o System Card. Oct. 25, 2024. Dor: 008550//aXii1 2410.212766.6.
arxiv: 212.0.21111 [cs]
URL: http://arxiv org/ abs 2410. 21276 Pre-published.
[46] Maxime Oquab et al. DINOv2: Learning Robust Visual Features without Supervision. en.
arXiv:2304.07199 [cs]. Feb. 2024. URL:: htt::/ttp:///aa //attp org/ abs //0/2/3344..00..33
[47] Maxime Oquab et al. DINOv2: Learning Robust Visual Features without Supervision. Feb, 2.
2024. Dor 48550 arXiv 2304 07193 arXiv: 2304. 07193 [cs]
URL: http:
arxiv. org/ 2304. 07193 Pre-published.
[48]
Roni Paiss et al. Teaching CLIP to Count to Ten. Feb. 23, 2023. DOI: 10. 48550 arxiv
2302 12066 arXiv: 2302. 12066 [cs]] Pre-published.
URL: http: //arxivv.vv//////////// org/ / abs / 2302. 12066
[49] Frank Palermo, James Hays, and Alexei A. Efros. "Dating Historical Color Images". In: Computer Vision - ECCV 2012. Ed. by Andrew Fitzgibbon et al. Red. by David Hutchison
al. Vol. 7577. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 499-512. ISBN:
978-3-642-33782-4 978-3-642-33783-3 DOL: 10. 1007 /908/998-3- 3- wwt.commmmo.m. 10. 1007/978- 637---22233333--1
3_ 36 URL:
http://link.. http://iiikkkriingg... wwm.i/gerrrommmmoo
[50] Devi Parikh and Kristen Grauman. "Relative Attributes". In: 201 International Conference on
Computer Vision. 2011 IEEE International Conference Computer Vision (ICCV), Barcclona,
Spain: IEEE, Nov. 2011, pp. 503-510. ISBN: 978- 1-4577-1 102-2 978-1-4577-1101-5 978-
1-4577-1100-8. DOI: 10. / ICCV 2011. 6126281
URL: http://aseexplore. ieee
org/document.c /6162281/
[51] Zhi Qiao et al. "HYDEN: Hyperbolic Density Representations for Medical Images and Re- ports". In: International Conference on Computational Linguistics. 2024. URL: httpss//app
semanticschol
org/ (CorpusId:2119033
[52] Alec Radford et al. "Learning Transferable Visual Models From Natural Language Supervi- sion". In: International Conference on Machine Learning. 2021. URL:
pdf /21// 00020.0000000 .dff
13

---

[53] Alec Radford et al. "Learning Transferable Visual Models From Natural Language Super- vision". In: Proceedings of the 38th International Conference on Machine Learning. In- ternational Conference on Machine Learning. PMLR, July 1, 2021, pp. 8748 87668. URL.: https://prooeeddings mmr wttpa//eeeeee///aaaaaaaaa///////
[54] Nina Rimsky et al. "Steering Llama 2 via Contrastive Activation Addition". In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers). ACL. 2024. Ed. by Lun-Wei Ku, Andre Martins, and Vivek Srikumar. Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 15504-15522. DOL: URL: htll:hhtps::/ attp:///cllaaooooo/oooooo org/ 2024. acl
10. D0.18653/v1/2000 acl long. 828
long.
[55]
Alexander Rubinstein and Arnas Uselis. STAI-tuned Experiment Scheduler: Structured Ex- periment Management with Google Sheets, Weighty & Biases, and HPC Integration. https //github. com / quee/biquu/uu / stnd, GitHub repository. Utility code for reproducible and streamlined experiment management with Google Sheets integration, W&B logging, and HPC support (e.g.. Slurm). 2025.
[56]
Dylan Sam et al. Finetuming CLIP to Reason about Pairwise Differences. Sept. 15, 2024. DOI: 10. 4 arxiv 2409 09721 arXiv: 2409. 09721 [cs]
URL: http: //axxiv org/
09721 Pre-published.
[57] Konstantin Schall et al. "Improving Image Encoders for General- Nearest Neighbor Search and Classification". In: Proceedings of the 2023 ACM International Conference on Multimedia Retrieval. ICMR '23. Thessaloniki, Greece: Association for Computing Machinery, URL: https
2023, pp. 57-66, ISBN: 9798400701788 3592266
DOI: 10. 1145 /0445/555906666 3592266
//doi wtt.:orr//...11//35555330 169833322611
[58] Christoph Schuhmann et al. LATON-400M: Open Dataset of CLIP-Filtered 400 Million Image-
Text Pairs. arXiv:2111 02111111111111111 [cs]. Nov. 2021. DOI: 10. -8850/arXiv. 2111. 2111.02114 URL.:
http://arxiv..rgg http//gabb///11111
[59] Curt Tigges et al. Linear Represennaatinss of Sentiment Large Language Models. Oct. 23.
2023. DOT: 10. 48550 arXiv 2310 15154 arXiv: 2310 15154 [cs]
URL. http: /7
arxiv. org.v.rr//ee 2310. 15154 Pre-published.
[60] Matthew Trager et al. "Linear Spaces of Meanings: Compositional Structures in Vision-
Language Models". In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
(2023). pp. 15349-15358 URL: https: / api semanticscholar org Carppssi
257766294
[61] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. "Deep image prior". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2018, pp. 9446 9454.
[62] Arnas Uselis, Andrea Dittadi, and Scong Joon Oh. "BEYOND DECODABILITY: LINEAR FEATURE SPACES ENABLE VISUAL COMPOSITIONAL GENERALIZATION" In: Work- shop Spurious Correlation and Shortcut Learning: Foundations and Solutions. Mar. 6, 2025. [63] Arnas Uselis, Andrea Dittadi, and Seong Joon Oh. "Does Data Scaling Lead to Visual Compo- sitional Generalization?" In: arXiv preprint arXiv:2507.071
(2025).
[64] Arna Uselis and Seong Joon Oh. "Intermediate Layer Classifiers for OOD Generalization". In: International Conference on Learning Representations (ICLR). 2025.
[65] Ivan Vendrov et al.
of Images and Language". In: CoRR abs/ Abs//1511.0631
(2015). URI:: https://taaattttteee ftttpp:://ariiv.. org/ wwtpg//df///// 1511 06361 pdf
[66] Jianyi Wang, Kelvin C. K. Chan, and Chen Change Loy. Exploring CLIP for Assessing the Look and Feel of Images. Nov. 23, 2022. DOI: 10. /8550/ arXiv. 2207 12396 arXiv: [cs]]
2207. 12396
URL: httt:htt:://e arxivv .rg/
12396 Pre-published.
[67] Jianyi Wang, Kelvin C.K. Chan, and Chen Change Loy. "Exploring CLIP for Assessing the Look and Feel of Images". In: Proceedings of the Thirty-Seventh AAAI Conference Artificial Intelligence and Thirty- -Fitt Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium Educational Advances in Artificial Intelligence.
Vol. 37. AAAI 23/22 23. AAAI Press, Feb. 7, 2023, pp. 2555 2563,2256,320 ISBN: 978- URL: https::///ee/c wro. 1609/
1-57735-8800 880--0 DOI: 10. 1609/ aaai v3712. 25353
aaai. aaai.v37:33 www..372.2...2 25353
[68]
Rui Wang et al. "Learning- to- Rank Meets Language: Boosting Language-Driven Ordering Alignment for Ordinal Classification". In: (2023).
14

---

[69] Rui Wang et al. "Learning- to to-r meets language: Boosting language driven ordering align- ment for ordinal classification". In: Advances in Neural Information Processing Systems 36 (2023).
[70] Tongzhou Wang and Phillip Isola. "Understanding Contrastive Representation Learning through Aligmment and Uniformity on the Hypersphere". In: ArXiv abs/2005. 10242 (2020).
URL: https: / arxiv org/ pttp 2005 10242. pdf
[71] Xin Wang al. "Disentangled representation learning". In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).
[72] Ross Wightman. PyTorch Image Models. https: / github. com wtt. wwtp.//wwghttaaa//pytorrr.oo image-models 2019. DO1: 10. 0055522122oddl 44148644
[73] Sanghyun Woo et al. Xo V2: Co-designing and Scaling ConNNets with Masked Autoen-
coders. Jan. 2. 2023. DO1: 10. /8550/arXii 2301 00808 arXiv: 2301 00808
[cs] URL:
http://arxiv. org/ abs 23011 00808 Pre-published.
[74] Liwu Xu et al. "Clip brings better features to visual aesthetics learners". In: arXiv preprint
arXiv:2307. /5640 (2023).
[75] Lewei Yao et al. "Filip: Fine-graing interactive language-iman -mmag pre-training", In: arXiv (2021).
preprint arXiv:2111.077
[76] Aron Yu and Kristen Grauman. "Just noticeable differences in visual attributes". In: Proceed- ings of the IEEE International Conference on Computer Vision. 2015, pp. 2416-2424.
[77] Jiahui Yu et al. "CoCa: Contrastive Captioners are Image-Text Foundation Models". In: Transactions on Machine Learning Research (2022). URL: https / //pp///eeie....ee//emeeemeeoeee net.:iiwwwieeeeeeeeeee......eeee net/
forum?id=M_ V_____3222V_HHHH
[78] Wei-Hsiang Yu et al. "RANKING-AWARE ADAPTER FOR TEXT-DRIVEN IMAGE OR-
DERING WITH CLIP". In: ICLR (2025).
[79] Wei-Hsiang Yu et al. Ranking-Awat Adapter for Texv-Driven Image Ordering with CLIP.
Dec. 9, 2024. DOI: 10. 48550 / arXiv 2412. 06760 arXiv: 2412. . 06760 [cs] URL: http://arxil: org/ abs/ 2412. 06760, Pre-published.
[80] Mert Yuksekgonul et al. "When and why vision-language language models behave like ofgs-of-words, and what to do about it?" In: arXiv preprint 10.00/936 (2022).
[81] Kaiwen Zha et al. "Rank-N-Contrast: Learning Continuous Representations for Regression". In: NeurlPS (2023).
[82] Yingying Zhang et al. "Single-Image Crowd Counting via Multi-Column Convolutional Neural Network". In: 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR). 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV.
USA: IEEE, June 2016, pp. 589-597. ISBN: 978- 778-1-4673-8851-L. DOI: 10. 1t.11199/VVPP 2016
70, URL: httt::
leee. org /ttg//ooommnn/778043//////
[83] Zhifci Zhang, Yang Song, and Hairong Qi. Age ProgresstionRRegresiion by Conditional Adversarial Autoencoder. Mar. 28, 2017. DO1: 10 48550 arXiv 1702. 08423 arXiv: 08423 Pre-published.
1702. 08423 [cs]] URL:
[84] Daniel Zoran et al. "Learning Ordinal Relationships for Mid-Level Vision". In: Proceedings of
the IEEE International Conference on Computer Vision (ICCV). Dec. 2015.
15

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency. research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carcfully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] pprovided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our main claim is that modern visual encoders have "rankable" embedding spaces and that rankings can often be recovered using just a fraction of the original dataset. We provide empirical justification for the first part of the claim through extensive empirical validation in Section g and for the second part in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
16

---

Answer: [Yes]
Justification: We discuss the caveats of our approach wherever they apply: in particular, we discuss them the end of Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only lested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our paper is purely empirical and do not include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
17

---

Section 3.2 details the model architectures and datasets used by us (all of which are freely available online), as well as our hyperparameter searches. Using these details, is straightforward to reproduce our conclusions.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
cases, in which
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Our codebase is ready be released to the public upon acceptance of this submission. However, we do not provide code with the current submission, as our codebase is fairly large and anonymizing it thoroughly poses risk of inadvertently revealing identifying information.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
public/guide htte://gddd//eee/ee httgggdeee/eeeubbiiieeeeeeeeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
18

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
reproducible, they paper) is recommended, but including URLs to data and code is permitted.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Table details the datasets and splits. Section 3.2.2 and Section 3.2.3 describe the model architectures and hyperparameter choices, respectively.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report error bars where applicable. For instance, in Figure 2 each few-shot experiment is repeated three times with different random seeds. The plot displays the mean performance, with the standard deviation shown as shaded regions.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
19

---

For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory. time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We discuss details of compute resources (including type of GPU, memory and average walltime for different kinds of experiments) in Section 3.2.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our work complies with every section of the NeurlPS Code of Ethics. As do not perform human experiments, the section "Research involving human subjects or participants" does not apply. We well-known open-source datasets and cite the original sources. Further, our research is foundational and has no direct societal consequences.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Our work is largely general, foundational analysis into the structure of visual embedding spaces. It intends to enhance the community' understanding of visual representations, and motivate further research questions. As such, we do not see any direct societal impacts. However, we discuss potential impacts in the Conclusion.
Justification:
Guidelines:
The answer NA means that there no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
20

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We only use standard, existing datasets and pre-trained models (i.e., we do not release any new ones). Therefore, our paper does not introduce any risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use widely available datasets that are frequently used in machine learning conferences. We cite the original authors while introducing the datasets in Table
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are relcased, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssii thcode. hateee.comdaaaa com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
21

---

Answer:
[NA]
Justification:
We do not release any new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: Our work does not involve crowdsourcing or researching directly with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing or researching directly with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
22

---

Answer: [Yes]
Justification; We briefly use an LLM to help with generating zero-shot prompts for our experiment in Section 4.4 this has been documented in the same section.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy for what should or should not be described.
cc/Conferencees22225
23

---

Contents
Why are vision embeddings rankable?
25
A.1
Attribute-wise variability and label noise
25
Why does DINO outperform CLIP on yaw and roll
25
A.3 On which attributes would CLIP sigmificantly outperiorm DINO
25
Behavior of out-of-scope values
26
Results on more attributes
26
Ordinal attributes dSprites
26
Qualitative study: "luxuriousness"
26
D Further details on datasets and dataset-specific results
27
D.I Age
27
D.2 Crowd count
28
D.3 Headpose (Euler angles)
29
D..4
Aestheties (Mean Opinion Score)
30
D.5 Image recency
Comparison with existing works
31
Comparison with SOTA works
31
Is the finetuning upper bound good enough?
24

---

A
Why are vision embeddings rankable?
While theoretical analysis was outside of the scope of our study, we conduct a few further analyses to support our main claims.
A.1 Attribute-wise variability and label noise
We conjecture that rankability along given ordinal axis is directly related to variability of different attributes in the training data. The wider the attribute distribution, the stronger the optimization pressure may be for rankability to emerge.
Background. In our experiments using CLIP (obtained from timm (72), trained on LAION-400M 58D, find that rankability for pitch (SRCC 0.91) is higher compared to rankabilities for yaw (0.31) and roll (0.40). We set out to verify whether the variability of the head orientation angles in LAION-400M face images follows the same trend.
Setup. We randomly sample 100 faces LAION-400M using a face detection model [43] We use the same model to estimate yaw, pitch and roll angles of each face. We quantify the variability of each attribute in the LAION-400M dataset using the standard deviations of angles among the 100 faces.
Results and discussion. The standard deviations for pitch, roll and yaw are 21.2, 11.7 and 11.0 degrees, respectively, in LAION-400M. This concurs with the ranking of rankability among the three attributes: pitch yaw > roll. This corroborates our hypothesis to certain degree.
As for why CLIP models perform better, we conjecture that label noise present in captions may play a role. Below, we reason via an example.
When training CLIP model on image-caption pairs that include age-related descriptors, there may often exist ambiguity in the use of age terms. For instance, middle-aged individuals may sometimes be described as "young", while young individuals are described as "middle-aged". However, one rarely describes young individuals as "old". This asymmetric labeling noise may implicitly encourage the model to embed images such that the representations of "young" and "middle-aged" individuals
are closer to each other than either of them is to representations of "old" individuals, i.c.,
dist(old, young) > dist(middle-agede young).
This may, in turn, indirectly produce the ordinal structure observed our study.
A.2
Why does DINO outperform CLIP on yaw and roll?
CLIP relies on language supervision. As long as the embedding space places similar image-text pairs together and dissimilar ones apart, there is limited incentive for fine-grained understanding of the image. We hypothesize that LA1ON-400M also has far fewer captions describing yaw and roll. Intuitively, for yaw, "looking the left/right" can be ambiguous, and it is even more unnatural to describe roll using language. We verify this intuition empirically by counting the frequency of such phrases in LAION captions: out of approx. 13M captions, pitch is described in approx. 800k captions, yaw in approx. 5k captions while roll is only described in approx. 30 captions. Hence, while there is some incentive to learn pitch, there are reduced incentives for learning yaw and roll.
DINO relies on self-distillation for training, i.e., without language guidance. Additionally, the training process involves several augmentations. As such, fine grained understanding of the visual structure in the image becomes important, likely leading to enhanced coverage of attributes like yaw and roll.
A.3 On which attributes would CLIP significantly outperform DINO?
We observe that CLIP significantly outperforms DINO (7 on image-level attributes like aesthetics and recency. We hypothesize that this can again be explained on the basis of attribute variability. CLIP was pretrained on largely uncurated dataset (LAION) with very basic caption filters in place. This would have allowed varying qualities of images into CLIP's training pipeline, leading to high variation in quality. On the other hand, DINO's training dataset, LVD-142M, is much better curated, and likely contains only high quality images. Without variation in quality, DINO does not learn much about this attribute.
25

---

B Behavior of out-of-scope values
We design an experiment below for analysing extrapolation behaviour.
Setup. The UTKFace dataset split used by us (following prior work (78) contains age labels between 21 and 60. We split the dataset into two subsets: one with ages 21-50 and the other with ages 51-60. We test whether the rank axis obtained over the 21-50 split applies to the 51-60 split by computing the SRCC against the ground-truth labels on the 51-60 split. As control, we compare against another rank axis computed using the full age range 21-60.
Results and conclusion. We observe that the rank axis obtained on ages 21-50 achieves SRCC of 0.174 on the 51-60 split, compared to 0.267 when trained on the full range 21-60. Our results suggest that extrapolation is indeed harder than interpolation.
C Results on more attributes
Broadly, our study focuses on four categories of attributes:
Age progression.
Geometric understanding (head pose).
Scene complexity and counting (crowd count).
Image quality (aesthetics, modernness).
Our reliance on continuously annotated attributes naturally constrains us to this selection. However, a qualitative exploration of other attributes is also interesting; address this next.
C.1
Ordinal attributes on dSprites
dSprites (38 provides a synthetic testbed for rankability along more "difficult" attributes like color and size.
Setup. We use pretrained CLIP ViT-B/32 embeddings to analyze the rankability of attributes: hue, x-position, y-position, scale, and orientation. For hue, we consider the segment between red and yellow to avoid cyclicity of the attribute. We compare the no-train, linear, non-linear, and finetuned performances in Table A8
Table A8: dSprites results.
Attribute
No-train
Linear
Nonlinear
Finetuned
Hue
0.919
0,920
0.996
1,000
PosX
0.551
0.877
0.966
0.998
PosY
0.706
0,966
0.993
0.999
Scale
0.788
0.980
0.985
0.986
Orientation 0.151
0.716
0.908
0.982
Results and discussion. Hue is already well-represented in untrained representations (no-train, 0.919), whereas CLIP pretraining does not seem to boost ordinality (linear, 0.920). We conjecture that hue is often not a discriminatory signal in caption- based contrastive training. Other attributes are significantly better represented (e.g. 0.151 no-train - 0.716 linear for orientation). For some attributes, frozen embeddings are as rankable as finetuned ones; for example, scale ordering using rank axis (0.980) vs dedicated fine-tuning (0.986). We conclude that CLIP embeddings are inherently capable of ranking these attributes.
C.2 Qualitative study: "Iuxuriousness"
We study the rankability of CLIP with respect to "luxuriousness" using images from the "kitchen room & dining room table" class in Open Images V7 (28)
26

---

Setup. We randomly sample 20 pairs of images and ask three annotators (in this case, the authors themselves) answer which image the pair is more "luxurious". The "ground truth" luxuriousness is determined via majority vote. We measure quality of the rank axis by measuring accuracy of the prediction of the more luxurious image in each pair. The random chance baseline is 0.50 and the upper bound is given by average inter-annotator agreement. We repeat this process over independent trials with different sets of 20 pairs.
Obtaining a rank axis from visual extremes. We prompt GPT-4o (55 to generate two synthetic reference images: one depicting minimal dining room table setup, while another represents a visually luxurious version of the same scene. We compute the rank axis as the L2 L2-nommalized difference between their CLIP image embeddings.
Results, The model achieves an accuracy of 0.75 + 0.07, well above the random chance accuracy of 0.50 and at the same level as the average inter-annotator agreement of 0.73 + 0.10. This suggests that CLIP encodes linear ordinal structure to support ranking along semantic attributes like luxuriousness. As shown in the main paper, the rank axis can easily be obtained using just two extreme samples.
D Further details on datasets and dataset-specifie results
In the main paper, we present results for each dataset aggregated all models (Table in the main paper) and rankabilities (Spearman p) for each model-dataset pair (Table 3 in the main paper). While the main results convey the primary evidence towards claim (vision embeddings are rankable), we also report more detailed results on each dataset in this section. Please also refer to Table in the main paper for a condensed overview of all datasets considered.
D.1 Age
UTKFace, introduced in [83]. is dataset of face images with age labels ranging from 0 to 116. Following (78,21
smaller subset with ages ranging between 21 and 60. The dataset was downloaded from the official website (https://susangg.ginnn githuu. www.gg.ii..uuu.ree. wttpb///////////CCCa/ We report results in Table A9
Table A9: UTKeaace (Age). Spearman's rank correlation p across evaluation strategies. No-train: linear probe on untrained encoder. Rankability: linear probe on pretrained encoder. Nonlinear: MLP on frozen encoder. Finetuned: encoder head trained end-to-end. Higher better.
No-train lower bound
Rankability main
Nonlinear upper bound
Finetuned upper bound
Model
ResNet-50
0.212
0.633
0.636
0.762
ViT-B/32
0.283
0.739
0.749
0.737
ConvNeXtVV-L.
0.283
0.772
0.776
0.815
DINOv2 VIT-B/14
0.054
0.770
0.770
0.812
OpenAl CLIP ResNet-50
0.130
0.820
0.830
0.790
OpenAl CLIP VIT-B/32
0.250
0,810
0.830
0.830
Ope ConvNeXt-L (D, 320px)
0.180
0.820
0.840
0.850
Mean
0.199
0.766
0.776
0.799
Adience, introduced in is another age dataset. Unlike UTKFace, it contains coarse labels (8 age groups instead of exact ages). We use the "aligned" version of the images and five-fold cross-validation as in 78 Results can be found in Table
27

---

Table A10: Adience (Age). Spearman's 's rank correlation p across evaluation strategies.
No-train lower bound
Rankability main
Nonlinear upper bound
Finetuned upper bound
Model
ResNet-50
0.328
0.723
0.759
0.894
ViT-B/32
0.310
0.828
0.860
0.892
Conv-NeXtV2-L
0.522
0.871
0.885
0.910
DINOv2 ViT-B/14
0.120
0.853
0.877
0.914
OpenAI CLIP ResNet-50
0.070
0,898
0.914
0.894
OpenAl CLIP ViT-B/32
0.292
0.924
0.922
0.928
OpenCLIP ConvNeXt-L (D, 320px)
0.220
0.928
0.932
0.938
Mean
0.266
0.861
0.878
0.910
D.2 Crowd count
UCF-QNRF, introduced (7]. is large crowd counting dataset containing images from diverse and
parts of the world. We use the official download link at http://wwww.rrv...f.dddddaaaaeee the official train-test splits. Results can be found Table All
Table All: UCF-QNRE (Crowd Count). Spearman's rank correlation across evaluation strategies.
No-train lower bound
Rankability main
Nonlinear upper bound
Finetuned upper bound
Model
ResNet-50
0.466
0.864
0.870
0.826
ViT-B/32
0.288
0.837
0.840
0.794
ConwNNextV2-L
-0.054
0.810
0.816
0.938
DINOv2 VIT-B/14
0.219
0.788
0.842
0.961
OpenAl CLIP ResNet-50
0.240
0.870
0.880
0.820
OpenAl CLIP ViT-B/32
0.280
0.870
0.870
0.900
OpenCLIP ConvNeXt-L (D, 320px)
0.100
0.860
0.860
0.960
Mean
0.220
0.843
0.854
0.886
ShanghaiTech, introduced in is another crowd counting dataset consisting of two parts: and B. While part A was crawled from the Internet and features larger crowds in general, part B was taken from metropolitan areas of Shanghai and features much smaller crowds. We use the DropBox link available at httpp::/githhbb https://tthb..cmm//ee//wwo
DachDaaaeet and official train-test splits for both parts. Results can found Table A12 and Table A13
Table A12: ShanghaiTech--A (Crowd Count). Spearman's rank correlation across evaluation strategies.
No-train lower bound
Rankability main
Nonlinear upper bound
Finetuned upper bound
Model
ResNet-50
0.359
0.799
0.802
0.529
ViT-B/32
0.1 148
0.700
0.623
0.558
Con-NeXtV2-L
0.061
0.695
0.753
0.834
DINOv2 VIT-B/14
0.017
0.653
0.722
0.786
OpenAl CLIP ResNet-50
0.200
0.760
0.770
0.510
OpenAl CLIP VTT-B/32
0.010
0.750
0.770
0.700
OpenCLIP ConyNeXt-L (D, 320px)
0.080
0.780
0.800
0.910
Mean
0.120
0.734
0.749
0.689
28

---

Table A13: ShanghaiTech-B (Crowd Count). Spearman's rank correlation p across evaluation
strategies.
No-train lower bound
Rankability main
Nonlinear upper bound
Finetuned upper bound
Model
ResNet-50
0.280
0.879
0.906
0.672
VIT-B/32
0.225
0.878
0.889
0.710
ConVNeXXVV2-L
0.070
0.867
0.876
0.955
DINOv2 ViT-B/14
0.020
0.821
0.869
0.972
OpenAI CLIP ResNet-50
0.270
0.890
0.900
0.690
OpenAl CLIP ViT-B/32
0.040
0.860
0.860
0.900
OpenCLIP ConvNeXi-L (D, 320px)
0.040
0.890
0.910
0.980
Mean
0.535
0.869
0.887
0.840
D.3
Headpose (Euler angles)
The BIWI Kinect dataset, introduced in 11 is a collection of 24 different videos wherein the subject of the video sits about meter away from Kinect chttps://en.wikippeeeee wikipedia httppeeiiiogwwiiiiiinneeee sensor and rotates their head span the entire range of possible head-pose angles pitch (rotation about the x-axis), yaw (rotation about the y-axis) and roll (rotation about the z-axis). As there exists no official split that we know of, we randomly hold 6 sequences for testing. Results can be found in Table A14 (pitch), Table A15 (yaw) and Table A16 (roll).
Table A14: Kinect (Pitch). Spearman's ' rank correlation across evaluation strategies.
No-train lower bound
Rankability main
Nonlinear upper bound
Finetuned upper bound
Model
ResNet--50
0.359
0.663
0.615
0.973
VIT-B/32
0.401
0.673
0.505
0.951
Conv-NXXLV2LL
0.548
0.909
0.882
0.981
DINOv2 ViT-B/14
0.231
0.716
0.986
0.979
OpenAl CLIP ResNet-50
0.450
0.860
0.870
0.970
OpenAl CLIP VIT-B/32
0.400
0.920
0.940
0.980
OpeaCLIP (D, 320px)
0.450
0.880
0.880
0.990
Mean
0.405
0.803
0.811
0.975
Table A15: Kinect (Yaw). Spearman's rank correlation p across evaluation strategies.
No-train lower bound
Rankability main
Nonlinear upper bound
Finetuned upper bound
Model
ResNet-50
0.160
0.624
0.726
0.990
VIT-B/32
0.209
0.305
0.305
0.838
ConVNeXtV2-L
0.113
0.384
0.716
0.989
DINOv2 ViT-B/14
-0.046
0.804
0.871
0.994
OpenAl CLIP ResNet-50
-0.060
0.120
0.530
0.980
OpenAl CLIP V/322/32
0.160
0.360
0.330
0.990
OpenCLIP ConvNeXt-L (D, 320px)
0.010
0.440
0.700
0.990
Mean
0.078
0.434
0.597
0.967
29

---

Table A16: Kinect (Roll). Spearman's 's rank correlation p across evaluation strategies.
No-train lower bound
Rankability main
Nonlinear upper bound
Finetuned upper bound
Model
ResNet-50
0.202
0.352
0.430
0.930
ViT-B/32
0.098
0.196
0.375
0.477
Conv-NeXtV2-L
0.550
0.298
0.368
0.963
DINOv2 ViT-B/14
0.256
0.512
0.551
0.912
OpenAI CLIP ResNet-50
0.140
0.090
0.300
0,920
OpenAl CLIP ViT-B/32
-0.060
0.020
0.170
0.850
OpenCLIP ConvNeXt-L (D, 320px)
-30
0.060
0.090
0.960
Mean
0.151
0.218
0.326
0.859
D.4
Aesthetics (Mean Opinion Score)
The Aesthetics Visual Analysis (AVA) dataset, introduced in [41] is a large-scale dataset includ- ing aesthetic preference scores provided by human annotators. Each image is labeled by mul- tiple annotators, each assigning a score in the range 1-10. The mean opinion score (MOS) of the image is then computed
weighted average over the ratings where the weight of ing is provided by its frequency. We use the split provided by [78] in their official repository Results are reported in Ta-
A17
Table A17: AVA (Image Aesthetics). Spearman's 's rank correlation across evaluation strategies.
No-train lower hound
Rankability main
Nonlinear upper bound
Finetuned upper bound
Model
ResNet-50
0.237
0.589
0.628
0.672
ViT-B/32
0.157
0.609
0.666
0.672
ConwNeXXVV2LL
0.158
0.644
0.685
0.728
DINOv2 ViT-B/14
0.057
0.566
0.648
0.590
OpenAl CLIP ResNet-50
0.150
0.700
0.710
0.700
OpenAl CLIP ViT-B/32
0.200
0.710
0.730
0.700
OpenCLIP ConyNeXt-L. (D, 320px)
0.130
0.750
0.780
0.790
Mean
0.156
0.653
0.692
0.693
KonIQ-10k, introduced 17 is another aesthetics or image quality assessment (IQA) dataset that aims model naturally occurring image distortions with mean opinion scores ranging roughly between and 100. We use the official train-test splits. Results can be found in Table A18
30

---

Table A18: KonIQ-10k (Image Aesthetics). Spearman's 's rank correlation across evaluation strate-
gies.
No-train lower bound
Rankability main
Nonlinear upper bound
Finetuned upper bound
Model
ResNet-55
0.563
0.739
0.739
0.874
VIT-B/32
0.488
0.713
0.753
0.813
ConVNNXXLV2-L
0.487
0.744
0.765
0.930
DINOv2 ViT-B/14
0.324
0.681
0.753
0.948
OpenAI CLIP ResNet-50
0.400
0.800
0.840
0.900
OpenAl CLIP ViT-B/32
0.460
0.790
0.830
0.890
OpenCLIP ConvNeXI-L (D, 320px)
0.320
0.860
0.870
0.950
Mean
0.435
0.761
0.793
0.901
D.5
Image recency
Historical Color Images (HCI), introduced in 449] was designed for the task of classifying an image by the decade during which it was taken. Therein emerges a natural ordering over the decades, defining the ordinal attribute of image "modernness" "recency". We the split provided by (78) in their repository
and report
the results Table A19
Table A19; HCI (Historical Color Images). Spearman's 's rank correlation p across evaluation strate- gies.
No-train lower hound
Rankability main
Nonlinear upper bound
Finetuned upper bound
Model
ResNet-50
0.351
0.600
0.592
0.614
ViT-B/32
0.377
0.592
0.618
0.529
ConwNeXXXV2-L
0.362
0.631
0.663
0.771
DINOv2 ViT-B/14
0.131
0.571
0.601
0.748
OpenAl CLIP ResNet-50
0.320
0.780
0.770
0.760
OpenAl CLIP ViT-B/32
0.430
0.770
0.780
0.760
OpenCLIP ConvNeXt-L (D, 320px)
0.300
0.820
0.790
0.870
Mean
0.324
0.680
0.688
0.722
Comparison with existing works
Our main aim in this work to understand the rankability emerging out of the structure visual embedding spaces, and we contextualize numbers using reference metrics (lower bound provided by the no-encoder baseline and upper bound provided by nonlinear regression finetuned encoders). This section additionally provides comparisons with prior work to further support claims.
E.1
Comparison with SOTA works
Prior research has presented results from dedicated or general efforts solve the datasets considered in our study. Such works often require modification of the entire encoder or final embeddings, with dedicated loss functions or even architectural changes. This significantly limits adaptability, For each new altribute to rank dataset, user needs to employ a considerable amount of compute to train dedicated embedding.
In Table A20 we also provide comparisons with such saute-of-the-ad results to further contextualize our results. Although architectural and training dataset differences mean that this comparison is not
31

---

always fair, we emphasize the contrast in implementational simplicity between dedicated efforts and simple linear regression over pretrained embeddings that are often readily available and easy to use. Our comparisons suggest that many existing off-the-shelf encoders contain sufficient information to rank datasets according to common attributes and simple linear regression over pretrained embeddings is a strong baseline that must be considered before developing dedicated regression methods (please see next page for Table A20).
32

---

Table A20: Comparing linear / nonlinear regression against recent state-of-the-ar methods. "Linear" and "Nonlinear" use regression over CLIP-ConYNeXtd embeddings. Dashes indicate metries unreported prior work. We take the numbers for age. aesthetics and recency from [78] and crowd count from [37 Under "Downstream model", we report the components used on top of pretrained visual embeddings (CLIP or non-CLIP models); sometimes, we also report if the encoder itself was retrained. Under "Downstream data", we report the additional data used for training the method. Finally. under "Other ingredients", we also report miscellaneous extra components used by the corresponding method. All method interpretations are to the best of our knowledge.
Attribute
Dataset
Method
Spearman
MAE
Downstream model
Downstream data
Other ingredients
Age
UTKFace
eu al.
78
3.83
Cross-attn encoder, ranking heads, Images with age labels learnable text prompt tokens Regression heads
Text encoder
MiVOLO 27
4.23
Body images, face patches, labels
Feature enhancer module for fused joint representations
Linear
4.25
Linear regressor
Images with age labels
None None
Nonlincar
4.10
2-layer MI.P regressor
Images with age lahels
Age
Adience
Yu et al. [88
0.36 (0.03)
Cross-atin encoder, two ranking heads,
Images with age labels
Text encoder
learnable text prompt tokens
OrdinalCLIP
0.47 (0.06)
(Retrain image encoder for the task)
Images with age labels
Text encoder, learn "continuous" rank prototype (text) embeddings for each rank
33
Linear regressor
0.48 (0.02)
Linear
Images with age labels Images with age labels
None
Nonlincar
0.45 (0.02)
2-layer MLP regressor
None
Crowd Count
UCF-QNRF
CLIP-EBC
37
80.3
Blockwise classification module
Images with count labels Crowd images
Text encoder
CrowdCLIP
283.3
(Retrain image encoder)
Text encoder, three-stage progressive filtering during inference
Linear
246.4
Linear
Images with count labels Images with count labels
None
Nonlinear
248.0
2-layer MI.P
None
Crowd Count
ST-A
CLIP-EBC
37
52.5
Blockwise classification module
Images with count lahels Crowd images
Text encoder
CrowdCLIP
146.1
(Retrain image encoder)
Text encnder, three-stage progressive filtering during inference
Linear
167.1
Linear
Images with lahels Images with count labels
None
Nonlinear
151.7
2 layee MLP
None
Crowd Count
ST-B
CLIP-EBC
6.6
Blockwise classification module
Images with count labels Crowd images
Text encoder
CrowdCLIP
69.3
(Retrain image encoder)
Text encoder, three-stage progressive filtering during inference
Continued on page

---

Table A20 Continued from previous
Attribute
Dataset
Method
Spearman
MAE
Downstream model
Downstream data
Other ingredients
Linear Nonlinear
34.7
Linear
Images with count labels
None
29.7
2-layer MLP
Images with count labels
None
Aesthetics
AVA
Yu et al. (78
0.747
Cross-attn encoder, two ranking heads,
Images with MOS labels (1-10)
Text encoder
learnablc text prompt tokens
CLIP-IQA 67
0.415
Softmax over two similarity scores
None
Text encoder, prompt engineering, re- move position embedding
Linear
0.749
Linear 2-layer MLP
Images with MOS labels(1-10
None
Nonlinear
0.775
Images with MOS labels (1-10)
None
Aesthetics
KonIQ-10k
Yu et al. (78
0.911
Cross-attn encoder, two ranking heads,
Images with MOS labels (1-100)
Text encoder
learnable text prompt tokens
CLIP-IQA 67
0.727
Softmax over two similarity scores
None
Text encoder, prompt engineering, re-
move position embedding
Linear Nonlinear
0.860
Linear 2 Plyer MLP
Images with MOS labelss (1-100) Images with MOS labels (1-100)
None
0.870
None
34
Recency
HCI
Yu et al. 778
0.32 (0.03) Cross-attn encoder, ranking heads,
Images with decade labels
Text encoder
learnable text prompt tokens
OrdinalCLIP
34
0.67 (0.03)
(Retrain image encoder for the task)
Images with decade labels
Text encoder, learn "continuous" rank prototype (text) embeddings for each rank
Linear Nonlinear
0.64
Linear
Images with decade labels
None
0.60
2-layer MLP
Images with decade labels
None

---

E.2 Is the finetuning upper bound good enough?
While we train our finetuned (upper bound) models using regular MSE loss, there exist specialised ranking loss functions introduced in prior works that we will refer to as GOL [29) and RnC [81]. While direct comparisons training the encoders used in our main paper using the loss functions from GOL and RnC were infeasible because of resource constraints, we confirm using small ablation whether finetuning strong-enoug upper bound for our analysis. We achieve this by comparing finetuning with MSE loss against the loss functions proposed in 22 8
Table A21: Comparison with GOL 29
Linear
Nonlinear
Finetuned
GOL
7.06
6.23
4.88
4.35
Table A22: Comparison with RnC (81).
Linear [ResNet18; DINO] Nonlinear [ResNet18; DINO]
FT [ResNet18]
RnC
9.97; 8.18
9.50; 7.76
6.57
6.14
Observations and conclusions.
We present our findings in Table A2T and Table A22 We observe that standard finetuning using MSE loss downstream tasks (using the same similar models) yields performance quite close that of
29 818 On UTKFace and AgeDB [40). finetuned VGG16 and ResNet18 models achieve MAEs of 4.88 and 6.57, only modestly outperformed by [29] (4.35) and RnC [8] (6.14), respectively. For RnC, we use the ResNet1 8 model from timm while theirs was trained from scratch, however, for GOL, used the same initial weights
29
We emphasize that our goal is complementary to GOL and RnC: while they propose methods for improving rank estimation, visual embeddings. Given our primary focus on probing intrinsic structure of pretrained embeddings using architecture-agnosti tools, and that finetuning itself already reduces the gap significantly, conclude that finetuning provides a sufficiently strong upper bound for our analysis.
study aims to evaluate rankability already present in off- shelf
35


Reviews:
[{"review_id": "o17qNRFJ7f", "reviewer": "Reviewer_kX7E", "summary": "This paper presents an intriguing experimental study that delves into the rankability of visual embeddings in great detail. It begins with concise mathematical definitions, followed by extensive experiments conducted from multiple perspectives: the input side (using different datasets), the output side (considering various ranking attributes), and the research subjects (different model types and specific models). The study finds that, under the paper's settings, visual features often exhibit good linear rankability, and CLIP-based models generally outperform non-CLIP models. Additionally, the paper explores efficient methods for obtaining projection directions under different settings.", "questions": "1. Regarding Weakness 1, we strongly recommend that the authors add relevant theoretical analysis to enhance the theoretical depth of the paper.​\n2. Regarding Weakness 2.A, an in-depth exploration of the reasons for the poor performance of the Finetuned method compared to the Nonlinear method in specific scenarios is expected.​\n3. Regarding Weakness 2.B, a more comprehensive summary of the impact of different factors on rankability is desired.​\n4. Regarding Weakness 3, experiments on challenging datasets to verify the effectiveness of the method for obtaining projection directions are suggested.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. **Clarity of Writing:** The paper is written with a highly logical and smooth flow, making it very accessible to readers. The conclusions are presented clearly and straightforwardly.​  \n2. **Comprehensive Experiments:** The paper conducts cross-research on multiple datasets, a wide range of attributes, and various models. The experimental results are thus relatively reliable.​ \n3. **Rigorousness in Writing:** The authors clearly define the applicable boundaries of the experimental conclusions in each section and analyze the results meticulously and rigorously.\n\n## Weaknesses​\n1. **Theoretical Analysis:** Although the paper provides extensive experimental research and analysis, it would be beneficial if the authors could offer some theoretical explanations. For instance, why do these visual embeddings possess good linear rankability? And why do CLIP models typically perform better in this regard? Such analyses would help summarize more universal and generalizable fundamental principles.​\n2. **Experimental Results:**\n    -  **A. Exceptional Cases**:​ In Table 2, we observe that in some scenarios, such as Crowd Count (ST-A) and Crowd Count (ST-B), the finetuned results are even worse than those of the Nonlinear method. Does this indicate potential optimization difficulties for these two methods? Or are there other underlying reasons?​\n    - **B. Conclusion Summarization**: The authors mainly summarize the differences in rankability between different model types (CLIP and non-CLIP). It would be valuable if they could further explore and summarize how different dataset types, different attribute types, and even the loss functions used in model training affect rankability.​\n3. **Method Design:** In Section 4, the paper analyzes methods for efficiently obtaining projection directions. However, in Figure 2, only datasets with good rankability performance are presented. We are curious whether the proposed efficient methods for obtaining projection directions remain effective on more challenging datasets, such as the Kinect dataset shown in Table 2.", "was_revised": true, "final_justification": "The authors have addressed all our questions adequately in their rebuttal. This paper investigates a highly interesting phenomenon and offers valuable insights to the research field. Therefore, we propose upgrading the decision to **Accept**.", "rebuttal": "We thank the reviewer for their recognition of our findings and for their commendation of our experimental setup. Below, we address the remaining concerns. \n\n## W1: Theoretical explanations: why do visual embeddings possess good linear rankability, and why do CLIP models typically perform better in this regard?\n\nWhile theoretical analysis was outside of the scope of our study, we conjecture that rankability along a given ordinal axis is directly related to variability of different attributes in the training data (lines 189-193). The wider the attribute distribution, the stronger the optimization pressure may be for rankability to emerge. To confirm this, we conduct a case study using CLIP.\n\n**Background.** In our experiments using CLIP (obtained from `timm`, trained on LAION-400M), we find that rankability for pitch (SRCC \\= 0.91) is higher compared to rankabilities for yaw (0.31) and roll (0.40). We set out to verify whether the variability of the head orientation angles in LAION-400M face images follows the same trend. \n\n**Setup.** We randomly sample 100 faces in LAION-400M using a face detection model \\[1A\\]. We use the same model to estimate yaw, pitch and roll angles of each face. We quantify the variability of each attribute in the LAION-400M dataset using the standard deviations of angles among the 100 faces. \n\n**Results and discussion.** The standard deviations for pitch, roll and yaw are 21.2, 11.7 and 11.0 degrees, respectively, in LAION-400M. This concurs with the ranking of rankability among the three attributes: pitch \\> yaw \\> roll. This corroborates our hypothesis to a certain degree. \n\n\\[1A\\] Narayan, Kartik, et al. \"Facexformer: A unified transformer for facial analysis.\" *arXiv* (2024).\n\nAs for why CLIP models perform better, we conjecture that label noise present in captions may play a role. Below, we reason via an example.\n\nWhen training a CLIP model on image-caption pairs that include age-related descriptors, there may often exist ambiguity in the use of age terms. For instance, middle-aged individuals may sometimes be described as “young”, while young individuals are described as “middle-aged.” However, one rarely describes young individuals as “old.” This asymmetric labeling noise may implicitly encourage the model to embed images such that the representations of “young” and “middle-aged” individuals are closer to each other than either is to that of “old” individuals, i.e.,\n\n$$\n\\text{dist}(\\text{old}, \\text{young}) > \\text{dist}(\\text{middle-aged}, \\text{young}).\n$$\n\nThis may, in turn, indirectly produce the ordinal structure observed in our study.\n\n---\n\n## W2: Experimental results\n\nA. *Finetuned results for some datasets like ST-A and ST-B are even worse than their Nonlinear counterparts.*\n\nThis is a valid concern. In principle, finetuning should outperform regression on a frozen encoder. However, finetuning is highly sensitive to hyperparameter choices. \n\nThat said, for smaller datasets such as ST-A and ST-B, the restricted hyperparameter range may lead to overfitting during finetuning. As a sanity check, we conduct a wider hyperparameter search (four times as long as the original search) for CLIP-RN50 on ST-A and ST-B, with results presented below (model selection using validation SRCC). We observe that with a wide enough parameter search, finetuning does outperform linear and nonlinear regression, albeit still not drastically (\\~0.06 SRCC in each case). For most datasets, finetuning slightly outperforms linear and nonlinear regression by default; while the current sanity check indicates that even longer hyperparameter searches would not have altered our conclusions drastically. \n\n| Dataset | No-train | Linear | Nonlinear  | Finetuning (12 runs) | Finetuning (48 runs) |\n| :---- | :---- | :---- | :---- | :---- | :---- |\n| ST-A | 0.20 | 0.76 | 0.77 | 0.51 | 0.83 |\n| ST-B | 0.27 | 0.89 | 0.90 | 0.69 | 0.96 |\n\n---\nB. *How do different dataset types, attribute types and training loss functions affect rankability?*\n\nWe believe that rankability emerges out of the interaction between the attribute, the evaluation dataset, and the training objective.\n\n**Effect of attribute and dataset.** While we observe non-trivial rankabilities for all attributes and datasets (Table 2 in the paper), some patterns emerge. Age, crowd count and pitch angle are ranked better than image aesthetics and recency, while pitch is ranked significantly better than yaw and roll (lines 183-187). We validate our hypothesis around this observation in W1 above. In addition, we also observe that within the same attribute (age), Adience is ranked better (0.723) than UTKFace (0.623). We believe that this stems from the fact that labels are coarser in Adience with 8 age group labels instead of exact year as in UTKFace. We also invite the reviewer’s feedback on \n\n- our analysis on other attributes under Reviewer pn5U’s W1, and  \n- our evaluations on other datasets under Reviewer SNnm’s W5.\n\n**Effect of training objective.** We consider 7 pretrained vision encoders. CLIP encoders were trained using the standard InfoNCE loss. Among the remaining encoders: DINOv2 was trained using self-distillation loss, ConvNeXtv2 using self-supervised FCMAE loss, and RN50 and ViTB32 were trained using supervised classification. \n\n- We observe that CLIP is, in general, more rankable than or at par with non-CLIP encoders on all attributes except for yaw and roll where DINO performs best (lines 175-182). We observe a mixed, attribute-dependent trend among these four encoders.  \n- On age, both self-supervised models DINO and CNX perform on par with each other while outperforming the fully supervised models. On crowd count, pitch, aesthetics and image recency, CNX outperforms DINO to varying extents.   \n- The fully supervised models win out over the self-supervised models on crowd count, while underperforming on all other attributes (when the best models out of both groups are considered). \n\nIn conclusion, self-supervised training objectives seem to lead to better rankability than supervised ones, while CLIP's training outperforms both. Future studies may also consider the role of pre-training datasets.\n\n---\n## W3: Method design: do efficient methods for finding rank axes generalize to more difficult datasets like Kinect?\n\nTo check this, we conducted experiments using Kinect (pitch) on CLIP. We report results in the table below. We observe again that using just 512 samples (\\~5% of the full dataset), we already recover an SRCC of 0.9, i.e., more than 95% of the SRCC obtained upon training on the full training set.\n\n| \\#shots | Test SRCC on pitch |\n| :---- | :---- |\n| 2 | 0.357 ± 0.132 |\n| 32 | 0.510 ± 0.090 |\n| 512  | 0.900 ± 0.040 |\n| 2048 | 0.934 ± 0.007 |\n| 9924 (full dataset) | 0.941 ± 0.000 |\n---\n## Questions \n\n**Q1: Theoretical analysis**\n\nSee W1 above.\n\n**Q2: Why does Finetuned sometimes perform worse than Nonlinear?**\n\nSee W2.A above.\n\n**Q3: Impact of different factors on rankability**\n\nSee W2.B above.\n\n**Q4: Effectiveness of efficient rank-axis finding methods on more challenging datasets**\n\nSee W3 above."}, {"review_id": "ohQTL6KBJ6", "reviewer": "Reviewer_pn5U", "summary": "The paper investigates whether visual encoders (like CLIP model) inherently encode ordinal attributes along linear directions in the embedding space, an ability which the authors name as rankability. Through extensive evaluation on multiple models across multiple attribute datasets, the authors state that existing visual embedding spaces are indeed rankable, and often times the rank axis of ordinal attribute can be estimated using only 2 extreme samples (one sample possessing the least of the attribute, one sample possessing the most). In addition, the authors also propose few-shot and zero-shot learning methods to estimate this rank axis, as well as analyze how well a rank axis is transferrable across different datasets (within the same attribute). In overall, this paper shows a different research direction into understanding the structure of visual embedding space (in addition to existing work that study hierarchies, disentanglement, separability of embedding space).", "questions": "Questions presented in weaknesses section above, which are mostly around:\n- Expand attribute diversity.\n- Downstream applications of rankability. Is rankable a good thing?\n- Should we train visual encoders to have good rankability in its embedding space? How can we train for it?", "limitations": "- Yes, the authors did discuss the limitation of the work.\n- The authors state that their work does not have societal impacts. However, I believe it does to some extent. Discovering rankability of embedding space, as well as proposing zero-shot / few-shot to discover rank axis can make it become a lot easier to develop ranking feature for image database where we can rank images in terms of attributes that discriminate against certain groups of people.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- Good and clear writing.\n- Estimating rankability is a clearly defined, new problem for understanding embedding space that has not been explored.\n- Evaluation is done thoroughly across multiple models and attribute datasets, although the selected attributes are still a bit not diverse yet.\n- It’s interesting to learn that discovering rank axis may sometimes require only 2 extreme samples, or even as few as 10% of the training data. Using only text prompts from VLMs to discover rank axis is also an interesting idea.\n\nWeaknesses:\n1/ Limited attributes and domain diversity: attributes are face-related (age) or low-level (yaw, pitch, roll, aesthetics), so generalizability is still questionable. It would be better to also study this on more difficult attributes (could be as a qualitative study only) such as color, size, or semantic attributes such as “luxurious”. To generate datasets for more diverse attributes, one way is to prompt state-of-the-art text-to-image generators to generate object images with increasing order of attributes.\n\n2/ L187, the authors state one hypothesis on rankabiliy could be dependent on attribute-wise variety in the training data, but did not provide more details. This is interesting and is also worth studying in the paper, at least for the attributes with low rankability such as yaw and roll. In addition, is there any insight into why DINO performs so much better on yaw and roll than CLIP-based models? What do you think will be the attributes that CLIP-based model performs much better than DINO?\n\n3/ Applicability: one potential application of the proposed approach is fast ranking in image database with respect to an attribute, but the authors did not discuss anything else. Apart from that, is there any other application? If we learn that a particular visual embedding space has high rankability, is that a good thing? Should we train visual encoders to have good rankability? If model A has better rankability than model B in terms of attribute X, would A perform better than B on classification task of attribute X? Is it necessary to study rankability?", "was_revised": true, "final_justification": "Thanks to the authors' rebuttal which includes a lot of high quality and useful discussion into the understanding of visual embedding space, I increase the Quality score from 3 to 4, while maintaining the original Rating as 5.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and encouraging feedback. We are especially glad that the reviewer finds the framing of rankability as a property for understanding embedding spaces novel and well-motivated. We appreciate the suggestions for expanding attribute diversity and applications. We address each point below.\n\n## W1: Attribute diversity\n\n1\\. *Attributes are face-related or low-level; generalizability is questionable*.\n\nWe appreciate this observation. Broadly, our study focuses on four categories of attributes:\n\n- Age progression. \n- Geometric understanding (head pose).  \n- Scene complexity and counting (crowd count).  \n- Image quality (aesthetics, modernness).\n\nOur reliance on continuously annotated attributes naturally constrains us to this selection. However, as the reviewer notes in the following section, a qualitative exploration of other attributes is also interesting; we address this next.\n\n*2\\. Qualitative study on more difficult attributes such as color, size, and semantic attributes like \"luxurious\"*.\n\nThese experiments would indeed add meaningfully to the empirical breadth; we thank the reviewer for proposing them. Below, we present the results.\n\n- ### Ordinal attributes on dSprites: color, scale, position, orientation\n\ndSprites provides a synthetic testbed for rankability along more \"difficult\" attributes like color and size.\n\n**Setup.** We use pretrained **CLIP ViT-B/32 embeddings** to analyze the rankability of **5** attributes: **hue**, **x-position**, **y-position**, **scale**, and **orientation**. For hue, we consider the segment between red and yellow to avoid cyclicity of the attribute. We compare the no-train, linear, non-linear, and finetuned performances.\n\n| Attribute | No-train | Linear | Nonlinear | Finetuned |\n| :---- | :---- | :---- | :---- | :---- |\n| Hue | 0.919   | 0.920  |  0.996   | 1.000 |\n| PosX | 0.551 | 0.877 | 0.966 | 0.998 |\n| PosY | 0.706 | 0.966 | 0.993 | 0.999 |\n| Scale | 0.788 | 0.980 | 0.985 | 0.986 |\n| Orientation | 0.151 | 0.716 | 0.908 | 0.982 |\n\n**Results and discussion.** The table above shows the results. Hue is already well-represented in untrained representations (no-train, 0.919), whereas CLIP pretraining does not seem to boost ordinality (linear, 0.920). We conjecture that hue is often not a discriminatory signal in caption-based contrastive training. Other attributes are significantly better represented (e.g. 0.151 no-train $\\\\rightarrow$ 0.716 linear for orientation). For some, frozen embeddings are as rankable as finetuned ones; for example, scale ordering using rank axis (0.980) vs dedicated fine-tuning (0.986). CLIP embeddings are inherently capable of ranking these attributes.\n\n- ### Qualitative study: “Luxuriousness”\n\nWe study the rankability of CLIP with respect to \"luxuriousness\" using images from the `kitchen room & dining room table` class in Open Images V7. \n\n**Setup.** We randomly sample 20 pairs of images and ask three annotators to answer which image in the pair is more \"luxurious\".  The \"ground truth\" luxuriousness is determined via majority vote. We measure quality of the rank axis by measuring accuracy of the prediction of the more luxurious image in each pair. The random chance baseline is 0.50 and the upper bound is given by average inter-annotator agreement. We repeat this process over 3 independent trials with different sets of 20 pairs.\n\n**Obtaining a rank axis from visual extremes.** We prompt GPT-4o to generate two synthetic reference images: one depicting a minimal dining room table setup, while another represents a visually luxurious version of the same scene. We compute the rank axis as the L2-normalized difference between their CLIP image embeddings.\n\n**Results.** The model achieves an accuracy of 0.75 ± 0.07, well above the random chance accuracy of 0.50 and at the same level as the average inter-annotator agreement of 0.73 ± 0.10. CLIP encodes linear ordinal structure to support ranking along semantic attributes like luxuriousness. As shown in the main paper, the rank axis can easily be obtained using just two extreme samples.\n\n## W2: Impact of attribute-wise variety in the training data; why does DINO outperform CLIP on yaw and roll; on which attributes would CLIP significantly outperform DINO?\n\nWe thank the reviewer for highlighting this direction. As a preliminary test for our hypothesis, we conducted a case study using headpose angles on CLIP. \n\n- ### Case study on headpose angles with CLIP\n\nWe hypothesize that rankability w.r.t. a given attribute depends on its variability in training data (lines 189-193). We present an empirical study over head-pose attributes using CLIP and its training dataset, LAION-400M. \n\n**Background.** In our experiments using CLIP, we find that rankability for pitch (SRCC \\= 0.91) is higher compared to yaw (0.31) and roll (0.40). We verify whether the variability of the head orientation angles in LAION-400M face images follows this trend. \n\n**Setup.** We randomly sample 100 faces in LAION-400M using a face detection model \\[1A\\] and the same model to estimate yaw, pitch and roll angles of each face. We quantify variability of each attribute using standard deviations among the 100 faces. \n\n**Results.** The standard deviations for pitch, roll and yaw are 21.2, 11.7 and 11.0 degrees, respectively, in LAION-400M. This concurs with the ranking of rankabilities: pitch \\> roll \\> yaw. This corroborates our hypothesis to a certain degree. \n\n\\[1A\\] Narayan, Kartik, et al. \"Facexformer: A unified transformer for facial analysis.\" *arXiv* (2024).\n\n- ### CLIP vs DINO \n\n**DINO outperforms CLIP on yaw and roll.** CLIP relies on language supervision. As long as the embedding space places similar image-text pairs together and dissimilar ones apart, there is limited incentive for fine-grained understanding of the image. We hypothesize that LAION-400M also has far fewer captions describing yaw and roll. Intuitively, for yaw, “looking to the left/right” can be ambiguous, and it is even more unnatural to describe roll using language. We verify this intuition empirically by counting the frequency of such phrases in LAION captions: out of ~13M captions, pitch is described in \\~800k captions, yaw in \\~5k captions while roll is only described in \\~30 captions. \n\nHence, while there is some incentive to learn pitch, there are reduced incentives for learning yaw and roll.\n\nDINO relies on self-distillation for training, i.e., without language guidance. Additionally, the training process involves several augmentations. As such, fine grained understanding of the visual structure in the image becomes important, likely leading to enhanced coverage of attributes like yaw and roll. \n\n**CLIP outperforms DINO on image aesthetics and recency.** We observe that CLIP significantly outperforms DINO on image-level attributes like aesthetics and recency. We hypothesize that this can again be explained on the basis of attribute variability. CLIP was pretrained on a largely uncurated dataset (LAION) with very basic caption filters in place. This would have allowed varying qualities of images into CLIP’s training pipeline, leading to high variation in quality. On the other hand, DINO’s training dataset, LVD-142M, is much better curated, and likely contains only high quality images. Without variation in quality, DINO does not learn much about this attribute.\n\n## W3: Applicability\n\n*1\\. Is there any other application other than fast database sorting?* *Is rankability a good thing and should we train visual encoders to have it?*\n\nHigher rankability, by definition implies we may perform continuous regression without changing the underlying embedding. Current works on ordinal embeddings (\\[57, 40, 20, 10\\] among others), make dedicated efforts to refine the embedding space for ranking-awareness with respect to one attribute at a time. After applying the ranking-aware finetuning, it is often also unclear whether the same model can then continue to support other downstream tasks. \n\nIn contrast, our work studies whether *existing* vision encoders *linearly* encode *general* ranking-awareness (i.e., with respect to several attributes) even without dedicated optimisation pressures or post-training modifications. Indeed, we find that vanilla CLIP embeddings, for instance, are already ordinal with respect to many attributes studied in prior works. \n\nStrikingly, simple linear regression over frozen embeddings often performs on par with these dedicated ordinal embeddings (we report detailed SOTA comparisons in Table A12 of the supplement). \n\nWe believe that this observation in itself is surprising and practically valuable, as it eases all the ordinal regression tasks that would otherwise be handled by dedicated ordinal encoders.\n\n\\[3A\\] Yu et al. \"Ranking-aware adapter for text-driven image ordering with CLIP.\" *ICLR 2025\\.*\n\n*2\\. If model A outperforms model B on rankability over attribute X, will the trend transfer over to classifying on the basis of X?*\n\nWe expect that the existence of a rank axis should also help classification. Below, we conduct a preliminary experiment. As expected, the better regression model is also the better classification model. \n\n| Model | Regression (SRCC) | Classification (Accuracy) |\n| :---- | :---- | :---- |\n| ResNet50 | 0.63 | 0.48 |\n| CLIP-ViT-B/32 | 0.81 | 0.60 |\n\n**Q1: Expand attribute diversity.**\nSee W1 above.\n\n**Q2: Downstream applications; is rankability good?**\nSee W3 above. \n\n**Q3: Training vision encoders for rankability**\n\n*1\\. Should we train for rankability?* \n\nSee W3 above.\n\n*2\\. How can we train for rankability?*\n\nOur study finds that rankability naturally emerges in many vision encoders, especially modern ones (e.g., CLIP). However, given the benefits of rankability, future work may also explore ways to induce it explicitly, perhaps, by including pairwise comparison annotations (image 1 \\> image 2 for a given attribute), and adding a hinge loss to the pretraining objective."}, {"review_id": "AVFEqQdq9A", "reviewer": "Reviewer_nvX2", "summary": "The paper shows that image embeddings from common vision encoders can be ranked for specific attributes; e.g., for person counting or image quality. The method proposes different ways of learning such a rankable embedding by relying on linear or non-linear models to be trained on a small percentage of the dataset. Evaluation on a variety of tasks and attributes reveal that such rankable properties exist and that some attributes are easier to rank than others.", "questions": "I would be happy to increase my score following discussions on the linearity of the embedding, few-shot phrasing, and comparisons with very similar methods in the field of image quality.", "limitations": "The paper has not discussed limitations or potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n*S1. The idea of learning rankable embeddings in common vision encoders is simple and effective.*\n* This reveals that common encoders have this information and it can be learned easily with a low percentage of the training set.\n\n*S2. The paper does not require fancy components, is applicable to a variety of vision encoders and tasks.*\n* It is appreciated that the paper assess a variety of vision encoders and tasks, showing its wide applicability.\n\n**Weaknesses**\n\n*W1. The claim of linear embedding is unverified*\n* The evaluation only focuses on measuring the Spearman rank correlation coefficient. What about the Pearson linear correlation coefficient?\n* It would have been interesting to also provide qualitative or quantitative analysis on algorithmic operations in the embedding space. Does moving by the same magnitude and direction yield the same effect? (e.g., the number of people in the image always increases by N with this vector)\n\n*W2. Few-shot claims are over-stated*\n* While it is interesting to find attribute embeddings from existing models, the method still requires a thousand of examples! It seems like 5-10% of the dataset is still needed to learn the attribute embeddings. This is in contrast with the few-shot literature where few-shot usually means less than 10 examples. I would suggest to rephrase \"few-shot\", or mentions of \"a small number of samples\" to something more accurate.\n\n*W3. Behavior of out-of-scope values is unclear*\n* The method requires to provide extreme values to learn how to rank. Let say we present examples ranging from 0 to 100. What happens if at inference we present an image where the value is 200?\n\n*W4. Comparisons are missing.*\n* The evaluation only focuses on the selected vision encoders on various ranking tasks. But, what about the SOTA methods in each of these respective tasks? It would have been interesting to know what is the upper-band.\n* Furthermore, the idea of ranking attributes had been recently explored in image quality assessment, where for example related work (e.g., [55], among many others) find a linear embedding between what is a \"bad photo\" and a \"good photo\". Direct comparisons with such methods is possible and would benefit the paper.", "was_revised": true, "final_justification": "The rebuttal was help to support claims in the paper, and clarify limitations of the method.", "rebuttal": "We thank the reviewer for the careful reading and constructive questions. We hope that the additional experiments and clarifications below address the remaining concerns.\n\n## W1: The claim of linear embedding is unverified.\n\n### 1. What about Pearson linear correlation coefficient (PLCC)?\n\nWe wish to clarify two related but distinct notions of linearity in the context of ordinal attribute embeddings:\n\n- Linearity of the rank axis refers to whether there exists a consistent direction in the embedding space along which an ordinal attribute (e.g. age, brightness) increases. Our work focuses on identifying this direction. To evaluate this, we compute Spearman’s rank correlation coefficient (SRCC) between the true attribute ranks and the projections of embeddings onto the identified rank axis.\n\n- Linearity of the ordering along the rank axis concerns whether items are spaced in a regular, linear fashion along this direction. For example, if the true attribute values increase by a fixed amount, do the corresponding embedding projections also show equal spacing? This reflects the regularity or uniformity of the embedding distribution along the rank axis.\n\nWhile our primary focus is on (1), we agree that further analysis of (2) could reveal useful insights into how ordinal attributes are represented in embedding space.\n\nWe report PLCC  for the CLIP-ViT-B/32 embedding against 5 attributes below. For most attributes, the PLCC measures are close to SRCC (0.921 vs 0.932 for head pitch). This indicates that the ordering of attributes along the rank axis is largely linear. This is an interesting further insight into the structure of the embeddings. We will include the PLCC versions of Tables 2 & 3 in the paper and discuss the implications. We thank the reviewer for the additional insight.\n\n| Attribute                    | Dataset      | SRCC  | PLCC  |\n|-----------------------------|--------------|-------|-------|\n| Age                         | UTKFace      | 0.814 | 0.848 |\n| Aesthetics (Mean Opinion Score) | KonIQ-10k    | 0.793 | 0.818 |\n| Crowd count                 | UCF-QNRF     | 0.870 | 0.817 |\n| Head pitch                  | BIWI Kinect  | 0.921 | 0.932 |\n| Head yaw                    | BIWI Kinect  | 0.360 | 0.370 |\n| Head roll                   | BIWI Kinect  | 0.020 | -0.030 |\n| Image recency               | HCI          | 0.770 | 0.760 |\n\n### 2. Algorithmic operations in the embedding space: does moving by the same magnitude and direction yield the same effect?\n\nYes, to some degree. The Pearson's correlation results above show this indirectly. We show this more directly with a dedicated experiment below.\n\n**Setup**. We measure the calibration of the rank-axis projected distances of the embeddings against the ground-truth ordinal attribute. We use CLIP-ViT-B/32 embeddings on the ShanghaiTech-B and UTKFace datasets for the age and crowd count attributes, respectively. We then measure the linearity of the relationship between the ground-truth age and the projected distances using the coefficient of determination ($R^2$). \n\n**Results and conclusion**. The $R^2$ values are 0.761 for crowd count and 0.849 for age. This indicates a largely linear agreement between projected distance along the rank axis and the ground-truth attribute values.\n\n## W2: Few-shot claims are over-stated.\n\nWe thank the reviewer for pointing this out. We will revise all mentions of “few-shot” in the paper to a more accurate wording like “sub-1k supervision” or “5-10% of the dataset” to avoid confusion and false expectations. \n\n## W3: Behavior of out-of-scope values: what happens if the inference image is outside the range of training images?\n\nWe thank the reviewer again for a very interesting question. This will shed further light on the structural regularity of the embedding space.  We design an experiment below for the extrapolation behaviour.\n\n**Setup**. The UTKFace dataset contains age labels between 21 and 60. We split the dataset into two subsets: one with ages 21-50 and the other with ages 51-60. We test whether the rank axis obtained over the 21-50 split applies to the 51-60 split by computing the SRCC against the ground-truth labels on the 51-60 split. As a control, we compare against another rank axis computed by the full age range 21-60.\n\n**Results and discussion**. We observe that the rank axis obtained on ages 21-50 achieves an SRCC of 0.174 on the 51-60 set, compared to 0.267 when trained on the full range 21-60. Extrapolation is indeed harder than interpolation. It is thus recommended that users obtain extreme examples to cover the full dynamic range of attributes. \n\n## W4: Comparisons are missing.\n\n### 1. What about SOTA methods?\n\nWe already have comparisons against the state-of-the-art models in Table A12 of the supplementary material. We observe that\nOn UTKFace (age estimation), our rank axis over fixed embeddings achieves an MAE of 4.25, slightly underperforming [1] (MAE = 3.83) and performing on par with [2] (MAE = 4.23). This is remarkable given that [1, 2] rely on further modifications to the embeddings designed to enhance ordinality.  On UCF-QNRF (crowd-counting), likewise, our approach achieves an MAE of 246.4, even outperforming CrowdCLIP [3] a model designed for crowd counting.The results further support our thesis that pretrained vision embeddings are highly rankable, even compared to more complex specific systems designed for the prediction of respective attributes. We will include this discussion in the main paper.\n\n### 2. Comparisons with image quality assessment works like CLIP-IQA [55] that discuss similar ideas.\n\nWe thank the reviewer for mentioning CLIP-IQA. In Table A12 of the supplementary material, we include comparisons with CLIP-IQA. On KonIQ-10k (aesthetics), our linear rank axis achieves an SRCC of 0.860, outperforming CLIP-IQA (0.727). We emphasise that this comparison is not entirely direct: CLIP-IQA does not use supervised training data and relies solely on prompt-based scoring. However, we agree that such comparisons provide further contextualization of our work. We will include more of such quantitative comparisons in the main paper, instead of supplementary materials.\n\n## Limitations: the paper has not discussed limitations or potential negative social impact.\n\nWhile we discuss the limitations in lines 189-193, we agree that we should discuss the potential negative societal impacts of our work. We recognize a few potential issues. First, the ability to rank images containing personal or sensitive information (e.g. faces)   with respect to arbitrary attributes may risk exposing certain individuals that are at the extreme ends of a spectrum (e.g. income level). Second, ordering individuals along a single attribute axis, such as gender, may reinforce existing stereotypes and offend and marginalize certain demographic groups. This discussion will be added to the manuscript."}, {"review_id": "Xm17Rh0Mop", "reviewer": "Reviewer_SNnm", "summary": "The paper analyzes whether existing visual encoders have rankable embeddings. The authors define that an embedding space is rankable, if there exists a rank axis such that the projection onto it preserves the ordering of a continuous attribute. They claim that embeddings of modern visual encoders are highly rankable compared to lower bound metrics provided by the untrained-encoder baseline, and upper bound metrics provided by nonlinear regression and finetuned encoders. The paper also finds that a small number of extreme samples often suffice to recover meaningful rank axes.", "questions": "- For the upper-bound settings, are the MLPs and encoders trained using just a regular MAE or MSE loss?\n\n- It seems that the few-shot performance would vary significantly depending on how the instances were sampled. If random sampling was used, how many trials were run to obtain the reported results?\n\n- Attributes like age have a large number of available datasets, such as MORPH II[1], CLAP2015[2], AgeDB[3], and CACD[4]. Would it be possible to conduct experiments on several of these additional datasets to further verify the rankability and rank axis transferability across a broader range of age datasets?\n\n[1] Karl Ricanek and Tamirat Tesafaye. MORPH: A longitudinal image database of normal adult age-progression. In Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition, 2006.\n\n[2] Sergio Escalera, Junior Fabian, Pablo Pardo, Xavier Baró, Jordi Gonzàlez, Hugo J. Escalante, Dusan Misevic, Ulrich Steiner, and Isabelle Guyon. ChaLearn looking at people 2015: Apparent age and cultural event recognition datasets and results. In Proc. ICCV Workshops, 2015. \n\n[3] Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. AgeDB: The first manually collected, in-the-wild age database. In Proc. CVPR, 2017.\n\n[4] Bor-Chun Chen, Chu-Song Chen, and Winston H Hsu. Face recognition and retrieval using cross-age reference coding with cross-age celebrity dataset. IEEE Trans. Multimedia, 2015.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The proposed algorithm in this paper demonstrates a reasonable formulation for analyzing whether existing encoders capture ordinal information. \n- The paper supports its claims through extensive evaluations over a diverse range of datasets and model architectures. \n- It also covers few-shot and zero-shot learning scenarios, showing that rank axes can be effectively recovered using only a small number of extreme samples. \n\n**Weaknesses**\n- Some important details are missing, such as how the nonlinear and fine-tuned methods were trained. \n- The paper lacks sufficient references to existing literature on methods for constructing ordinal embeddings. Are the performance gaps between linear regression over trained embeddings still small when models are trained using the loss functions proposed in [1], [2]?\n- For an embedding to be practically useful in real-world ranking tasks, it must have a high degree of ordinality. Given that there are existing encoders specifically trained to produce strong ordinal embeddings, it is unclear whether analyzing or relying on the ordinality of a generic encoder serves a meaningful purpose. \n- The paper lacks in-depth analysis on why certain attributes yield lower SRCCs. It instead relies on general, unsupported hypotheses(L183-188).\n- The number of datasets per attribute was limited to a maximum of 2~3. Given the abundance of datasets available for attributes such as age and aesthetics, a more extensive evaluation across a broader range of datasets would have further substantiated the generalization capability and rank axis transferability proposed in this paper. \n\n\n[1] Seon-Ho Lee, Nyeong-Ho Shin, and Chang-Su Kim. Geometric order learning for rank estimation. In Proc. NeurIPS, 2022.\n\n[2] Kaiwen Zha, Peng Cao, Jeany Son, Yuzhe Yang, and Dina Katabi. Rank-N-Contrast: Learning continuous representations for regression. In Proc. NeurIPS, 2023.", "was_revised": true, "final_justification": "Many of my initial concerns have been addressed. Given the interesting topic and its potential for diverse applications, I have decided to raise my score.", "rebuttal": "We appreciate the reviewer’s detailed comments and insightful questions. We address the remaining concerns below.\n\n## W1: How were the nonlinear and finetuned methods trained?\n\nWe provide the training details in §3.2.3 below.\n\n**Nonlinear regression.** We use a 2-layer MLP with 128 hidden dimensions and ReLU non-linearity (we will add this part). We randomly sample 30 hyperparameter configurations from a search space defined as follows. Learning rate and weight decay are sampled log-uniformly from ranges \\[10^{-6}, 10^{-1}\\] and \\[10^{-7}, 10^{-4}\\] respectively. Horizontal flipping augmentation is likewise switched on or off randomly. The learning rate is decayed to zero over the course of training using a cosine annealing schedule. We train for 100 epochs with a batch size of 128\\. \n\n**Finetuning.** For ResNet50 and ConvNeXt, we conduct a grid search over 2 learning rates {10^{-4}, 10^{-3}}, and 2 weight decays {0, 10^{-5}}. For the other models, we conduct a wider grid search over 3 LRs {10^{-7}, 10^{-6}, 10^{-5}}, 4 weight decays {0, 10^{-5}, 10^{-4}, 10^{-3}}. The downstream learning rate is always fixed at 0.1, and we always use horizontal flip augmentations. We train all models for 20 epochs. Batch size was fixed at 128 except for ConvNeXt models and DINOv2 where it was reduced to 16\\.\n\nPlease let us know if further clarification is needed.\n\n## W2.1: Lacking references to existing literature on methods for constructing ordinal embeddings.\n\nWe discuss existing methods for constructing ordinal embeddings \\[57, 40, 20, 10\\] in §2. In essence, these works are application-driven, aiming to modify CLIP’s embeddings or the encoder itself for specific ordinal regression tasks while we study whether *general* vision encoders already possess rankability with respect to *arbitrary* attributes. We thank the reviewer for mentioning \\[1, 2\\] below. As with prior methods discussed above, \\[1,2\\] train attribute-specific encoders (VGG16, ResNet18, and so on) for solving specific ordinal regression tasks. In contrast, our work aims to understand the linear, ordinal structure *already present* in the embedding space.\n\nOther interesting future applications include:\n\n- **Zero-shot control in generative models**: Rank axes from the encoder can transfer to latent control in generation tasks, offering an interpretable way to steer outputs along semantic attributes. \n\n- **Model auditing and diagnosis**: When a model fails to predict certain continuous attributes, our method may provide an additional debugging tool for the intermediate representations to understand better why and how it fails.\n\nWe will add the discussion in §2. \n\n\\[1\\] Lee et al., Geometric order learning for rank estimation. NeurIPS, 2022\\.  \n\\[2\\] Zha et al., Rank-N-Contrast: Learning continuous representations for regression. NeurIPS, 2023\\.\n\n## W2.2: Are the performance gaps between linear regression over trained embeddings still small when models are trained using the loss functions proposed in \\[1\\], \\[2\\]?\n\nThis is a good point. While direct comparisons training the encoders used in our main paper using the loss functions from GOL and RnC were infeasible because of resource constraints, we evaluated regression over the models used in the original studies. Our interpretation of the reviewer's question is whether finetuning is a strong-enough upper bound for our analysis (please correct us if this is not the case); our goal is to evaluate this by comparing finetuning with MSE loss against the loss functions proposed in \\[1, 2\\].\n\n**Results on UTKFace using VGG16**.\n\n| Linear | Nonlinear | Finetuned | GOL |\n| :---- | :---- | :---- | :---- |\n| 7.06 | 6.23 | 4.88 | 4.35 |\n\n**Results on AgeDB**.\n\n| Linear \\[ResNet18; DINO\\] | Nonlinear \\[ResNet18; DINO\\] | Finetuned \\[ResNet18\\] | Rank-N-Contrast |\n| :---- | :---- | :---- | :---- |\n| 9.97; 8.18 | 9.50; 7.76 | 6.57 | 6.14 |\n\n**Observations and conclusions.** \n\nWe observe that standard finetuning using MSE loss on downstream tasks (using the same or similar models,) yields performance quite close to that of \\[1, 2\\]. On UTKFace and AgeDB, finetuned VGG16 and ResNet18 models achieve MAEs of 4.88 and 6.57, only modestly outperformed by \\[1\\] (4.35) and RnC \\[2\\] (6.14), respectively. For RnC \\[2\\], we use the ResNet18 model from `timm` while theirs was trained from scratch, however, for GOL \\[1\\], we used the same initial weights as \\[1\\].\n\nWe emphasize that our goal is complementary to GOL and RnC: while they propose methods for improving rank estimation, our study aims to evaluate rankability already present in off-the-shelf visual embeddings. Given our primary focus on probing intrinsic structure of pretrained embeddings using architecture-agnostic tools, and that finetuning itself already reduces the gap significantly, we conclude that finetuning provides a sufficiently strong upper bound for our analysis. We again thank the reviewer for mentioning these works, and will discuss them in our paper.\n\n## W3: does analysing ordinality of a generic encoder serve a real purpose given that dedicated ordinal encoders exist?\n\nYes, studying the presence of ordinality in off-the-shelf encoders has significant practical implications. \n\nCurrent solutions for ordinal embeddings \\[57, 40, 20, 10\\] require modification of the entire encoder or final embeddings, with dedicated loss functions or even architectural changes. This significantly limits adaptability. For each new attribute to rank a dataset, a user needs to employ a considerable amount of compute to train a dedicated embedding.   \n\nOur findings suggest that such a procedure is unnecessary. Many existing off-the-shelf encoders contain sufficient information to rank datasets according to common attributes. In fact, our approach to find a linear rank axis in frozen embeddings yields comparable performances against state-of-the-art embeddings designed to rank each attribute of interest \\[Table A12\\]. \n\nWe will make this clearer in the revision.\t\n\n## W4: Lacking in-depth analysis on why certain attributes yield low rankability; unsupported hypotheses.\n\nWe hypothesize that rankability with respect to a given attribute depends on the variability of that attribute present in the training data (lines 189-193 in the submission). We present an empirical case study over the head-pose attributes using CLIP and its training dataset, LAION-400M. \n\n**Background.** In our experiments using CLIP (obtained from `timm`, trained on LAION-400M), we find that rankability for pitch (SRCC \\= 0.91) is higher compared to rankabilities for yaw (0.31) and roll (0.40). We set out to verify whether the variability of the head orientation angles in LAION-400M face images follows the same trend. \n\n**Setup.** We randomly sample 100 faces in LAION-400M using a face detection model \\[1A\\]. We use the same model to estimate yaw, pitch and roll angles of each face. We quantify the variability of each attribute in LAION-400M dataset using the standard deviations of angles among the 100 faces. \n\n**Results and discussion.** The standard deviations for pitch, roll and yaw are 21.2, 11.7 and 11.0 degrees, respectively, in LAION-400M. This concurs with the ranking of rankability among the three attributes: pitch \\> yaw \\> roll. This corroborates our hypothesis to a certain degree. \n\n\\[1A\\] Narayan, Kartik, et al. \"Facexformer: A unified transformer for facial analysis.\" *arXiv* (2024).\n\n\n## W5: Number of datasets per attribute was limited to 2-3; Evaluation on more datasets?\n\nGiven practical limits in computational capacity, we strived to hit the right balance between the breadth of model choices (7 models), number of attributes (7 attributes), and number of datasets per attribute (2-3 datasets per attribute where available), which already surmounts to 150 GPU days of compute (§3.2.4). \n\nAt the same time, we agree that additional datasets will further support our claims. We present additional results using CLIP-ViT-B/32 on the datasets referenced by the reviewer. We will include the full results in the camera-ready version. \n\nWe confirm the finding in our work that rankability is observed for the suggested datasets too.\n\n|  | No-train | Linear | Nonlinear | Finetuning |\n| :---- | :---- | :---- | :---- | :---- |\n| AgeDB | 0.470    | 0.895   | 0.899    | 0.906 |\n| MORPH II | 0.496    | 0.920    | 0.926    | 0.954 |\n| CLAP2015 | 0.078    | 0.934    | 0.944    | 0.944 |\n| CACD | 0.148    | 0.768   | 0.771 | 0.702 |\n\nBelow we also report the **transfer results**.  For every age dataset we calculate two metrics:\n\n* **SRCC of rank axis obtained using the same dataset**. *We train on the dataset itself, then test on a held-out split from that same dataset.*  This is the first numeric column in the table.\n\n* **Mean transfer SRCC from other datasets**. We *train five separate rank-axis models, each on one of the other datasets; evaluate all five on the current dataset and take their average SRCC.*  This corresponds to the second column.\n\nFor brevity we show only this averaged “Mean transfer SRCC from other datasets” in the table; the full per-source breakdown will be included in the camera-ready version.\n\nAs in the main paper, we confirm non-trivial transfer of rank axes across age datasets. \n\n| Dataset | SRCC of rank axis obtained using the same dataset | Mean transfer SRCC from other datasets  |\n| :---- | :---- | :---- |\n| UTKFace | 0.814 | 0.712 |\n| Adience | 0.912 | 0.686 |\n| AgeDB | 0.895 | 0.706 |\n| MORPH II | 0.920 | 0.822 |\n| CLAP2015 | 0.934 | 0.820 |\n| CACD | 0.771 | 0.672 |\n\n## Questions\n\n**Q1: Were upper-bound models trained using MAE or MSE loss?**\n\nWe used the MSE loss across all settings.  \n\n**Q2: How many trials were used to report the few-shot results?**\n\nWe conducted three random trials. We plot the standard deviations in our results – the variation across trials gets smaller as we approach the “95% of full-training performance” point. \n\n**Q3: Experiments on additional datasets.**\n\nSee W5 above."}]

Author rebuttal:

Meta-review: The paper investigated if embeddings learned by current deep encoders have ordinal information. The answer turns out to be positive, and mapping between the embeddings and ordinal information is not difficult to do. This investigation is quite in depth, the task interesting and not sufficiently well explored. While reviewers initially had issues with some of the claims, they were fullyy ocnvinced by the rebuttal and all recommend acceptance, most without reservation. The AC recommends acceptance.
Decision: Accept (poster)