Geometric Order Learning for Rank Estimation
Authors: Seon-Ho Lee
Nyeong Ho Shin
Chang-Su Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: rank estimation
order learning
metric learning

Abstract: A novel approach to rank estimation, called geometric order learning (GOL), is proposed in this paper. First, we construct an embedding space, in which the direction and distance between objects represent order and metric relations between their ranks, by enforcing two geometric constraints: the order constraint compels objects to be sorted according to their ranks, while the metric constraint makes the distance between objects reflect their rank difference. Then, we perform the simple $k$ nearest neighbor ($k$-NN) search in the embedding space to estimate the rank of a test object. Moreover, to assess the quality of embedding spaces for rank estimation, we propose a metric called discriminative ratio for ranking (DRR). Extensive experiments on facial age estimation, historical color image (HCI) classification, and aesthetic score regression demonstrate that GOL constructs effective embedding spaces and thus yields excellent rank estimation performances. The source codes are available at https://github.com/seon92/GOL

Geometric Order Learning for Rank Estimation
Seon-Ho Lee
Nyeong-Ho Shin
Chang-Su Kim
School Electrical Engineering
Korea University
seonholee@mer korea ac. kr, nhshin@mcl korea kr, changaukin@korea.accc
Abstract
A novel approach to rank estimation, called geometric order learning (GOL), is proposed in this paper. First, construct embedding space, in which the di- rection and distance between objects represent order and metric relations between their ranks, by enforcing two geometrie constraints: the order constraint compels objects to be sorted according to their ranks, while the metric constraint makes the distance between objects reflect their rank difference. Then, perform the simple nearest neighbor (k-NN) search the emhedding space to estimate the rank of test object. Moreover, assess the quality of embedding spaces for rank estimation, we propose a metric called discriminative ratio for ranking (DRR). facial age estimation, historical color image (HCI) clas- sification, and aesthetic score regression demonstrate that GOL constructs effective embedding spaces and thus yields excellent rank estimation performances. The source codes are available at https: : // thub com/ com/seae /em///enn2//Gt
Extensive experiments
Introduction
In rank estimation, we estimate the rank (or ordered class) an object. is different from ordinary classification, for its classes are arranged in natural order. For example, in movie rating, classes can be ordered from 'outstanding' to 'very good,' 'satisfactory, 'unsatisfactory" and 'poor.' Rank estimation is a fundamental problem and, e.g., used for various computer vision tasks including facial age estimation (Shin al., 2022), aesthetic quality assessment (Schifanella et al., 2015), and HCI classification (Palermo et al., 2012).
For rank estimation, many techniques (Li et al., 2021; Liu al., 2018) adopt the ordinal regression framework, which employs a classifier or regressor to predict the rank of an object directly. However, they may fail to yield reliable estimates, for there no clear distinction between ranks many cases. For instance, in facial age estimation, the aging process causing variations in facial shapes, sizes, and texture has large individual differences due factors such as genes, diet, and lifestyle.
To address this issue, comparison-bet based algorithms (Lim et al., 2020; Lee & Kim, 2021; Li et al., 2014; Nguyen et al., 2018) have been proposed. Instead of predicting the rank directly, they learn binary relation between objects, such as order or metric (Hrbacek & Jech, 1984). These relations provide useful information for rank estimation: an order indicates the relative priority between objects & and y, while metric informs of the distance between them.
In order learning (Lim al., 2020; Lee & Kim, 2021), comparator is learned to classify the relationship between r and into one three cases: x is 'greater than,' 'similar to," or 'smaller than' y. Then, they estimate the rank of test object by comparing it with multiple reference objects with known ranks. This approach is based on the idea that it easier to predict ordering relationships between objects than estimate the absolute ranks; telling the older between two people is easier than estimating their exact ages. However, order learning disregards how much x is different from In other words, it ignores metric information.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) Orderloaming
(b) Metric Metricaaaming
(c) Proposed
Figure 1: Comparison of embedding spaces of (a) order learning (Lim et al., 2020), (b) metric learning (Schroff' et al., 2015), and (c) the proposed algorithm. For each method, we train ResNet18 (He et al., 2016) to classify the RetinaMNIST data (Yang et al., 2021) 1) into the five ordinal classes, depicted by blue, orange. yellow, pink, and green markers. For visualization on the unit sphere, the output dimension of a network is reduced to 3 via a fully connected layer, and each feature is normalized.
On the other hand, metric learning algorithms (Li et al., 2014; Nguyen et al., 2018) employ the triplet constraint on three objects (x, y,, 2). It enforces the distance between x and y to be less than that between z and z in the embedding space if the ranks of (x, z) are in the increasing decreasing order. By its design, the triplet constraint does not fully exploit the order among objects.
Figures 1 (a) and (b) compare embedding spaces, obtained by order learning and metric learning, respectively. Order learning sorts instances according to ranks in general, but instances in each class are scattered in the embedding space. In contrast, metric learning reduces within-class scattering but does not sort instances properly. For example, both blue and orange ranks are adjacent to the yellow one, although they should be arranged in the order of blue, orange, and yellow. Unlike ordinary classification, different errors have different severities in rank estimation: misclassifying an object in the blue rank as yellow one is severer than mistaking it for an orange one. Ordering in the embedding space is important to avoid such severe errors.
We propose novel algorithm, GOL, to estimate the rank of an object reliably by exploiting both order and metric relations. To this end, we construct an embedding space, which the direction and distance between objects represent the order and metric relations between their ranks. For the construction, we formulate two geometric constraints in the embedding space: 1) the order constraint enforces the feature vectors of instances to be arranged according to their ranks, and 2) the metric constraint makes the distance between instances reflect their rank difference. To satisfy these two constraints simultaneously, we introduce reference points that guide the region of each rank in the embedding space. Then, we use the simple k-NN rule in the embedding space to estimate the rank of test instance. Extensive experiments show that GOL constructs high-quality embedding spaces and thus provides excellent rank estimation performances.
The contributions of this paper can be summarized as follows.
GOL is the first attempt to design embedding space in which the direction and distance between objects represent their order and metric relations.
We introduce a novel metric, called DRR, to assess the quality of embedding spaces for rank estimation. Then, is shown that GOL effectively sorts and separates instances according to their ranks in an embedding space, as illustrated in Figure (c).
GOL achieves state-of-the-ar performances on various benchmark datasets for facial age estimation, HCI classification, and aesthetic score regression. Specifically, GOL, performs the best in 20 out of 25 benchmark tests.
2 Related Work
Ordinal regression: Many ordinal regression methods have been developed to estimate the rank of an object directly using classifiers or regressors. Rothe et al. (2015) employed tens of classifiers to yield the average of their predictions as output. Yi et al. (2014) developed a regressor to estimate the rank of an image using multi-scale patches. Also, Frank & Hall (2001) employed multiple binary classifiers, each of which tells whether the rank of an object is higher than series of thresholds or not. However, such direct estimation of ranks is challenging even for human beings in general; e.g.,

---

humans usually predict only a rough range of another one's age with limited confidence. Thus, Diaz & Marathe (2019) trained regressor using soft ordinal labels alleviate penalties on close predictions. Furthermore, Li et al. (2021) and Li et al. (2022) modeled the uncertainty of each prediction Gaussian distribution. In contrast to these methods, the proposed algorithm provides more accurate rank estimates, although it uses only single encoder network with the simple k--N rule and makes no complicated probabilistic assumptions.
Order learning: Lim et (2020) first proposed the notion of order learning, which learns ordering relationships between objects and determines the rank of an unseen object by comparing it with references with known ranks. It yields promising results because relative assessment easier than absolute assessment in general. Lee & Kim (2021) improved the performance of order learning by finding more reliable references. They decomposed object information into an order-related feature and an identity feature and showed that objects with similar identity features can be compared more reliably. Also, Shin et al. (2022) extended the classification approach in (Lim et al., 2020; Lee & Kim, 2021) regression- based one. These order learning methods, however, require significant computational cost to find reliable references from an entire training set. Moreover, for rank estimation, they should do comparisons with many references with different ranks because they consider only relative priorities between objects. On the contrary, the proposed algorithm simply carries out the k-NN search to yield outstanding rank estimation results.
Metric learning: Metric learning aims to construct embedding space in which the distance between objects reflects their semantic difference. Most metric learning algorithms (Schroff et al., 2015; Deng et al., 2019) are for ordinary classification, clustering, or image retrieval tasks. Therefore, they enforce an object to be located near other objects in the same class in the embedding space but far from objects in different classes. However, in rank estimation, this approach may be suboptimal because it does not consider ordinal relationships among classes. For example, in movic rating, it does not discriminate the class difference between 'outstanding" and 'very good' from that between "outstanding' and 'poor.' To alleviate this problem, Xiao et al. (2009) designed metric, called labeled distance, to measure semantic similarities between objects and attempted to preserve local semantic structures in the feature space. Moreover, to preserve the ordinal relationships, Li et al. (2012) developed a metric learning algorithm to make the distances between objects proportional to their rank differences. Also, Tian et al. (2016) employed series of margins to explicitly impose different embedding distances according to rank differences. Suárez et (2021) attempted sort the embedding distances between pairs of objects according to their rank differences.
3 Proposed Algorithm
3.1 Preliminary - Order and Metric
Mathematically, both order and metric are binary relations (Hrbacek & Jech, 1984). An order (Schröder, 2003), denoted by S, set + (6u, o ...001] should satisfy the properties of
Reflexivity: 6; I 0; for all i,
Antisymmetry: 0, < 0, and 02 < 0, imply 0, = 0z,
Transitivity: 0, 0 and 03 < 02 imply 0, S O2-
On the other hand, a metric (Rudin, 1991) is distance function d satisfying
Nonnegativity: d(0,, 0,) 2 for all i,j. and d(O,, 0,) = 0 and only if 0, = 0,,
Commutativity: d(f;, 02) d(0,, 0,) for all i,j.
Triangle inequality: d(f,, 0&) < d(e,, 8,) + d(f,, 0x) for all i,
In rank estimation, an order describes the priorities of ranks or classes the set = {lu...... ....,,0M11 } where each rank represents one or more object instances. For example, in age estimation, 0g may represent i-year-olds, and 017 < 032 indicates that 17-year-olds are younger than 32-year- olds. Let 0(.) be the rank function, and let and be instances. Then, A(x) 017 means that person x is 17-year-old. Also, metric describes the difference between ranks in O. For example, d(0(z), ((y)) 15 means that two people and y 15 years apart.
3

---

Rank
Reference points Instance embockinggs
+ocy+1
h
By
hy
X
Ta(y))
Te(y))
Pb
nx
fxx
y
h
Ta(K)
Direction vector
To(#)
Byy
Forward
Attratti
Tec0-1
Backward dinsction
Fe00-1
Repulsion
Instances
Encoder
Embedding space
Order constraint
Metric constraint
Figure 2: An overview of the proposed GOL algorithm. It recommended watch the accompanying video of the proposed algorithm.
3.2 Embedding Space Construction
Given an object instance x, the objective is to estimate its rank (()). such rank estimation, an order and metric convey complementary information: the order provides directional information between ranks, while the metric does length (or magnitude of difference) information. In age estimation, let us consider three age ranks 02 017, and 0z2. Since 82 017 < 632, the order informs that 0g and 032 are at the opposite sides with respect to 817- On the other hand, since d(02, 017) = d(0g2, 817) = 15, the metric indicates how far O2 and 032 from 017. this case, both lengths are identically 15.
For rank estimation, the pairwise comparison methods (Lim et al., 2020; Lee & Kim, 2021; Nguyen et al., 2018) attempt to learn these relations. However, Lim et al. (2020) and Lee & Kim (2021) exploit the order relation only, whereas Nguyen et al. (2018) use the metric relation only to train their neural networks. Thus, the conventional methods may yield sub-optimal results.
To learn both order and metric relations, we propose a geometric approach called GOL. It contains two types of geometric constraints that enforce directional (order) and distance (metric) relationships between object instances according their ranks in an embedding space. Specifically, the order constraint sorts instances directionally according to the ranks, while the metric constraint separates two instances farther if their rank difference is larger. Figure an overview of GOL.
Order constraint:
Suppose that there are M ranks in training set X. Without loss of generality, the ranks are assumed be consecutive integers in = {0, 1, I....: M - 13. In Figure 2, an encoder h maps each instance E X' into a feature vector h. = h((x) in embedding space. As h, we adopt VGG16 (Simonyan & Zisserman, 2015) without fully connected layers. The output of the last pooling layer is normalized so that h 1. Thus, the embedding space is a unit hypersphere.
As in Lim et al. (2020) and Lee Kim (2021), we classify the ordering between two instances æ and in X' into three categories:
x y if 0(x) - 0(v) >T, IR yf (f 0(v) TT, x K if 0(x) 0(v) < -T, (1)
where is threshold. For instance ordering, notations ' ~,, > used instead of =,=,>,²'
The order constraint encourages instances to be sorted according their ordering relationships. In other words, for two instances & and with ordering < the vector from h₂2 to hy should be aligned with the direction of the rank increment in the embedding space. To model such rank directions, we introduce M reference points, To. rl,...., F---1 which are learnable parameters guiding the positions of the M ranks in the embedding space. These reference points are randomly initialized by the Glorot normal method (Glorot & Bengio. 2010) and jointly optimized with encoder parameters during training-
Let us define the direction vector v(r, 8) from point r to point 8 on the unit hypersphere as
v(r, (s r)//|s - r|]-
Then, v(r,,r)) ry) is called the rank direction from rank i to rank j. Also, the rank direction v(rs, ry) is forward if i < and backward if > j. Note that forward directions may differ from one another, for they may represent different physical changes. For example, in age estimation, visual variations from 0-years-old to 5-years-old are mainly due to craniofacial development, whereas those from 45-ycars-old old to 50-years-old are due to skin aging (Geng al., 2007).

---

If x < we determine the forward and backward rank directions, respectively, by
t = v(Ta(z), Te(s)) (ex))
(3)
vb = u(Ta(z), T0(x)-1).
(4)
Then, the encoder is trained so that the embedded features hg and hy satisfy the order constraint:
I x + ofu(hz: hy) > ofz(h,,))))) hg).
(5)
In other words, the direction vector vhz hy) should be aligned more with the forward direction Vp than with the backward direction Ub.
To enforce the order constraint in (5), we compute the softmax probability pzv [p :P'" where and P Torder as the cross entropy between pa8 and g0 = [or" ay " = 0)". given by
Pf e(fu(h_.At)) (h)))()) + e c"ba(hashg))
TV We then define the order loss
Lorder - OF log Pr + %b logp T.
(6)
The order loss for case is formulated similarly in symmetric manner.
Metric constraint: Next, we formulate a metric constraint to make the distance between instances in the embedding space reflect their rank difference. Specifically, it is desirable that
(0(x) - (vv) > T
+ do hy) >
(7)
where de is the Euclidean distance in the embedding space, and y is a margin. Note that [0(z) (vy) > either < y or x > y in (1).
Hence, the metric constraint in (7) equivalent to
x xzy
do (1 hyy 5
From the triangle inequality, we have Ide(r; hx) do (rriill))) | S de (ha, ht) for every reference point r;. So, if x Rs y, we also have Ide (r., hw) - do (r,, hg)) S To encourage this inequality, we define loss Lazsy
Tanzy
Σ max - de(ri, hn)) 7. 0). IE0
(9)
On the contrary, if x y. it should be that (1 hy) 7- Thus, define another loss
max(de(r;;))) hyy de (r3, ha) max(do hx) (re, h2y + +7, 0) jo2
0).
(350(2)
(10)
To minimize the first sum, de hr) should be reduced, while (ri, hy) should be increased. Thus, reference points rs, < S 0(x), are trained to attract h2 and repel hy. as illustrated in Figure The second sum in (10) is similar. We do not use reference points rı, ((r) < < 0(y), which tend to be between ht and hy and unhelpful for guiding them. The derivation of Ly in (10) from the metric constraint in (7) is provided in Appendix B. Also, Lavy is formulated symmetrically.
Then, define the metric loss Lemetric as
Lmmetric [x y] Lury + [x RJ y/ Luzy + [x - y] La-y
(11)
where |-] is the indicator function. Note that, in (9) or (10), we make h2 and hy attract repel each other indirectly through reference points r;. This is because the direct attraction or repulsion using de Ayy may cause hú, and h, to move in arbitrary directions. Because the reference points are trained also with the order loss in (6), they provide proper directional guidance of attraction or repulsion in (9) or (10). Experimental analysis on Lzz<y in (10) is available in Section 4.4.
Loss function: addition to the losses for the order and metric constraints, we employ the center loss (Nguyen et al., 2018), which aims at locating each reference point the center of all instances with rank
Leenter de(re())) hww + do(re())) hy).
(12)
5

---

Table 1: Comparison of B2W and DRR scores on the MORPH II, CACD, and Adience datasets.
MORPH (setting A)
CACD (validation split)
Adience
Algorithm
B2W
DRR;
DRRs.5
B2W
DRR1.
DRRo.s
B2W
DRR1.o
DRRo.5
ML (Schroff el ul., 2015)
41.20
10.06
8.94
3.90
2.89
2.55
5.12
3.21
2.98
MV (Pan et al., 2018)
72.47
10.37
9.80
24.21
6.23
5.84
7,08
3.29
3.24
OL (Lim et al., 2020)
43.94
12.79
11.44
8.77
5.60
4.86
13.29
7.27
6.53
MWR-G (Shin et al., 2022)
30.42
10.66
9.28
9.99
6.00
5.24
7.84
5.20
4.71
Proposed GOL.
292.72
29.40
25.77
53.06
12.59
10.82
70.31
32.09
28.37
(a) OL
(b) MWR-G
(c) ML
(d) MV
(e) GOL
Figure 3: Comparison of embedding spaces on Adience: (a) OI (Lim et al., 2020), (b) MWR-G (Shin
et al., 2022), (c) ML (Schroff et al., 2015), (d) MV (Pan et al., 2018), and (e) GOL.
Finally, we use the overall loss function to optimize the encoder parameters and the reference points Ts, which is given by
Luotal
Lorder
+ Lmmetric + Leenter.
(13)
is worth pointing out that reference points play essential roles in all three loss terms in (13). They used to define forward and backward rank directions in Lorder, they help to sort instances directionally according to the ranks. The reference points themselves are also sorted since instances clustered around the reference points because of Teenter- Also, mentioned before, the reference points provide proper directional guidance of attraction or repulsion in Tmmetric to satisfy the metric constraint. In other words, GOL uses the reference points satisfy both order and metric constraints simultaneously and thus constructs a well-arranged, well-clustered embedding space.
3.3 N- Rank Estimation
For rank estimation, we use the simple K-NN rule. Given a test instance x, in the embedding space, we find set N of its k NNs among all training instances in X. Then, the rank of a is estimated by
1 Σ (gy). k
((x)
(14)
VEN
4
Experimental Results
We conduct experiments on embedding space construction and rank estimation. Implementation details and more results available in Appendices C and D, respectively.
4.1 Implementation
We initialize an encoder h with VGG16 pre-trained on ILSVRC2012 (Deng et al., 2009) and reference points with the Glorot normal method (Glorot & Bengio, 2010). The Adam optimizer (Kingma & Ba, 2015) is used with batch size of 32 and a weight decay of 5 x 10-4, and the initial learning rates for the encoder and the reference points are set 10-4 4 and 10-3 respectively. We perform the scheduled learning according to cosine annealing cycles (Huang et al., 2017). For data augmentation, we do random horizontal flips only.
4.2 Embedding Spaces
The proposed GOL algorithm attempts to design an embedding space in which both order and metric constraints are satisfied: instances should be well sorted according to their ranks and well separated if they have big rank differences.

---

Table Comparison of facial age estimation results the four evaluation settings (A, B, C, and D) of MORPH II. Here, means that IMDB- WIKI pre vre-training is performed.
Setting
Setting B
Setting C
Setting
Algorithm
MAE
CS (%)
MAE
CS (%)
MAE CS (%)
MAE
CS (%)
DRFs (Shen al., 2018)
2.91
82.9
2.98
2.17
91.3
MV (Pan et al., 2018)*
2.79
2.16
C3AE (Chao et al., 2019)*
2.75
BridgeNet (Li al., 2019)*
2.38
91.0
2.63
86.0
AVDL (Wen al., 2020y*
2.37
2.53
1.94
OL (Lim al., 2020)*
2.41
91.7
2.75
88.2
2.68
88.8
2.22
93.3
DRC-ORID (Lec & Kim, 2021)*
2.26
93.8
2.51
89.7
2.58
89.5
2.16
93.5
MWR-G (Shin et al., 2022)*
2.24
93.5
2.55
90.1
2.61
89.5
2.16
93.0
Proposed GOL.
2.17
93.8
2.60
89.3
2.51
90.0
2.09
94.2
To measure the quality of embedding space for rank estimation, may adopt the between-class variance to within-class variance (B2W) criterion in Fisher's ' linear discriminant (Duda et al., 2006),
B2W = Σ ic0
(c,, c)) ( (E>
Ce ci)
(15)
iE NER:
where X; = X 8(x) = i], = Σ NEN h2 / is the centroid for X, and c -- Egsx h2/ X is the centroid for all instances in X. A high B2W score indicates that the rank sets X,, 0 M 1, are well separated from one another, while instances in each rank set are compactly distributed.
However, B2W does not consider the ordinal relationships of ranks in θ since it was formulated for ordinary classification. Therefore, to assess embedding spaces for rank estimation, we propose the discriminative ratio for ranking (DRR), given by
Eijenne - i) do (ci, G2)/ E 1 i)
DRR, EiEo Ex.pEX de do hty) Eiso Ex.vEX
(16)
which is the ratio of the average pairwise centroid distance to the average pairwise instance distance in each rank set. Note that, the numerator, ordinal weights (j - -i) are used to emphasize the difference between pair of rank sets with a large rank difference. Here, ,3 is nonnegative parameter for controlling the level of emphasis. A high DRR score is obtained when instances are well sorted and well separated according to their ranks in the embedding space.
Table compares the B2W and DRR scores on the MORPH II (Ricanek & Tesafaye, 2006), CACD (Chen et al., 2015), and Adience (Levi & Hassner, 2015) train data. To compare the embedding spaces as fairly as possible, we use the same encoder backbone of VGG16 for all algorithms. GOL significantly outperforms all conventional algorithms in all tests. In GOL, instances within each rank set X; are located compactly around reference point r; by Leenter in (12), while those different rank sets are well separated by T.metric in (11). Hence, GOL yields outstanding B2W scores. Moreover, GOL excels with larger score gap in DRR, than DRR which means that it arranges the rank sets effectively in the embedding space to reflect their ordinal relationships using Lorder in (6).
Figure visualizes the embedding spaces on Adience. As Figure 1, we add fully connected layer with 3 output neurons to each encoder for the visualization. In (a) and (b), instances are sorted but scattered over the spaces, as the order learning algorithms (Lim et al., 2020; Shin al., 2022) ignore the metric relation of ranks. In (c), instances in each rank set are well clustered, but the rank sets are not sorted because ML (Schroff al., 2015) neglects the order relation. In (d), MV (Pan et al., 2018), which is an ordinal regressor, exhibits large within-class scattering, well as large between-class scattering. In contrast, in (e), GOL constructs well-sorted, well-clustered embedding space.
4.3 Rank Estimation
Since GOL constructs high-quality embedding spaces, it provides excellent rank estimation perfor- mances even with the simple N--N rule.
Facial age estimation: We use four datasets of MORPH II (Ricanek & Tesafaye, 2006). CACD (Chen et al., 2015), UTK (Zhang et al., 2017), and Adience (Levi & Hassner, 2015), as detailed in Appendix D.3.
7

---

Table 3: Comparison in the train and valida- tion settings of CACD and also on UTK.
Table 4: Accuracy (%) and MAE comparison on the Adience and HCI datasets.
Trann
Validation
UTK
Adience
HCI
Algorithm
MAE
MAE
MAB
Algorithm
Acc.
MAE
Acc.
MAE
dLDLF (Shen at. 2017)
4.73
6.77
OR-CNN (Niu al., 2016)
56.7
0.54
0.95
AGEn (Tan et al., 2017)
4.68
CNNPORR (Liu ct al., 2018)
57.4
0.55
50.1
0.82
DRFs (Shen at. 2018)
4.64
5.77
GP-DNNOR (Liu et 2019)
57.4
0.54
46.6
0.76
CORAT. (Can et al. 2020)
5.47
SORD (Diaz & Marathe, 2019)
59.6
0.49
Gostafsson (2020)
DRC-ORID (Lee Kim, 2021)
44.7
0.80
Berg ol. (Berg at al., 202.1)
4,55
POE (1. al., 2021)
60.5
0.47
54.7
0.66
MWR-G (Shin 2022)
4.76
5.75
MWR-G (Shin cl. 2022)
62.2
0.46
52.2
0.60
Proposed GOL
4.52
5.58
4.35
Proposed GOL
62.5
0.43
56.2
0.55
Table 5: Accuracy (%) and MAE comparison on the aesthetics dataset.
Nature
Animal
Urban
People
Overall
Algoritham
Acc.
MAE
Acc.
MAE
Acc.
MAE
Acc.
MAE
Ace.
MAE
CNNN (Liu et al., 2018)
71.0
0.31
68.0
0,34
68.2
0.36
71.6
0.32
69.5
0.33
CNNPOR (Liu al., 2018)
71.9
0.29
69.3
0.32
69.1
0.33
69.9
0.32
70.1
0.32
SORD (Diaz Marathe, 2019)
73.6
0..77
70.3
0.31
73.3
0.28
70.6
0.31
72.0
0.29
POE al., 2021)
73.6
0.27
71.1
0.30
72.8
0.28
72.2
0.29
72.4
0.29
Proposed GOL
73.8
0.27
72.4
0.28
74.2
0.26
69.6
0.31
72.7
0.28
In Table 2, we compare the performances in the four evaluation settings of MORPH II, which is one of the most popular datasets in age estimation. We use the mean absolute error (MAE) and cumulative score (CS) metrics. MAE is the average absolute error between estimated and ground truth ages (i.e. ranks), and CS computes the percentage of images whose absolute errors are less than equal to a tolerance level = 5. Note that all algorithms in Table 2 use VGG16 as the encoder backbones, except for C3AE employing a shallow CNN. As for Shin et al. (2022), the scores of the global p-regressor are compared since their local scheme employs as many as six independent VGG16 encoders recursively. GOL performs the best in 5 out of 8 tests, including setting C, which is the most challenging task. Furthermore, unlike most conventional algorithms, GOL does not do the IMDB-WIKI pretraining to boost performances. Even without the pretraining, i.e. by employing only MORPH II training data, GOL yields outstanding results in Table 2.
Table 3 compares the results on CACD, which is bigger dataset containing over 100,000 natural face shots in diverse environments. GOL outperforms the second- best methods with meaningful gaps of 0.12 and 0.17 in the train and validation settings, respectively, which indicates that it can cope with large and diverse data effectively as well.
Table 3 also compares the results on UTK. Note that Gustafsson et al. (2020) and Berg et al. (2021) employ the deep ResNet50 network (He et al., 2016) as their encoders, and MWR-G (Shin et al., 2022) predicts ranks using complicated, recursive regressor. In contrast, GOL uses the shallower VGGI6 encoder and performs rank estimation based on the simple k-NN search. Nevertheless, yields an excellent result, for its geometric constraints help to construct effective embedding space.
Table 4 shows the results on Adience, where each image labeled as one of 8 age groups. Compared with the second- best MWR-G, GOL improves the accuracy by 0.3% and reduces MAE by 0.03.
HCI classification: The HCI dataset (Palermo et al., 2012) is used for estimating the shooting decade of a photograph. It contains 1,325 images from five decades 1930s 1970s. Table 4 lists the results on HCI. GOL yields the best performances as well, by improving the accuracy by 1.5% and reducing MAE by 0.05 compared with the second-best methods. This means that GOL yields reliable results even for a small dataset, which may be unfavorable for the N--N estimation.
Aesthetic score regression: The aesthetics dataset (Schifanella et al., 2015) contains 15,687 images in four categories. Each image is annotated with a 5-scale aesthetic score. In Table 5, follow the experimental setting in (Liu et al., 2018). It is challenging to estimate aesthetic scores reliably due to the subjectivity and ambiguity of aesthetic criteria, but GOL performs the best in 8 out of 10 tests. Compared with the tate-of-the-ar POE (L.i et al., 2021), GOL improves the accuracy by 0.3% and reduces MAE by 0.1 overall.
8

---

Table 6: Ablation studies for the loss function in (13) on the MORPH II and CACD datasets.
MORPH II (setting A)
CACD (validation split)
Lorder
Lemetric
Licenter
MAE
B2W
DRR1.0
DRRg.5
MAE
B2W
DRR,
DRRo.s
8.71
10
0.07
10.0
0.05
0.38
2.71
21.48
8.89
7.69
8.59
5.16
v
2.43
44.94
10.93
9.63
6.14
12.72
5.79
5.07
/
2.17
292.7
29.40
25.77
5.58
53.06
12.59
10.82
Age
10-14 15-19 20-24
30-34 35-30
40-44
45-49
55-59
60-64
45-43
50-M4
25.00
30-18
20-20
10-18
(6)
(b) Licenter
Forder
(c) Foonter Ammttric
(d) GOL
Figure 4: Visualizing the embedding spaces of the ablated methods in Table 6 the CACD dataset:
(top) reduced 3D spaces and (bottom) t-SNE visualizations original 512D spaces.
Table 7: MAE comparison of alternative choices for Luzzy in (10) on MORPH and CACD.
Method
Altemntive Lu-cp
MORPH (setting
CACD
max(? da(As, hy
2.35
5.77
II
max(de(toiz), ha) - de (e(oc))) by) + 7,0) + max (Pocs) (rocan ha) + + (10)
2.34
5.75
5.58
4.4 Analysis
Ablation study: Table 6 compares ablated methods for the loss function in (13). Method employs Leenter only. In II and III, Lmmetric and Lorder are excluded, respectively. Compared with IV (GOL), method I degrades the performances severely, since the center loss alone cannot construct meaningful embedding space; a trivial solution to minimize Lcenter can be obtained by merging all instances and all reference points into single point. Also, from and IV, we see that the metric constraint significantly improves the performances by separating instances with different ranks from each other. From III and IV, we see that the order constraint also improves results by arranging instances directionally according to their ranks. To summarize, both order and metric constraints improve the results and are complementary to each other.
Figure (d)~(d) show the embedding spaces for the ablated methods I~IVV in Table 6, respectively. The top row visualizes the reduced 3D embedding spaces, as described Appendix C.2, while the bottom row does the original 512D embedding spaces via t-SNE (Maaten & Hinton, 2008). In (a), single point. In (b), instances
only Leenter is used, so the 3D embedding space almost collapses are directionally aligned, but adjacent rank sets overlap one another. In (c) and (d), the 3D embedding spaces seem similar. However, GOL yields more clearly ordered instances the t-SNE visualization of the original embedding space than 'Loxnter + Temetric" does.
Alternatives to LIzy: Table 7 compares alternative loss terms for L in (10). Method directly increases de (hz. Ay) to make larger than 7. Method uses only two reference points Te(a)) and re(y)- In other words, unlike method III (i.e. GOL.) it does not employ reference points r;, S 0(x), and 0(y) < ry I M 1. For each method, Lavy is also modified accordingly. Method degrades

---

Step 1 (00
1000
5000
10000
Refenmo points Centrnids
Figure 5:
The transition of embedding space for the Adience dataset during the GOL training.
Table 8: Comparison with order learning algorithms in terms of testing time and network complexity.
Algorithm
Time (ms)
# parameters (M)
OL (Lim et al., 2020)
11.18
15.51
DRC-ORID (Lee & Kim, 2021)
4.93
45.86
MWR-G (Shin al., 2022)
1.61
15.77
Proposed GOL
0.07
14.75
the performances badly, har and by move in arbitrary directions. Also, GOL performs better than because attraction and repulsion with many reference points facilitate the positioning of instances in the embedding space, as well the reference points themselves.
Embedding space transition: Figure 5 visualizes the transition of an embedding space for Adience. As the GOL training step goes on, we see that instances are gradually sorted and separated according to their ranks to satisfy the geometric constraints. After the convergence, for each rank, the reference point r; and the centroid c; are close each other, indicating that instances are clustered around their reference points. Moreover, those reference points are well sorted as well.
Comparison with order learning: The conventional order learning algorithms (Lim et al., 2020; Lee & Kim, 2021; Shin et al., 2022) provide competitive ranking performances, but they require relatively heavy computations. Table lists the testing times. To estimate the rank of a test instance, OL uses five references for each of the M ranks; e.g., 300 comparisons should be made if M 60 as in typical age estimation. Moreover, OL and DRC-ORID need additional processes, such as MAP estimation, to estimate rank based on ordering relationships. Also, MWR-G estimates the rank by comparing a test instance with multiple reference pairs recursively. In contrast, GOL simply performs the k-NN estimation, which be done efficiently in parallel manner, and thus is about 160 and 23 times faster than OL and MWR-G, respectively.
Furthermore, the order learning algorithms should select references among a training set X' through a complicated optimization process; e.g., OL compares all possible pairs of training instances with ((1x2²) complexity to select the most reliable references, and DRC-ORID (Lee & Kim, 2021) performs joint network training and clustering for reference selection. On the contrary, GOL requires such process. Also, in Table 8, the proposed algorithm demands the fewest parameters because it directly estimates the rank of an instance in the embedding space through the k-NN search, whereas the order learning algorithms should adopt comparators regressors tailored for each task.
5 Conclusions
The GOL algorithm for rank estimation was proposed in this work. First, we construct an embedding space based on the two geometric constraints, which enforce the direction and distance between instances to represent the order and metric relations between their ranks. Then, we perform the simple k-NN search in the embedding space for rank estimation. Extensive experiments on various rank estimation tasks demonstrated that GOL constructs high-quality embedding spaces and yields excellent rank estimation results.
Acknowledgments
This work was conducted by Center for Applied Research in Artificial Intelligence (CARAl) grant funded by DAPA and ADD (UD1900318)) IRD) and supported by the NRF grants funded by the Korea government (MSIT) (No. NRF-202 1R A4A 1664 and No. 2RF-2022RJ
10

---

References
Axel Berg, Magnus Oskarsson, and Mark O' CoCnnor. Deep ordinal regression with label diversity. In ICPR, 2021. 8
Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. Rank-consistent ordinal regression for neural networks. Pattern Recog. Lett., 140:325-331, 2020.
Zhang Chao, Shuaicheng Liu, Xun Xu, and Ce Zhu. C3AE: Exploring the limits compact model for age estimation. In CVPR, 2019. 7
Bor-Chun Chen, Chu-Song Chen, and Winston H. Hsu. Face recognition and retrieval using cross-age reference coding with cross-age celebrity dataset. IEEE Trans. Multimedia, 17:804-83 815, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 6
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. ArcFace: Additive angular margin loss for deep face recognition. In CVPR, 2019. 3
Raul Diaz and Amit Marathe. Soft labels for ordinal regression. In CVPR, 2019. 8,
Richard O. Duda, Peter E. Hart, and David G. Stork. Pattern Classification. John Wiley & Sons, 2006. 7
Eibe Frank and Mark Hall. A simple approach ordinal classification. ECML-PKDD, 2001. 2
Xin Geng, Zhi-Hua Zhou, and Kate Smith-Miles. Automatic age estimation based on facial aging patterns. IEEE Trans. Pattern Anal. Mach. Intell., 29:2234-222 022224--2220, 2007. 4
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. AISTATS, 2010. 4,
Fredrik K. Gustafsson, Martin Danelljan, Goutam Bhat, and Thomas B. Schon. Energy-based models for deep probabilistic regression. In ECCV, 2020. 8
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016. 2,
Karel Hrbacck and Thomas Jech. Introduction to Set Theory. Marcel Dekker, Inc., 2nd edition, 1984. 1,3
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q. Weinberger. Snapshot ensembles: Train 1, get M for free. ICLR, 2017.
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Scon-Ho Lee and Chang-Su Kim. Deep repulsive clustering of ordered data based on order-identitiy decomposition. In ICLR, 2021. 1, 3, 7, 8, 10
Gil Levi and Tal Hassner. Age and gender classification using convolutional neural networks. In CVPR Workshops, 2015.
Changsheng Li, Qingshan Liu, Jing Liu, and Hanqing Lu. Learning distance metric regression for facial age estimation. In ICPR, 2012. 3
Changsheng Li, Qingshan Liu, Jing Liu, and Hanqing Lu. Ordinal distance metric learning for image ranking. TNNLS, 26: 15511111559, 2014. 1,2
Qiang Li, Jingjing Wang, Zhaoliang Yao, Yachun Li, Pengju Yang, Jingwei Yan, Chunmao Wang, and Shiliang Pu. coimodal-concenttaaed loss: Fully adaptive label distribution learning for ordinal regression. In CVPR, 2022. 3
Wanhua Li, Jiwen Lu, Jianjiang Feng Chunjing Xu, Jie Zhou, and Qi Tian. BridgeNet: A continuity- aware probabilistic network for age estimation. In CVPR, 2019. 7

---

Wanhua Li, Xiaoke Huang, Jiwen Lu, Jianjiang Feng, and Jie Zhou. Learning probabilistic ordinal embeddings for uncertainty-awaat aware regression. In CVPR, 2021. 1,
Kyungsun Lim, Nyeong- Ho Shin, Young- Yoon Lee, and Chang-Su Kim. Order learning and its application to age estimation. In ICLR, 2020. 1, 2, 3, 4, 6, 7, 10
Yanzhu Liu, Adams W. K. Kong, and Chi Keong Goh. A constrained deep neural network for ordinal regression. In CVPR, 2018. 1,
Yanzhu Liu, Fan Wang, and Adams W. K. Kong. Probabilistic deep ordinal regression based on Gaussian processes. In CVPR, 2019. 8
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579-2605, 2008.
Bac Nguyen, Carlos Morell, and Bernard D. Baets. Distance metric learning for ordinal classification based triplet constraints. Knowledge-Basee Systems, 142:17-28, 2018. 2, 4,
Zhenxing Niu, Mo Zhou, Le Wang. Xinbo Gao, and Gang Hua. Ordinal regression with multiple output CNN for age estimation. In CVPR, 2016. 8
Frank Palermo, James Hays, and Alexei Efros. Dating historical color images. In ECCV, 2012. 1,
Hongyu Pan, Hu Han, Shiguang Shan, and Xilin Chen. Mean-variance loss for deep age estimation from face. In CVPR, 2018. 6, 7
Karl Ricanek and Tamirat Tesafaye. age-progression. In FGR, 2006.
MORPH: A longitudinal image database of normal adult
Rasmus Rothe, Radu Timofte, and Lue V. Gul. DEX: Deep expectation of apparent age from single image. In ICCV Workshops, 2015. 2
Walter Rudin. Functional Analysis. McGraw Hill,, 2nd edition, 1991. 3
Rossano Schifanella, Miriam Redi, and Luca M. Aiello. An image is worth more than a thousand favorites: Surfacing the hidden beauty of Flickr pictures. In ICWSM, 2015. 1,
Bernd S. W. Schröder. Ordered Sets: An Introduction. Springer, 2003. 3
Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A unified embedding for face
recognition and clustering. In CVPR, 2015. 2, 3, 6.
Wei Shen, Kai Zhao, Yilu Guo, and Alan Yuille. Label distribution learning forests. In NIPS, 2017. 8
Wei Shen, Yilu Guo, Yan Wang, Kai Zhao, Bo Wang, and Alan Yuille. Deep regression forests for age estimation. CVPR, 2018. 7.8
Nycong-Ho Shin, Seon-Ho Lee, and Chang-Su Kim. Moving window regression: A novel approach to ordinal regression. In CVPR, 2022. 1, 3, 6, 8, 10
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICML, 2015.
Juan L. Suárez, Salvador Gareía, and Francisco Herrera. Ordinal regression with explainable distance metric learning based on ordered sequences. Machine Learning, 110:2729-2783 2762, 2021. 3
Zichang Tan, Jun Wan, Zhen Lei, Ruicong Zhi, Guodong Guo, and Stan Z. Li. Efficient group-n encoding and decoding for facial age estimation. IEEE Trans. Pattern Anal. Mach. Intell., 40(11): 2610-2623, 2017. 8
Qing Tian, Songcan Chen, and Lishan Qiao. Ordinal margin metric learning and its extension for cross-distribution image data. Information Sciences, 349:50-64, 2016.
Xin Wen, Biying Li, Haiyun Guo, Zhiwei Liu, Guosheng Hu, Ming Tang, and Jinqiao Wang. Adaptive variance based label distribution learning for facial age estimation. In ECCV, 2020. 7
12

---

Bo Xiao, Xiaokang Yang, Yi Xu, and Hongyuan Zha. Learning distance metric for regression by semidefinite programming with application human age estimation, Proc. ACM Multimedia, 2009. 3
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. MedMNIST v2: large-scale lightweight benchmark for 2D and 3D biomedical image classification. arXiv preprint arXiv:2110.1477 2021. 2
Dong Yi, Zhen Lei, and Stan Z. Li. Age estimation by multi-scale convolutional network. In ACCV, 2014. 2
Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/reeression by conditional adversarial autoencoder. CVPR, 2017."
13


Reviews:
[{"review_id": "pCWKa3P8qh9", "reviewer": "Reviewer_4nRC", "summary": "A GOL embedding space represents the direction and distance between objects represent order and metric relations between their ranks, by enforcing two geometric constraints 1) the order/rank constraint  and 2) metric constraint reflects rank difference. \n\nEstimates a test object rank are achieved by kNN, and a metric called discriminative ratio for ranking (DRR) estimates the quality of rank estimation embedding spaces.\n\nExperiments on facial age estimation, historical color image (HCI) classification, and aesthetic score regression demonstrate that GOL constructs effective  embedding spaces and yields rank estimation performances.", "questions": "Would be interesting to know the link to traditional ranking theories such as SIFT-Rank.\n\nIn Figure 1 c), are we seeing ordered objects arranged in a \"band\" around a hypersphere?\n", "limitations": "There are no obvious limitations, except demonstration on more datasets & tasks.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "It seems GOL is the first attempt to design an embedding space in which the direction and distance between objects represent their order and metric relations. The GOL algorithm performs best in 80% of benchmark datasets for facial age estimation, HCI classification, and aesthetic score regression.\n\nIt is not clear how here, the rank and DRR metric are related to traditional (mathematical) methods defining according rank and order geometry, without learning. For example, ranked gradient orientation features are used for head age estimation, classification, with much success, prior to this work.\n\nE.g. please see SIFT-Rank, SIFT being the \"scale-invariant feature transform\" (the state-of-the-art before GPU-based deep CNN learning), and Rank being the order statistics of the SIFT descriptor (equivalent to uniform orientation sampled gradient filtering).\nhttps://ieeexplore.ieee.org/document/5206849\n\nToews, M., Wells, W.M. and Zöllei, L., 2012, October. A feature-based developmental model of the infant brain in structural MRI. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 204-211). Springer, Berlin, Heidelberg.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "99DWMlgx03q", "reviewer": "Reviewer_3S1v", "summary": "This paper introduces geometric order learning (GOL) method for rank estimation by enforcing two geometric constraints: the order constraint and the metric constraint. The order constraint enforces the feature vectors of instances to be arranged according to their ranks, and the metric constraint makes the distance between instances reflect their rank difference. The paper also proposes discriminative ratio ranking metric to assess the quality of embedding spaces for rank estimation. Extensive experiments demonstrate that GOL constructs effective embedding spaces and yields excellent rank estimation performances.", "questions": "- The proposed GOP method rely on reference points, it is not clear how to select these reference points and how they affect the performance of GOP.\n- As the network architectures of different methods in Table 1 are different, so whether the comparison is fair?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths：\n+ A geometric order learning (GOL) method for rank estimation is proposed by enforcing the order constraint and the metric constraint.\n+ A discriminative ratio ranking metric is introduced to assess the quality of embedding spaces for rank estimation.\n+ Experiments are conducted to demonstrate that GOL constructs effective embedding spaces and yields excellent rank estimation performances.\n\nWeaknesses:\n- The proposed GOP method rely on reference points, it is not clear how to select these reference points and how they affect the performance of GOP.\n- As the network architectures of different methods in Table 1 are different, so whether the comparison is fair?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3fcvCxF6E9_", "reviewer": "Reviewer_ptgw", "summary": "- The paper introduces an algorithm for learning an embedding space using which the rank of an object can be estimated, e.g. age of a person based on face images. \n- The embedding space is constrained such that the ordering and distance between training instances are preserved. \n- The authors introduce a metric for embedding space evaluation by repurposing the B2W metric (inter-class variance/intra-class variance) to account for the rank ordering of the data embeddings. ", "questions": "- The order constraint formulation uses $v_b$ which is based on consecutive ranks while the forward uses difference in references corresponding to $x$ and $y$ ranks - Is there a reason for this asymmetry?\n- Where does the gain in test runtime come from? I would assume comparing with 5 references/rank would be faster than searching for the k-nearest neighbor from $N$ training examples.\n- Also, comparing methods where one uses a compressed representation in the form of references while the other uses the entire training set is a fair setting. The proposed method does learn references during training but seems to ignore these for inference - can the rank estimation be performed with just the references?\n\nMinor: \n- How long is the model trained for? Is there a stopping condition?\n- Is there any particular reason for choosing the VGG backbone?\n", "limitations": "- The authors show specific failure cases in facial age estimation where the lighting condition, overexposure, and other image perturbations affect the performance of the model - I think this is a general limitation of the dataset and is not specific to the model. A more careful analysis of the proposed method and its limitations might be useful.\n- The paper lacks a discussion on why the embedding space has to be ordered - This is crucial for the premise of the method proposed. A method that embeds each rank in distinct spaces will be good at rank estimation (I believe this would explain the performance of other methods compared in the paper).\n- The reported results seem to be based on a single run - It would be useful to report results based on multiple random initializations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper is easy to read. However, parts of the introduction and related work section are repetitive. \n- The order and distance properties listed as preliminaries are common knowledge and seem to add no specific detail to the objective formulation in the paper.\n- The central idea of the paper is to combine concepts from order learning and metric learning. The paper presents an objective by combining the two constraints. \n- The visualization presented in the paper shows that the proposed method learns embeddings that are ordered.\n- Ablation study shows that each loss objective added helps improve the performance of the model. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept