Deep Repulsive Clustering of Ordered Data Based on Order-Identity Decomposition
Authors: Seon-Ho Lee
Chang-Su Kim
Venue: ICLR 2021 Poster (2021)
Keywords: Clustering
order learning
age estimation
aesthetic assessment
historical color image classification

Abstract: We propose the deep repulsive clustering (DRC) algorithm of ordered data for effective order learning. First, we develop the order-identity decomposition (ORID) network to divide the information of an object instance into an order-related feature and an identity feature. Then, we group object instances into clusters according to their identity features using a repulsive term. Moreover, we estimate the rank of a test instance, by comparing it with references within the same cluster. Experimental results on facial age estimation, aesthetic score regression, and historical color image classification show that the proposed algorithm can cluster ordered data effectively and also yield excellent rank estimation performance.

Published as a conference paper at ICLR 2021
DEEP REPULSIVE CLUSTERING OF ORDERED DATA BASED ON ORDER-IDENTITY DECOMPOSITION
Scon-Ho Lee and Chang-Su Kim
School of Electrical Engineering, Korea University seonhol eeêmal korea..... ac. kr, changsukii@kkrrea.co
ABSTRACT
We propose the deep repulsive clustering (DRC) algorithm of ordered data for ef- fective order learning. First, develop the order-identity decomposition (ORID) network to divide the information of an object instance into an order-related fea- and identity feature. Then, we group object instances into clusters ac- cording to their identity features using a repulsive term. Moreover, we estimate the rank of a test instance, by comparing with references within the same clus- ter. Experimental results on facial age estimation, aesthetic score regression, and historical color image classification show that the proposed algorithm can cluster ordered data effectively and also yield excellent rank estimation performance.
INTRODUCTION
There are various types of 'ordered' data. For instance, in facial age estimation (Ricanek & Tesafaye 2006), face photos are ranked according to the ages. Also, in video-sharing platform, videos can be sorted according to the numbers of views or likes, In these ordered data, classes, representing ranks or preferences, form an ordered set (Schröder 2003). Attempts have been made to estimate the classes of objects, including multi-class classification (Pan et al.) 2018). ordinal regression (Frank & Hall 2001) metric regression (Fu & Huang 2008). Recently, a new approach, called order learning [Lm et al. 2020), was proposed to solve this problem.
Order learning based on the idea that is easier to predict ordering relationship between objects than to estimate the absolute classes (or ranks); telling the older one between two people is easier than estimating their exact ages. Hence, in order learning. the pairwise ordering relationship is learned from training data. Then, the rank of test object is estimated by comparing with reference objects with known ranks. However, some objects cannot be easily compared. It is less easy to tell the older one between people of different genders than between those of the same gender. Lim al. (2020) tried to deal with this issue, by dividing an ordered dataset into disjoint chains. But, the chains were not clearly separated, and no meaningful properties were discovered from the chains.
In this paper, we propose a reliable clustering algorithm, called deep repulsive clustering (DRC), of ordered data based on order-identity decomposition (ORID). Figure [shoss clustering example of ordered data. Note that some characteristics of objects, such as genders or races in age estimation, related to their ranks, and the ranks of objects sharing such characteristics can be compared more reliably. To discover such characteristics without any supervision, the proposed ORID net- work decomposes the information of an object instance into an order-related feature and an identity feature unrelated to the rank. Then, the proposed DRC clusters object instances using their identity features; in each cluster, the instances share similar identity features. Furthermore, given a test in- stance, we decide its cluster based on the nearest neighbor (NN) rule, and compare it with reference instances within the cluster to estimate its rank. To this end, we develop a maximum a posteriori (MAP) estimation rule. Experimental results ordered data for facial age estimation, aesthetic score regression (Kong et al. 2016), and historical color image classification (Palermo et al. 2012) demonstrate that the proposed algorithm separates ordered data clearly into meaningful clusters and provides excellent rank estimation performances for unseen test instances.
The contributions of this paper can be summarized as follows.

---

Published as a conference paper at ICLR 2021
Ordered data
Chster
Chstesing
16
48
56
Chster
Figure I: A clustering example of facial photos, which are ordered according to ages. Without any supervision, the proposed algorithm can obtain meaningful clusters using identity features.
We first propose the notion of identity features of ordered data and develop the ORID network for the order-identity decomposition.
We develop the DRC algorithm to cluster data on unit sphere effectively using repulsive term. We also prove the local optimality of the solution.
We propose the MAP decision rule for rank estimation. The proposed algorithm provides the state-of-the-ee performances for facial age estimation and aesthetic score regression.
2 RELATED WORK
2.1 ORDER LEARNING
The notion of order learning was first proposed by Lim et al. (2020). It aims to determine the order graph of classes and classify an object into one of the classes. In practice, trains a pairwise comparator, which is a ternary classifier, to categorize the relationship between two objects into one of three cases: one object bigger than, similar to, or smaller than the other. Then, estimates the rank of test object, by comparing it with reference objects with known ranks. However, not every pair of objects are easily comparable. Although Lim al, (2020) attempted to group objects into clusters, in which objects could be more accurately compared, their clustering results were unreliable.
Pairwise comparison has been used to estimate object ranks, because relative evaluation is easier than absolute evaluation in general. Saaty (177)) proposed the scaling method to estimate absolute priorities from relative priorities, which has been applied various decision processes, including aesthetic score regression (Lee & Kim 2019). Also, some learning to rank (LTR) algorithms are
based on pairwise comparison (Liu) (2099 Cohen al. 1998 Burges et al. 2005 Tsai al. 2007).
Order learning attempts to combine (possibly inconsistent) pairwise ordering results to determine the rank of each object. Thus, is closely related the Cohen al. s LTR algorithm (1998). which learns a pairwise preference function and obtains a total order of set to maximize agreements among preference judgments of pairs of elements. Also, order learning is related rank aggrega- tion (Dwork et al. (200)) in which partially ordered sets are combined into a linearly ordered set achieve the maximum consensus among those partial sets. Rank aggregation has been studied in various fields (Brüggemann et al. 2004). Since optimal aggregation is harhard, Dwwrk et al. (200T) proposed an approximate algorithm, called Markov chain ordering. There are many other approxi- mate schemes, such as the local Kemenization, Borda count, and scaled footrule aggregation.
2.2 CLUSTERING
Data clustering is a fundamental problem to partition data into disjoint groups. such that elements in the same group are similar one another but elements from different groups are dissimilar. Al- though various clustering algorithms have been proposed (Hartigan & Wong 1979 Ester et al.) 1996 Kohonen 1990; Dhillon Modhal 2001 Reynolds 2009), conventional algorithms often yield poor performance on high-dimensional data due the curse of dimensionality and ineffec- tiveness of similarity metrics. Dimensionality reduction and feature transform methods have been studied to map data into ear transforms, such as PCA (Wold et al. 1987). and non-la non-linear transformations, including kernel methods (Hofmann al. 2008) and spectral clustering (Ng et al. 2002), have been proposed.
feature space, in which they more easily separated. Lin-
2

---

Published as a conference paper at ICLR 2021
Recently, deep neural networks have been adopted effectively feature embedding functions (Le- Cun et al., 2015). and these doep-leamnin--eee based feature embedding functions have been combined with classical clustering algorithms. For instance, [Cronn et al. (2018) proposed deep clustering algorithm based on k-means. It clusters features from a neural network and then trains the net- work using the cluster assignments pseudo-labels. This is done iteratively. Also, YYag et al. (2016) jointly learned feature representations and clustered images, based on agglomerative clus- tering. Chang et al. (2017) recast the image clustering task into binary classification problem to predict whether pair of images belong to the same cluster or different clusters. Similarly to these algorithms, we use neural network determine feature space in which clustering is done more effectively. However, we consider the clustering of ordered data, and each cluster should consists of elements, whose ranks can be compared more accurately.
There are conventional approaches to use clustering ideas to aid in classification or rank estimation. For example, Yan et al. (2015) developed hierarchical classifier, which clusters fine categories into coarse category groups and classifies an object into fine category within its coarse category group. For extreme multiclass classification, Daumé III et al. (2177 7 proposed to predict class label among candidate classes only, which are dynamically selected by the recall tree. is however noted that the leaves of the recall tree do not partition the set of classes. Also, for age estimation, [i et al. (2019) proposed tree-like structure, called bridge-tree, train a local regressor for each group. The set of local regressors can be more accurate than a global regressor to deal with the entire age range. Whereas these conventional approaches group data in the label dimension to perform their tasks more effectively, the proposed algorithm cluster data in the dimension orthogonal the label dimension. In other words, we cluster data using identity features, instead of using order features.
divide data into overlapping age groups and
3 PROPOSED ALGORITHM
3.1 PROBLEM DEFINITION
An order is a binary relation, often denoted by S, on a set C = (0,, ..................... 0m Schhröder 2003). It should satisfy three properties of reflexivity (0, 0, for all i), antisymmetry (0, Ug and 0, < O, imply 0g = 0,) and transitivity (0; < f, and 02 S 0g imply 0, < 0x). Then, θ is called partially ordered set. Furthermore, if every pair of elements are comparable (0. S 0, or f, S 0, for all i, f). O is called a chain or linearly ordered set.
An order describes ranks or priorities of classes. For example, in age estimation, 6, may represent the age class of i-year-olds. Then, 014 S 049 represents that 14-year-olds are younger than 49 49-year-olds. As mentioned previously, it is less easy to tell the older one between people of different genders. An algorithm, hence, may compare subject with reference subjects of the same gender only. In such a case, each age class 0g represents two subclasses Ofemale and gualle of different types, and the algorithm compares only subjects of the same type. [im al. (2020) assumed that subclasses of different types are incomparable and thus the set of subclasses is the union of & disjoint chains, where k is the number of types. However, in many ranking applications, objects of different types can be compared (although less easily than those of the same type are). Thus, instead of assuming incomparability across chains, we assume that there is a total order on θ {0,, Oz...., which each class 0; consists of k types of subclasses, and that object instances of the same type are more easily compared than those of different types.
... 0m J. in
Suppose that n training instances in X' = [I1, T2.1......... Fn are given. Also, suppose that there are m ranks and the ground-truth rank of each instance is known. In this sense, X' contains ordered data. The problem is twofold. The first goal is to decompose the whole instances X' into k disjoint clusters {C, _ in which instances are more easily compared;
== -
where CnC, = 0 for + j. In other words, we aim to partition the ordered data in X' into & clusters, by grouping them according to their characteristics unrelated to their ranks. These characteristics, which tend to remain the same even when an object experiences rank changes, are referred to as 'identity' features in this work. For example, in age estimation, genders or races can be identity features. However, we perform the clustering without any supervision for identity features. Notice
3

---

Published as a conference paper at ICLR 2021
har
G
C
hi
Instance
Encoder
Decoder
Comparator
hor
Red
y
G
Fabe
Irstance
Encoder
Decoder
Discriminetor
Figure 2: An overview of the ORID network.
that instances within cluster would be compared more easily than those across clusters, since they have similar identity features. The number k of clusters is assumed to be known a priori. Impacts of & on the clustering performance are discussed Appendix [.7 The second goal is assign an unseen test instance into one of the clusters and determine its rank by comparing it with reference instances within the cluster. To achieve these goals, we propose the ORID network and the DRC algorithm.
3.2 ORDER- DECOMPOSITION
In general, object instances can be compared more easily, as they have more similar identity features irrelevant to order. Therefore, we decompose the information of each object instance into an order feature and an identity feature. To this end, we propose the ORID network Figure composed of three parts: autoencoder, discriminator, and comparator.
1) Autoencoder: Similarly deep clustering algorithms in YYang al. 2017, Dizaji et al. 2017
Chen et al., 2017 et al. 2017). we use the autoencoder G F.)., based on a neural network, to extract feature vectors. The encoder h² F(x) maps input vector æ feature vector h², while the decoder ê = G(hh") reconstructs 7 from h² By minimizing the reconstruction loss ||x & i. F trained represent x compactly with as little loss of information as possible. We decompose the overall feature htd E Rd du into the order feature h and the identity feature hig given by
lor [MT, 11 1e....,,,7,
hid
[ha. /doo +2....... h tdra 1'/ /11h hh..
+dia
(3)
where dor and dia are the dimensions of h and hig However, without additional control, the output h²t of the neural network J' would be highly entangled (Higgins et al.) 2018). To put together order-related information into her- we employ the comparator.
2) Comparator: Using the order features h and hy of a pair of instances x and y, we train the com- parator, which classifies their ordering relationship into one of three categories 'bigger," 'similar,' and 'smaller":
I x>> if 0(z) - (vv) >T, I if 10(x) - 0(y) T. I kKY if ((x) f(y) < <-. (4)
where 0(-) denotes the class of instance. As in (Lim et al. 2020), ">, R. x' represent the ordering relationship between instances, while ",= do the mathematical order between classes. The comparator outputs the softmax probability nT (p my p It is trained minimize the cross-entropy between nd and the ground-truth one-hot vector gº (g - =²²).. Because it is trained jointly with the autoencoder, the information deciding the ordering relationship tends to be encoded into the order features hür and hy On the other hand, the remaining information necessary for the reconstruction of and i are encoded into the identity features h and h%a
3) Discriminator: We adopt the discriminator that tells real images from synthesized images, generated by the decoder G. Using the GAN loss (Goodfellow et al.) 2014), the discriminator helps the decoder to reconstruct more realistic output å and y.
Appendix A provides detailed network structures of these components in ORID.
3.3 DEEP REPULSIVE CLUSTERING
After obtaining the identity features h h ...., a. of all instances x; E X, we partition them into k clusters. Each cluster contains instances that more easily comparable one another. The
4

---

Published as a conference paper at ICLR 2021
identity features normalized in Eq. (3) and lie the unit sphere in R In other words, cluster data points on the unit sphere. Thus, the cosine similarity is a natural affinity metric. Let CJ. 1 S J k, denote the & clusters. Also, let cj. constrained to be the unit sphere, denote the "centroid" or the representative vector for the instances in cluster Cy. We define the quality of cluster Cj
(5) where the first term is the similarity of an instance in C, to the centroid cj. the second term with the negative sign quantifies the average dissimilarity of the instance from the other centroids, and nonnegative weight. For high quality cluster, instances should be concentrated around the centroid and be far from the other clusters. The second term is referred to as the repulsive term, as its objective is similar the repulsive rule in (Lee et al. 2013). Although conventional methods also try to increase inter-cluster dissimilarity (Ward Jr 1963; Lee et al., 2015), to the best of our knowledge, DRC is the first attempt to use an explicit repulsive term in deep clustering, which jointly optimizes clustering and feature embedding. Next, we measure the overall quality of the clustering by
Exec, (
(
(6) We aim to find the optimum clusters maximize this objective function J, yet finding the global optimum is comoleeet (Kleinberg et al., [1988, Garey et al.) 1982). Hence, we propose an iterative algorithm, called DRC, to find local optimum, in the k- k-means algorithm (Gersho & Gray 1991).
J(CC,) (3 Eÿ Exec, (1h)))ss - (1t)(
1. Centroid rule: After fixing the clusters {C, - we update the centroids { max- imize J in Eq. (6). Because the centroids should lie on the unit sphere, we solve the constrained optimization problem:
maximize (((r)))m1) subject to cfcy= =1 for all I,...,k
Using Lagrangian multipliers (Bertsekas) 1996), the optimal centroids are obtained as Cj = Exec, hi - Exexie //al Exec, 1 a - Exexic, Mall- (8)
2. NN rule: On the other hand, after fixing the centroids, we update the membership of each instance to maximize J in Eq. (6). The optimal cluster Cj is given by
C2 = { (he 2 (hir)' for all 1 < <(Sk) k}. In other words, an instance should be assigned to Cj if its nearest centroid is Cj.
We apply the centroid rule and the NN rule iteratively until convergence. Because both rules mono- tonically increase the same objective function J and the inequality J S + x always holds, J is guaranteed to converge to local maximum. Readers interested in the convergence are referred
to (Sabin & Gray 1986. Pollard 1982).
Without the repulsive term in Eq. (6) (i.e. at = 0), centroid ej in Eq. (8) is updated by
Cy Exsc, ht//ll Evec, h/gll.
(10)
as done in the spherical k-means (Dhillon & Modha 2001]. In contrast, with a positive a, the objective function / is reduced when the centroids are far from one another. Idcally, in equilibrium, the centroid of a cluster should be the opposite of the centroid of all the other clusters;
Exccy Pia [Exec Aaal
(Fiierree Deexre, Aia [EZenn,, Aill
= --1 for all j ..,
(1I)
Note that the ORID network and thus the encoded feature space are trained jointly with the repulsive clustering. As the training goes on, the centroids repel one another, and the clusters are separated more clearly due to the repulsive term.
We jointly optimize the clusters and the ORID network parameters, as described in Algorithm I First, we train the ORID network for warm-up epochs, by employing every pair of instances and as input. Then, using the identity features, we partition the input data into clusters using k-means. Second, repeat the fine-tuning of the ORID network and the repulsive clustering alternately. In the fine-tuning, a pair of x and are constrained to be from the same cluster, and the following loss function is employed.
A + Aetafetu + Acom + Agan Eaan-
(12)
Appendix B describbs this loss function in detail, proves the optimality of the centroid and NN rules Eq. g and and analyzes the impacts of the repulsive term in Eq. (6).
5

---

Published as a conference paper at ICLR 2021
Algorithm DRC-ORID
Input: Ordered data X' {21, #2,.... ................................ Fn h the number of clusters
1: Train ORID network for warm-up epochs to minimize loss Awolle + Acom coom + Agan &aan
2: Partition X' into C2,C2.... Cs using k-means
.22.....C,,
3: repeat
4: Fine-tune ORID network to minimize loss rcoofoo + Astulola + Aoomecom + Agan
5: repeat
., do 6: for all j 1,,,....
7:
Update centroid Cj via Eq. 8: end for
(8
> Centroid rule
for all j= 1, 2, k do Update cluster C via Eq. 11: end for
9:
D. NN rule
12: until convergence or predefined number of iterations
13:
until predefined number of epochs
Output:
Clusters {C centroids {cz1ý=1. ORID network
3.4 RANK ESTIMATION
Using the output of the DRC-ORID algorithm, we can estimate the rank of unseen instance x. First, we extract its identity feature hi using the ORID encoder. By comparing le with the cen- troids {er based on the NN rule, we find the most similar centroid qq Then, x is declared to belong to cluster C, Without loss of generality, let us assume that the classes (or ranks) are the first m natural numbers, θ = {1,2, 2.... m.. Then, for each E O, we select reference instance with rank from cluster C,, so that it the most similar x. Specifically.
1/1 arg max EC, (())= (hg) "
(13)
We estimate the rank ((x) of the test instance z, by comparing it with the chosen references yi+ S & m. For the rank estimation, Lim et al. (2020) developed the maximum consistency rule, which however does not exploit the probability information, generated by the comparator. In this paper, we use the maximum a posteriori (MAP) estimation rule, which is described in detail in Appendix B0
EXPERIMENTAL RESULTS
This section provides various experimental results. Due space limitation, implementation details and more results are available in Appendices C and
4.1 FACIAL AGE ESTIMATION
Datasets: We use two datasets. First, MORPH (Ricanek & Tesafaye 2006) is collection of about 55,000 facial images in the age range [16, 77.. It provides gender (female, male) and race (African American, Asian, Caucasian, Hispanic) labels as well. We employ the four evaluation settings A, B, C, and D in Appendix C.2 Second, the balanced dataset (Lim et al.) (2020) is sampled from the three datasets of MORPH II, AFAD (Niu et al.) 2016), and UTK (Zhang et al. 2017) to overcome bias to specific ethnic groups genders. It contains about 6,000 images for each combination of gender in (femalc, male) and ethnic group in [African, Asian, European].
Figure (aaree]sshwwe shows clustering results on MORPH II (setting A), when the number of clus- ters is k = Setting A contains faces of Caucasian descent only. Thus, the proposed DRC-ORID divides those faces into two clusters according to genders in general, although the annotated gen- der information is not used. Most males are assigned to cluster 1, while majority of females to cluster 2. On the other hand, setting B consists of Africans and Caucasians. Thus, those images are clustered according to the races, as shown in Appendix C.3 Figure 4 is the results on the balanced dataset at A: 3, which is composed of MORPH II, AFAD, and UTK images. Due to different characteristics of these sources, images are clearly divided according to their sources. At & = 2, MORPH II images separated from the others. This is because. unlike the MORPH II images, the boundaries of most AFAD and UTK images are zeroed for alignment using SectaFaceEngine (Zhang al. (014)).
Clustering:

---

Published as a conference paper at ICLR 2021
Cluster
Claster
Figure 3: MORPH II images in setting A are divided into two clusters.
Cluster
Cluster
Cluster 3
Chibae
MEAD
Choberr
N00011 UTK
Cloter:
10s
20s
30s
40s
50s
60s
70s
Figure 4: The balanced dataset is divided into three clusters, which are mostly composed of AFAD, UTK, and MORPH II images, respectively.
Table 1: Comparison of age estimation results on MORPH II. Here, * means that the algorithm is
pre-trained on the IMDB-WIK dataset (Rothe et al. 2018
Setting A
Setting
Settng
Setting
Algorithm
MAE
CS (%)
MAB
CS (%)
MAE
(S6
MAE
CS (%)
DRF's (Shen
2018
2.91
82.9
2.98
2.17
91.3
MO-CNN (Tan et 2017
2.52
85.0
2.70
MV (Pan al 2018
2.80
87.0
2.41
90.0
MV
(Pan ci al 2018
2.79
2.16
BridgeNet [L] cl al. 2019
2.38
91.0
2.63
86.0
AVDL" WWen of al. 2020
2.37
2.53
1.94
Lim
2020
2.41
91.7
2.75
88.2
2.68
88.8
2.22
93.3
Proposed- Vanilla (k
3.36
80.1
3.32
80.1
3.40
79.7
2.99
84.9
Proposed-VCE VGG (k 1)"
2.35
92.4
2.64
88.9
2.64
89.0
2.22
93.3
Proposed-VGr VGG (k:
2.26
93.8
2.51
89.7
2.58
89.5
2.16
93.5
Lim et al. 2020 also tried the clustering of the balanced dataset. Figure [ visualizes the feature space using I-SNE (Maaten & Hinton| 2008). Although their method aligns the features according to ages, their clusters are not separated, overlapping one another. In contrast, the proposed DRC-ORIN separates the three clusters clearly, as well as sorts features according the ages within each cluster. More t-SNE plots for analyzing the impacts of the repulsive term are available in Appendix B.5
Age transformation: We assess the decomposition performance of ORID. Although ORID is not designed for age transformation (Or-El al.) 2020), it decomposes an image x into the order and identity features, her and h Thus, the age can be transformed in two steps. First, we replace he of with hy of a reference image y at a target age. Second, we decode the resultant feature (concatenation of hy and hia)) to obtain the transformed image. Figure 6 shows some results MORPH II images. Order-related properties, such as skin textures and hair colors, are modified plausibly, but identity information is preserved. This indicates the reliability of ORID.
Table D compares the proposed algorithm with conventional age estimators on the four evaluation settings of MORPH II. These conventional algorithms take 224 x 224 or bigger im- ages as input, while ORID takes 64 64 images. Moreover, most of them adopt VGG16 (Simonyan & Zisserman, 2015) as their backbones, which is more complicated than the ORID encoder. Thus, for comparison, after fixing clusters using DRC-ORID, we train another pairwise comparator based on VGG16, whose architecture is the same as Lim et al. (2020). We measure the age estimation performance by the mean absolute error (MAF) and the cunulative score (CS). MAE is the average absolute error between estimated and ground-truth ages, and CS computes the percentage of test samples whose absolute errors are less than or equal to tolerance level of 5.
Age estimation:
Mainly due to the smaller input size of 64 x 64, the vanilla version yields poorer performances than the conventional algorithms. The VGG version, however, outperforms them significantly. First, be compared, as in the OL algorithm. In other
in the Soaposed-VGG (k 1), all instances words, the clustering is not performed. Thus, the pairwise comparators of OL and the proposed- VGG (k- 1) are trained in the same way, but their rank estimation rules different. Whereas
7

---

Published as a conference paper at ICLR 2021
10c
Cluster
Chater
20s
Cluster
38s
Cluster
40s
16
Claor
Clider
50s
60s
(a) OL (Lim et...
(b) DRC-ORID
70s
Figure 5: I-SNE visualization of the feature spaces of the balanced dataset at k 3.
GOL @ Mi
X
g
AtM @ %5
21
21
21-50
50
42-19
19
21
21-21
21
Age transformation results. For each test, the input 2, reconstruction & G
Figure 6:
F(x),
transformed result G(hy, @ h and reference y shown, where + denotes concatenation.
Cluster
Cluster 2
Cluster's 3
Cluster 4
Chaster5
Chaster
Cluster
Claster
Figure Example AADB images grouped into eight clusters (A:
8).
[0, 0.1)
Cluster
Cluster
Chustar8 8
[0.1, 0.2)
Cluster 2
[0.5, 0.6
10.0.1)
ClusterS
[0.2, 0.3)
Cluster
[0.6, Q.)
[0.3, 0.4)
Cluster 4
[0.7. 0.8
001,
C
[0.4 0.5)
Cluster
[0.5, 0.6)
Cluster6
[088
[0.6, 0.7)
Cluster
102 0.0
[0.9 1.0]
C
[0.7, 0.8)
Cluster
(0.3. 0.4)
[0.8, 0.9)
[04, 055)
[0.9, 1.0]
Figure 9: t-SNE visualization of feature space of AADB at k 8.
OL uses the maximum consistency rule, the proposed algorithm performs the MAP estimation. The score gaps between them confirm that the MAP estimation is more accurate. Moreover, by clustering facial images into two groups, the proposed- DGGG proposed- VGG (A: = 2) provides the bate-of-theear results, except for the MAE test in setting D.
2) improves the results meaningfully. The
4.2 AESTHETIC SCORE REGRESSION
The aesthetics and attribute database (AADB) is composed of 10,000 photographs of various themes such as scenery and close-up (Kong et al.) 2016). Each image is annotated with an aesthetic score in [0, 1]. We quantize the continuous score with step size of 0.01 to make 101 score classes. Compared to facial images, AADB contains more diverse data. It is hence more challenging to Images in the
cluster AADB images. Figure 8 shows example images in each cluster at A same cluster have similar colors, similar contents, or similar composition. This means that ORID extracts identity features effectively, corresponding contents or styles that are not directly related to aesthetic scores. Using those identity features, DRC discovers meaningful clusters. Figure 9 visualizes the feature space AADB. Aesthetic scores are sorted along one direction, while clusters separated in the other orthogonal direction. In other words, the scores look like latitudes, while the clusters appear to be separated by meridians (or lines of longitude). As a point on the earth surface can be located by its latitude and longitude, image is represented by its aesthetic score (order feature) and cluster (identity feature).
Table 2 compares regression results. Even without clustering process, the proposed algorithm out- performs the Reg-Net and ASM algorithms. Moreover, by using the eight unsupervised clusters in Figure the proposed algorithm further reduces the MAE yield the state-of-the-ar result.
8

---

Published as a conference paper at ICLR 2021
Table 2: Aesthetic score regression performances of the proposed algorithm and the conventional Reg-Net (Kong et al. 2016) and ASM (Lee & Kim 2019) on the AADB dataset.
Algurichm
Reg Net
ASM
Proposed (K- 1)
Proposed
MAE
0.1268
0.1141
0.1109
0.1056
Clustar
Chucer
Table 3: Comparison of classification perfor- mances on the HCI dataset.
Model
Accuracy (8)
MAE (docade)
Chuster
Cluster 4
Frank & Hall
2001
41.36
0.99
Cardaso
CONE
21007
0.95
Palermo 21 PL 2012
44,92
0.93
REDESV M
Tau &
2012
35.92
0.96
ORCNN Nht
2015
44.67
CNNPOIR CL 2015
50.12
0.82
Figure 7: Example HCI images grouped into four clusters (I: = 4).
GP-DNNOR [Lu al 2019
46.60
0.76
Proposed
44.72
4.3 HISTORICAL COLOR IMAGE CLASSIFICATION
HCI (Palermo et al., 2012) is a dataset for determining the decade when a photograph was taken. It contains images from five decades from 1930s to 1970s. Each decade category has 265 images: 210, 5, and 50 are used for training, validation and testing. 7 shows the clustering results at k == 4. We observe similarity of contents in each cluster. Table comppare the quinary classification results. FFank & Hall (2001), Cardoso & da Costal (2007), Palermo et al. (2012), and RED-SVM use traditional features, while the others deep features. The performance gaps between these two approaches are not huge, since 1,050 images are insufficient for training deep networks.
5 IMPACTS OF APPLICATIONS
The proposed algorithm can be applied to various ranking problems. In this paper, we demonstrated three vision applications: facial age estimation, aesthetic score regression, and historical image classification. In particular, the proposed age estimator has various potential uses. For example, it can block or recommend media contents to people according to their ages. However, it has harmful impacts, well as positive ones. Moreover, although age information lacks the distinctiveness to identify an individual, identity features, extracted by ORID, can be misused in facial recognition systems, causing serious problems such as unwanted invasion of privacy (Raji al. 2020). Hence ethical considerations should be made before the use of the proposed algorithm.
Recently, ethical concerns about the faimess and safety of automated systems have been raised Castelvecchi] [0200] Roussi| 22200. Noorden 2020). Especially, due to the intrinsic imbalance of facial datasets (Ricanek & Tesafaye [0066] Zhang al. 2017, NNu et al. 2016), most deep learning methods on facial analysis (Wen et al., 2020, Er-El et al.) (0200 have unwanted gender or racial bias. The proposed algorithm is not free from this bias either. Hence, before any practical usage, the bias should be resolved. Also, even though the proposed algorithm groups data in an unsupervised manner, data are clustered according to genders or races on MORPH II. These results should never be misinterpreted in such a way as to encourage any racial or gender discrimination. We recommend using the proposed age estimator for research only.
6 CONCLUSIONS
The DRC algorithm of ordered data based on ORID was proposed in this work. First, the ORID network decomposes the information of object into the order and identity features. Then, DRC groups objects into clusters using their identity features in repulsive manner. Also, we can estimate the rank of an unseen test by comparing it with references within the corresponding cluster based on the MAP decision. Extensive experimental results on various ordered data demonstrated that the proposed algorithm provides excellent clustering and rank estimation performances.
ACKNOWLEDGMENNS
This work was supported in part by the MSIT, Korea, under the ITRC support program (IITP-000-- 2016-0-00464) supervised by the IIT,, and in part by the National Research Foundation of Korea (NRF) through the Korea Government (MSIP) under Grant NRF-201 8R1 A2B3003896.

---

Published as a conference paper at ICLR 2021
REFERENCES
Dimitri P. Bertsekas. Constrained Optimization and Lagrange Multiplier Methods. Academic Press, 1996.
Rainer Brüggemann, Peter Sørensen, Dorte Lerche, and Lars Carlsen. Estimation averaged ranks by local partial order model. Journal of Chemical Information and Computer Sciences, 44(2):618-623 2004.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hul- lender. Learning to rank using gradient descent. In ICML, 2005.
Jaime S. Cardoso and Joaquim F. Pinto da Costa. Learning to classify ordinal data: The data repli- cation method. Journal of Machine Learning Research, 1:333-11429 2007.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In ECCV, 2018.
Davide Castelvecchi. Is facial recognition too biased to be let loose? Nature, 587(7834):347-349 2020.
Jianlong Chang, Lingfeng Wang. Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep adap- tive image clustering. In ICCV, 2017.
Dongdong Chen, Jiancheng Lv, and Zhang Yi. Unsupervised multi-manifold clustering by learning deep representation. In AAAI Workshops, 2017.
William W. Cohen, Robert E. Schapire, and Yoram Singer. Learning to order things. In NIPS, 1998.
Hal Daumé III, Nikos Karampatziakis, John Langford, and Paul Mineiro. Logarithmic time one- against-some. In ICML., 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Inderjit S. Dhillon and Dharmendra S. Modha. Concept decompositions for large sparse text data using clustering. Machine Learning, 42(1-2):143-1777 2001.
Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In ICCV, 2017.
Cynthia Dwork, Ravi Kumar, Moni Naor, and Dandapani Sivakumar. Rank aggregation methods for the web. In The WebConf,, 2001.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In KDD, 1996.
Eibe Frank and Mark Hall. A simple approach to ordinal classification. In ECML-PKDD, 2001.
Yun Fu and Thomas S. Huang. Human age estimation with regression on discriminative aging manifold. IEEE Trans. Multimedia, 10(4):578-584, 2008
M. R. Garey, D. S. Johnson, and Hans S. Witsenhausen. The complexity of the generalized Lloyd- Max problem. IEEE Trans. Inf. 28(2)255525 1982.
A. Gersho and R. M. Gray. Vector Quantization and Signal Compression. Kluwer Academic Pub- lishers Norwell, 1991.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
Ian J. Goodfellow, Jean Pouget-Abadice Mehdi Mirza, Bing Xu, David Harde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
10

---

Published as a conference paper at ICLR 2021
J. A. Hartigan and M. A. Wong. A K-means clustering algorithm. Journal of the Royal Statistical Sociery, Series C (Applied Statistics), 2801):100-108, 1979.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards definition of disentangled representations. In arXiv preprint arXiv:/812.02230,, 2018.
Thomas Hofmann, Berhaard Schölkopf, and Alexander J. Smola. Kernel methods in machine learn- ing. The Annals of Statistics, pp. 1171-1220, 2008.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Pan Ji, Tong Zhang, Hongdong Li, Mathieu Salzmann, and lan Reid. Deep subspace clustering networks. In NIPS, 2017.
Jon Kleinberg. Christos Papadimitriou, and Prabhakar Raghavan. A microeconomic view of data mining. Data Mining and Knowledge Discovery, 2(4):311 1-244, 1998.
Teuvo Kohonen. The self-organizing map. Proceedings of the IEEE, 7809)1464-1480, 1990.
Shu Kong, Xiaohui Shen, Zhe Lin, Radomir Mech, and Charless Fowlkes. Photo aesthetics ranking network with attributes and content adaptation. In ECCV, 2016.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436-444 2015.
Chulwoo Lee, Won-Dong Jang. Jae- Young Sim, and Chang-Su Kim. Multiple random walkers and their application to image cosegmentation. In CVPR, 2015.
Jun-Tae Lee and Chang-Su Kim. Image aesthetic assessment based on pairwise comparison - unified approach to score regression, binary classification, and personalization. In ICCV, 2019.
Wanhua Li, Jiwen Lu, Jianjiang Feng, Chunjing Xu, Jie Zhou, and Qi Tian. BridgeNet: A continuity- aware probabilistic network for age estimation. In CVPR, 2019.
Kyungsun Lim, Nyeong-Ho Shin, Young- Yoon Lee, and Chang-Su Kim. Order learning and its application to age estimation. In ICLR, 2020.
Hsuan- Tien Lin and Ling Li. Reduction from cost-sensitive ordinal ranking to weighted binary classification. Neural Compulation, 24(5):1329-136) 2012.
Tie-Yat Yan Liu. Learning to rank for information retrieval. Foundations and Trends in Information Retrieval, 3(3):225-333 331,,,,,2000 2009.
Yanzhu Liu, Adams Wai Kin Kong, and Chi Keong Goh. A constrained deep neural network for ordinal regression. In CVPR, 2018.
Yanzhu Liu, Fan Wang, and Adams Wai Kin Kong. Probabilistic deep ordinal regression based on Gaussian processes. In ICCV, 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579-2605, 2008.
Andrew Y. Ng, Michael I. Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. In NIPS, 2002.
Zhenxing Niu, Mo Zhou, Le Wang. Xinbo Gao, and Gang Hua. Ordinal regression with multiple output CNN for age estimation. In CVPR, 2016.
Richard Van Noorden. The ethical questions that haunt facial-recognition research. Nature, 587 (7834))354-358, 2020.
Roy Or-El, Soumyadip Sengupta, Ohad Fried, Eli Shechtman, and Ira Kemelmacher Shanhherrenaiiimaa.... Lifespan age transformation synthesis. In ECCV, 2020.

---

Published as a conference paper at ICLR 2021
Frank Palermo, James Hays, and Alexei A. Efros. Dating historical color images. In ECCV, 2012.
Hongyu Pan, Hu Han, Shiguang Shan, and Xilin Chen. Vean-variance loss for deep age estimation from face. In CVPR, 2018.
Duc Truong Pham, Stefan S. Dimov, and Chi D. Nguyen. Selection of k in k-means clustering. Proc. Inst. Mech. Eng. C, 219 2005.
David Pollard. Quantization and the method of k-means. IEEE Trans. Inf. Theory, 28(2):199-22) 1982.
Inioluwa Deborah Raji, Timnit Gebru, Margaret Mitchell, Joy Buolamwini, Joonseok Lee, and Emily Denton. Saving face: Investigating the ethical concerns of facial recognition auditing. In AAAI/ACM Conference on AI, Ethics, and Society, 2020.
Douglas Reynolds. Gaussian mixture models. Encyclopedia Biometrics, 741, 2009.
Karl Ricanek and Tamirat Tesafaye. MORPH: longitudinal image database of normal adult age- progression. In FGR, 2006.
Rasmus Rothe, Radu Timofte, and Luc Van Gool. Deep expectation of real and apparent age from single image without facial landmarks. Int. J. Comput. Vis., 126(2-4):1444-17, 2018.
Antoaneta Roussi. Resisting the rise of facial recognition. Nature, 587(7884433300-3355 2020.
Thomas L. Saaty. A scaling method for priorities in hierarchical structures. Journal of Mathematical Psychology, 15(3):234 281, 1977.
Michael J. Sabin and Robert M. Gray. Global convergence and empirical consistency of the gener- alized Lloyd algorithm. IEEE Trans. Inf. Theory, 3282):148-155 1986.
Bernd S. W. Schröder. Ordered Sets: An Introduction. Springer, 2003.
Wei Shen, Yilu Guo, Yan Wang, Kai Zhao, Bo Wang, and Alan Yuille. Deep regression forests for age estimation. In CVPR, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. In /CML, 2015.
Alexander Strehl and Joydeep Ghosh. Cluster ensembles a knowledge reuse framework for com- bining multiple partitions. Journal of Machine Learning Research, 3:583-61 -:583-617, 2002.
Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
Zichang Tan, Jun Wan, Zhen Lei, Ruicong Zhi, Guodong Guo, and Stan Z. Li. Efficient group-n encoding and decoding for facial age estimation. IEEE Trans. Pattern Anal. Mach. Intell., 40(11); 2610-2623, 2017.
Ming-Feng Tsai, Tie-Yar Liu, Tao Qin, Hsin-Hsi Chen, and Wei-Ying Ma. FRank: A ranking method with fidelity loss. In SIGIR, 2007.
Joe H. Ward Jr. Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58(301):236-25 1963.
Xin Wen, Biying Li, Haiyun Guo, Zhiwei Liu, Guosheng Hu, Ming Tang, and Jinqiao Wang. Adap- tive variance based label distribution learning for facial age estimation. In ECCV, 2020.
Svante Wold. Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and Intelligent Laboratory Systems, 2(1-3):37-52, 1987.
Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu. HD-CNN: Hierarchical deep convolutional neural networks for large scale visual recognition. In ICCV, 2015.
12

---

Published as a conference paper at ICLR 2021
Bo Yang, Xiao Fu, Nicholas D. Sidiropoulos, and Mingyi Hong. Towards k-means k-means-friendly Simultameous deep learning and clustering. JCML, 2017.
friendly spaces:
Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In CVPR, 2016.
Jie Zhang, Shiguang Shan, Meina Kan, and Xilin Chen. Coarse-to-fine auto-encoder networks (CFAN) for real-time face alignment. In ECCV, 2014.
Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/rrerresiinn by conditional adversarial autoencoder. CVPR, 2017.
13

---

Published as a conference paper at ICLR 2021
A NETWORK STRUCTURE OF ORID
As described in Section 3.2 the ORID network consists of the encoder F, the decoder G, the com- parator C. and the discriminator D. The network structures of these components detailed in 4- where "Л --C Conv' and "k -sa Deconv' denote the 2D convolution and 2D deconvolution with kernel size ky xkw, stride s, and e ouiput channels, respectively. 'BN' means batch normalization (loffe & | (2015), and 'c Dense' is a dense layer with output channels. Note that the encoder takes 64 64 RGB image as input, and the identity feature of the encoder output is nz-romaleieed in Eq.
Also, we set dor 128 and dia 896.
Table 4: The encoder F the ORID network.
Layers
Output
4x4-2-64 Conv BN LeakyRel.U
32x32x 64
4x4-1-64 Conv BN LeakyReI.U
32x32
3x3-2-128 Conv BN LeakyRel.U
16x 16x 128
3x3-1-128 Conv BN LeakyReL.U
16x 16x 128
3x3-2-256 Conv BN LeakyReLU
8x8x256
3x3-1-256 Conv BN LeakyReLU
8x8x256
3x3-2-512 Conv BN LeakyReLU
4x4x512
3x3-1-512 Conv BN LeakyReLU
4x4x512
3x3-2-512 Conv BN LeakyReLU
2x2x512
3x3-1-512 Conv BN LeakyReLU
2x2x512
2x2-2-1024 Conv BN LeakyReLU
1x1x
Table 5: The decoder G in the ORID network.
Layers
Output
2x2-2-1024 Deconv BN Dropout(0.3) LeakyReLU
2x2x 1024
3x3-2-512 Deconv BN (O03out(O.3) LeakyReL.U
4x4x512
4x4-2-512 Deconv BN LeakyReI.U
8x8x512
4x4-2-256 Deconv BN LeakyReLU
16x26
4x4-2-128 Deconv LeakyReLU
32x32x128
4x4-2-3 Deconv Tanh
64x64x3
Table The comparator C' in the ORID network.
Layers
Output
256 Dense BN LeakyReLU
256
256 Dense BN LeakyReLU
256
(r05
256
3 Dense Sofımax
Table 7: The discriminator D in the ORID network.
Layers
Output
5x5-2-64 Conv BN LeakyReLU Dropout(0.5)
32x32x64
32x32 64
5x.5-2-128 Conv BN LeakyReLLU
16x 16x 128
5x.5-2- 128 Conv BN LeakyyReeL Dropout(0.3)
8x8x 128
8x8x 128
Reshape
8192
1x1-1-1 Conv
14

---

Published as a conference paper at ICLR 2021
B ALGORITHMS DETAILS
B.1 OPTIMALITY OF CENTROID RULE
To solve the constrained optimization problem in Eq. 77. we construct the Lagrangian function
L= Ei=IE ((h a - Eie (1aa)' ~ (e- - 1)
(14)
where ,, S S k, are Lagrangian multipliers (Bertsekas) 1996). respect to Cj and setting it to zero, we have
By differentiating with
aL = Evec, hid k- Eijj Ezec, hid 2Xy0j de;
(15)
Evec, hig - 1 EmExie, hgs 22,0z
(16)
(17)
for j = 1....,k, ..... Therefore, the optimal centroid is given by
Cj Exec, h -a-- L Emeric haa
(18)
2)j
Because of the normalization constraint '/'c = 1, we have
21, Euec, hha ax= ExErc hiall:
(19)
which leads to the centroid rule in Eq. (
B.2 OPTIMALITY OF NN RULE
Let us consider two cases. First, instance x declared belong cluster Cj. It then contributes to the objective function J in Eq. (6) by
B, (h a
a Ei43 (hia)'
(20)
Second, æ is declared to belong to another cluster Cjr. Then, its contribution is
By (haa) arth Eiey
(21)
By comparing the two contributions, we have
Bg 3y- (1aa) -c) - ( (1g) -ci)
(22)
( -cp)).
(23)
This means that B, By when (e 2 (ha))' Therefore, should be assigned to the optimal cluster C,, such that the cosine similarity (h " ej. is maximized. Equivalently, we have the NN rule in Eq.
B.3 REGULARIZATION CONSTRAINT IN DRC
To prevent empty clusters and balance the partitioning, we enforce a regularization constraint so that every cluster contains least predefined number of instances. More specifically, when applying the NN rule, we enforce that at least 2F of instances are assigned to each cluster Cj. The instances are selected the decreasing order of cosine similarity (ha))
B.4 Loss FUNCTIONS
In the DRC-ORID algorithm, we use the loss function
8 Areefre Aetaketu Aammkaom Agan Gaangaa
(24)
15

---

Published as a conference paper at ICLR 2021
where the reconstruction, clustering, comparator, and GAN losses given by
free 2N Di (1|=; - G(F(x)))]l + Ilu G(F(G)))li),
(25)
fetu = 2N Ei=l (( (k)))
(26)
Room
EE (g logp> " 9= log Z. - log logp=" ")
(27)
ggan = 2N Ei (log(1 - D(G((((:))) + log(1 D(G(F(())))))
(28)
respectively. Here, N is the number of image pairs in minibatch. The weighting parameters are set to Area= = Actu 0.1, Aaom
1, and Agan 1.
B.5 IMPACTS REPULSIVE TERM ON CLUSTERING
To analyze the impacts of the repulsive term in Eq. (6). we first compare clustering qualities with = and a 0.1. At a 0, the repulsive term is excluded from the objective function J and the centroid rule is reduced to Eq. (TO) in the spherical m-means (Dhillon & Modhal 200T). However, different from the spherical k-means, even at a = 0, the clustering is jointly performed with the training of the ORID network.
We adopt two metrics to measure the quality of clustering: normalized mutual information (NMI) (Strehl & Ghosh| 2002) and centroid affinity (CA). NMI measures the information shared between two different partitioning the same data A 1 A, and B = Bj,,
V j- |A ByI log MAA.B Σ Σj
NMI(A, V |Ail log
(29)
AAn N |2) [B} log [Bd
where U and V are the numbers of clusters in A and B, respectively, N is the total number of samples, and | denotes the cardinality. Also, we define the centroid affinity (CA) as
CA((c)) स|ं - Gfe--
(30)
For high-quality clustering, the centroids should be far from one another and thus should yield a low CA score.
Figure plots how NMI and CA vary as the iteration goes on. In this test, MORPH II (setting B) is used and the number of clusters is is set to 2. Since setting B consists of Africans and Caucasians, use the race groups as the ground- the NMI score of DRC-ORID with partitioning for the NMI measurement. At early iterations, 0.1 is slightly better than that with a However, the iterative training and clustering go on, the score gap gets larger. After the convergence, DRC-ORID with a 0.1 outperforms the option a 0 by a significant NMI gap of 0.13. Also, CA of the option = 0.1 gradually decreases, whereas that of a = O does not. At a 0.1, the repulsive term makes the centroids repel each other. As result, CA, which is the cosine similarity between the two centroids, becomes almost -1 1, which means the equilibrium state in Eq. (II) is almost achieved.
0.9
0.5
NMI
3
-0.5
0.1
0.1
=
0.3
10
30
No. of iterations
No. of iterations
(b)
Figure 10: Comparison of (a) NMI and (b) CA curves with and without the repulsive term.
16

---

Published as a conference paper at ICLR 2021
Cluster
Cluster 2
(2 0.7
Iteration 4
Iteration 10
Iteration 20
Iteration 30
Iteration 40
Figure 11: Comparison of the feature space transition MORPH II (setting B) a and
0.1.
10s
Cluster
306
40s
50s
olc
(a)
(b) a 0.1
70s
Figure 12: t-SNE visualization of the feature spaces of MORPH (setting B) with age labels after the convergence.
We also visualize the feature spaces of the two options, a = 0 and a = 0.1, using t-SNE in It is observed that two clusters are more clearly separated by DRC-ORID with a 0.1. Figure 12 shows the t-SNE results after the convergence with age labels.
Figure 11
Figure compares the NMI curves different a's. The choice a affects the quality of clus- tering, controls the intensity of the repulsive force between centroids. When is too large, the centroids move too quickly, making the training of the ORID network difficult. On the other hand, when is too small, the repulsive term does not affect the clustering meaningfully. Hence, should selected to strike a balance between training reliability and effective repulsion. was found experimentally that clustering is performed well around 0.1.
0.9
0.05
NMI
0.1
0.6
0.15
0.2
0.3
10
20
30
No. of iterations
Figure 13: Comparison of NMI curves at different a's.
Finally, is worth pointing out that, if the identity features were not normalized as in Eq. 3 and the repulsive clustering were performed in unbounded space, the distances between centroids would get larger and larger the iteration goes Thus, convergence would not be achieved. This is why we perform DRC on the bounded unit sphere.
17

---

Published as a conference paper at ICLR 2021
B.6 IMPACTS REPULSIVE TERM ON RANK ESTIMATION
Table 8 compares the rank estimation results when the clustering is performed with and without the repulsive term. In this experiment, the repulsive term, lower-quality clusters make the training of the comparator more difficult. As result, the age estimation performance degrades significantly in terms of both MAE and CS. In other words, the quality of clustering affects the rank estimation performance greatly, and the proposed DRC algorithm provides high quality clusters suitable for the rank estimation.
MORPH II (setting A) and set k
Without
Table 8: Comparison of age estimation results on MORPH II (setting A) when clustering is per- formed with and without the repulsive term.
Algorithm
MAE
CS (%)
Proposed without repulsive term Proposed with repulsive term
2.47
90.7
2.26
93.8
B.7 IMPACTS OF THE NUMBER A OF CLUSTERS ON RANK ESTIMATION
Tables 9 and 10 compare the rank estimation results according to the number l of clusters on the MORPH II (setting A) and AADB datasets, respectively. On MORPH II, the age estimation perfor- mance decreases as k increases. Since the training set in setting A consists of only 4,394 images, each cluster large l contains too few instances. Thus, the comparator is trained inefficiently with fewer training pairs, degrading the performance. In contrast, AADB contains a large number of diverse images. Due to the diversity, a relatively large k should be used to group images into meaningful clusters. Also, even at large k, each cluster contains a sufficient number of data. Thus, as compared to MORPH II, results on AADB are less sensitive to addition, we provide age estimation results on the balanced dataset in Table 14, in which & has marginal impacts on the rank estimation performance.
As mentioned previously, the quality of clustering significantly affects the rank estimation perfor- mance. Also, similarly to other algorithms based on k--mans, the clustering quality of DRC affected by k. Hence, for the proposed algorithm to be used new ordered dataset, k should be determined effectively to obtain good clustering and rank estimation results. Readers interested in the selection of k are referred Pham et al. (2005).
Table 9: Age estimation results according to k on MORPH II (setting A).
MAE
CS (%)
Algorithm
k ==
1 ==
3
h: ==0
k = 2
k: = 3
= 1
Proposed
2.26
2.32
2.43
93.8
92.7
91.4
Table 10: Aesthetic score regression results according to l on AADB.
Algorithm
6
4
6==6
6
l:
8
10
Proposed 0.1073
0.50
0.1056
0.1060
B.8 CLUSTERING USING OTHER FEATURES
Instead of clustering identity features hid /.g...... 1. we test clustering order features ha 1.2 h or whole features 1tt 1tt,, ","",...,""" 11=". In this test, MORPH II (setting A) is used and t 2. Figure I4 compares the clustering results. When using order features whole features, instances are divided by their ages. We see that instances younger than 30 mostly belong to cluster and the others cluster compparss the performances of the age estimators trained using these clustering results. The best performance is achieved when the clustering is done on identity features.
18

---

Published as a conference paper at ICLR 2021
Cluster
Cluster 2
Female Male
200
100%
10s
20s
30s
40s
50s
60s
10s
20s
30s
40s
50s
60s
10s
20s
30s
40s
50s
60s
(b)
Figure 14: Clustering results using different features: (a) identity features, (b) order features, and
(c) whole features.
Table 11: Age estimation performances when the three clustering results in Figure [4 are used.
Feature type
MAE
CS (%)
Order Whole
2.42
91.5
2.59
90.7
Identity (proposed)
2.26
93.8
B.9 RELIABILITY OF FEATURE DECOMPOSITION
Performing the comparison using order features only does not theoretically guarantee that order- related information is fully excluded from identity features. However, we observed empirically that the decomposition is sufficiently reliable the dimension an identity feature is selected properly. If the dimension is too small, the encoder may lose a significant portion of order-irrelevant infor- mation. On the contrary, if the dimension is too large, the encoder may encode order information redundantly. In our experiments, we use 128 and 896 dimensional vectors for order and identity features (dor = 128 and da 896), and obtain satisfactory decomposition results.
To show that order-related information is excluded from identity features, we compare the accuracies of the comparator (i.e. ternary classifier), when identity features are used instead order features. Specifically, first extract order features and identity features from all instances in MORPH II using the pretrained ORID network. Then, we train two comparators that predict the ordering rela- tionship between two instances z and y: takes the order features h and hy as input and the other takes the identity features hi and hya Table 12 lists the comparator accuracies. We see that the comparator fails to predict ordering relationships from identity features.
Also, Figure 15 is t-SNE visualization of the identity feature spaces with age or cluster labels, which confirms that order-related information excluded effectively from identity features.
Table 12: Comparison of the comparator accuracies when different input features are used.
Order feature
Identity feature
Random guess
MORPH II (setting A) MORPH 1I (setting B)
81.4
43.2
33.3
75.4
40.8
33.3
10s
Cluster 1
20s
Cluster
Caker
40s
50s
60s
aa)
(b)
70s
Figure 15: t-SNE visualization of identity feature spaces of MORPH II with age cluster labels: (a) setting A and (b) setting B.
19

---

Published as a conference paper at ICLR 2021
B.10 MAP ESTIMATION
Let us describe the MAP estimation rule for rank estimation in Section 3.4 Given a test instance J, we select references y; by Eq.
(13). Then, by comparing with ye. the comparator yields the
probability vector pz (p> P dh .P²") for the three cases in Eq. (4) Thus, given ys the
probability of ((x)= = can be written
(r(x)(r I vi) Pz
uy
Pa Vb)
+ P Pa(a) x ye) + + (r(s)( x < v.)). (31)
Suppose that x: > y/- Then, f(x)) f(v)) rr-i>> from Eq. (4) Also, the maximum possible rank is m. We hence assume that ((z) has the uniform distribution between i + r + 1 and m. In other words, Po(cs) (r x>> yi) Uii+T + m,), where U denotes discrete uniform distribution. Similarly, have Pe(a) (r x yi) (i--,i++7) and Pe(e) (r y1) i 1). Then, approximate the a posteriori probability Po(z)(r (r y1..... Vm by averaging those single-referenee inferences in Eq. 33:;
(a(z)(1) 01.... ym)) m E Pa(a) (1
(32)
Finally, obtain the MAP estimate of the rank of xr, which is given by
((x) = arg max (r( y2....
(33)
ree!
20

---

Published as a conference paper at ICLR 2021
C FACIAL AGE ESTIMATION MORE EXPERIMENTS AND DETAILS
C.I. IMPLEMENTATION DETAILS
We initialize the parameters of the ORID network for facial age estimation using the Glorot normal method (Glorot & Bengio 2010). We use the Adam optimizer with a learning rate of 10 and decrease the rate by factor of 0.5 every 50,000 steps. For data augmentation, we do random horizontal flips only. This is because other augmentation schemes, such as brightness or contrast modification, may deform identity information such as skin colors. Also, dor and dia are set to be 128 and 896, respectively. In Eq. (6).
we set a to 0,1 and decrease it to 0.05 after 200 epochs.
C.2 EVALUATION SETTINGS
For evaluation on the MORPH II dataset, we adopt four widely used testing protocols.
Setting A 5,492 images of the Caucasian race are selected and then randomly divided into two non-overlapping parts: 80% for training and 20% for test.
Setting B 21,000 images of Africans and Caucasians straints: the ratio between Africans and Caucasians should 1 : 1, and that between females and males 3. They are split into three disjoint subsets S1, S2, and S3. The training and testing are repeated twice: 1) training on SI, testing on S2 + S3, and 2) train- ing on S2, testing on S1 + S3. The average performance the two experiments is reported.
selected satisfy two
This setting is the 5-fold vross-validation randomly split into five folds, but the same person's 's images should belong to only one fold. The average performance of the five experiments is reported.
Setting C
the entire dataset. Images
Setting D - This is called the 80-20 protocol. Without any constraint, the entire dataset is randomly divided into the training and test with ratio 8 2. Thus, setting similar to one experiment in setting C, but the same person's 's images may belong to both training and test sets.
C.3 CLUSTERING
We provide more clustering results MORPH II. Figure [6] the clustering results on setting B = Since setting consists of Africans and Caucasians, the images are clustered according to the races. Also, Table 13 summarizes the clustering results for settings A. B, and C at k 2. The clustering result on setting D is omitted, since it almost identical with that on setting C. In all settings, the proposed DRC-ORID divides facial images into two clusters with meaningful criteria, which are gender for setting and race for settings B, C, and D.
Cluster
Cluster
Clicker 2 Chalkur
Cauesike
10s
20s
30s
40s
50s
60s
Figure 16: MORPH II images in setting divided into clusters.
Table 13: Clustering results settings A, B, and C of MORPH II at =
2.
Setting A
Setting B
Setting C
Cluster
Male
Female
Caucasian
African
Caucasian
African
Hispanic
Asian
Others
897
3,507
63
33,008
46
2
209
3,152
26
3,404
8,427
1,065
50
1,384
C.4
AGE ESTIMATION
We implement a VGG-based pairwise comparator and follow the settings of Lim et al. (2020). Specifically, instead of Eq. 4 the ternary categorization based the geometric ratio and
21

---

Published as a conference paper at ICLR 2021
set 0.1. We initialize its feature extractor using VGG16 pre-trained the ILSVRC2012 dataset (Deng et al, 2009) and its fully connected layers using the Glorot normal method. We employ the Adam optimizer with minibatch size of 32. We start with learning rate of 10 and shrink it by factor of 0.5 after every 80,000 steps.
Table 14: Comparison of age estimation results on the balanced dataset.
MAE
CS (%)
Algorithm
k
k
k:
k 3
MV (Pan et 2018
4.52
4.44
4.40
70.1
70.3
69.6
OL-supervised al. 2020
4.23
4.18
4.19
4.18
73.2
73.4
73.4
74.0
OL-unsupervised Lim 2020
4.16
4.17
4.16
74.0
73.9
74.0
Proposed
4.15
4.01
4.02
4.05
74.0
74.8
74.8
74.6
Table 14 lists age estimation results on the balanced dataset according to the number l of clusters. Ou-supeee trains the comparator using supervised clusters separated according to gender or eth- nic group annotations. Specifically, the supervised clusters at k 2, and 6 are divided according to genders, ethnic groups, and both genders and ethnic groups, respectively. On the other hand, uL-unsupervised and the proposed algorithm determine their clusters in unsupervised manners. We see that the proposed algorithm performs better than the conventional algorithms in all tests. By employing multiple clusters, the proposed algorithm improves MAE by 0.12 and CS by 0.73% on average. In contrast, OL OL-unsupervised improves MAE by 0.04 and CS by 0.07% only. This indi- cates that, by employing identity features, the proposed DRC-ORIA algorithm groups instances into meaningful clusters, in which instance ranks can be compared more accurately.
C.5 AGE TRANSFORMATION
Note that, in Figure given an image x. we select the reference y at target age, whose identity feature is the most similar that of æ, in Eq. (3). Hence, the image æ and the reference y have similar appearance. On the other hand, Figure 17 shows transformed images using randomly selected references. The first cases transform the same image x with different references, but the transformed images are similar. Also, even when the gender and/or race of y different from those of ær the identity information of a is preserved well in the transformed image. This confirms the reliability of ORID.
More age transformation results are in Figure 17
y
GOK MS)
y
G(RZ Ki)
y
G0h + k
y
GOA, k
50
50-21
50
50-21
23
23-23
19-19
60ZZ @ MG
y
G(HZ, a MG)
y
G @ kE)
y
G(MZ @ kg)
C
6
19
19
19-119
25
25
25
25
27
27
27-+27
54
18
54-+18
Figure 17: More age transformation results.
C.6 RECONSTRUCTION
Figure 18 shows reconstructed faces using whole feature (h @ l//g), order feature only (ha O), and identity feature only (0 e hgg)) Without the order feature, each decoded face is degraded but the person can be identified. In contrast, without the identity feature, the reconstruction not related to the person except that it seems to an average face of people at the same age as the person. These results confirm that order and identity features complementary.
22

---

Published as a conference paper at ICLR 2021
R
C @ kg OEK, D 0)
X
2
G/0 @ Ag) GOA, @ B)
G(O Ag) GEA, @
18
45
Figure 18: Reconstruction results. For each test, the input æ, reconstruction Go ((x) using the whole feature, reconstruction G(O hifa)) using the identity feature, and reconstruction G(h 0) using the order feature are shown.
C.7 REFERENCE IMAGES
Figure 19 shows examples of reference images, which used for the rank estimation on MORPH II (setting D) at k = 2. Given a test image x, reference image y/ of age class is selected via Eq. (13) from the training However, the top most similar references can be selected and used for the estimation. We use single reference because multiple references improve the estimation performance only negligibly. In Figure 19 the top three reference images are shown for each age from 16 to 53. In setting D, the two clusters are divided by Africans and the others in general. However, we see that and reference images tend to have the same gender, as well as the same race. Furthermore, they have similar appearance, even when they have big age difference.
In the default mode, for each age i, single reference image is selected.
23

---

Published as a conference paper at ICLR 2021
Teat
Rafonoos
Tool
Figure 19: Examples of reference images in facial age estimation.
24

---

Published as a conference paper at ICLR 2021
D AESTHETIC SCORE REGRESSION MORE EXPERIMENTS AND DETAILS
D.1 IMPLEMENTATION DETAILS
For aesthetic score regression, implement pairwise comparator based on EfficientNeiB4 (Tan & LL),2019)
2019). The pairwise comparator has the same architecture as that for facial age estimation, except for the backbone network. To initialize the backbone, we adopt the parameters pre-trained on the ILSVRC2012 dataset. We initialize the other layers using the Glorot normal method. We update the network parameters using the Adam optimizer with minibatch size of We start with learning rate of 10 and shrink it by a factor of 0.8 every 8000 steps. Training images augmented by random horizontal flipping. We set T = 0.15 for the ternary categorization Eq.
D.2 CLUSTERING
Notice that the AADB dataset contains images of diverse contents and styles. Hence, when clus- tering with a small k. it is hard to observe the characteristics shared by images within each cluster, whereas &
is sufficient for facial age data. We empirically found that least eight clusters are required (k 8) to partition the AADB dataset by meaningful criteria. Figure 20 provides more examples of clustering results
8.
Cluster
Cluster 2
Cluster 3
Cluster 4
Cluster
Cluster 6
Cluster'7
Cluster 8
Figure 20: Example AADB images grouped into eight clusters (A: = 8).
25

---

Published as a conference paper at ICLR 2021
D.3 REFERENCE IMAGES
Figure 21 shows examples of reference images, which are used for the aesthetic score regression. Given a test image the reference image Yi of aesthetic class selected by Eq. (13). For the aesthetic score regression, we use a single reference image for each aesthetic class, as done in the facial age estimation. Thus, 101 reference images are used in total.
Test
References
IIT
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.80
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.00
0.96
Test
References
0.00
0.05
0.10
0.1
0.20
0.25
0.30
0.35
0.40
0.45
0.70
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
Test
Refereances
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0,45
0.70
0.50
0.55
0.60
0,65
0.70
0.75
0.80
0.85
0.95
Test
References
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.70
0.55
0.60
0.65
0.75
0.80
0.85
0.90
Test
References
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.60
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
Test
References
0.00
0.06
0.10
0.15
0.20
0.25
0.35
0.45
0.60
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
Figure 21: Examples of reference images in aesthetic score regression.
26

---

Published as a conference paper at ICLR 2021
E HCI CLASSIFICATION MORE EXPERIMENTS AND DETAILS
E.1 IMPLEMENTATION DETAILS
For DRC-ORID for HCI classification, we all hyper-parametten in the same way as we do in Appendix C.T We set = 1 for the ternary categorization of ordering relationship in Eq. 4 Note that there are five decade classes from 1 to 5.
E.2
CLUSTERING
Figure 22 shows some sample images in the HCI dataset, which are ordered according to the decade classes. Figure 23 shows more example HCI images grouped into four clusters (k 4).
1930s
1940s
1950s
1960s
1970s
Figure 22: Example images in the HCI dataset.
Cluster
Cluster
Cluster
Cluster 4
Figure 23: Example HCI images grouped into four clusters (k 4).
E.3 REFERENCE IMAGES
Figure 24 shows the five reference images for each of six test image examples. Note that, given test image, the reference images of similar contents, tones, or composition are selected from the five decade classes.
Test
Reférences
Test
Refereences
1960s
1930s
1940s
1950s
1960s
1970s
1940s
1930s
1940s
1950s
1960s
1970s
Test
References
Test
References
1950s
1930s
1940s
1950s
1960s
1970s
1960s
1930s
1940s
1950s
1960s
1970s
1eest
Neterences
Test
Reterences
1960s
1930s
1940s
1950s
1960s
1970s
1970s
1930s
1940s
1950s
1960s
1970s
Figure 24: Examples of reference images in historical color image classification.
27


Reviews:
[{"review_id": "w8AJ0yjLzIx", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The novelty of the network structure is marginal. The decomposition way of feature is very common in computer vision. Just utilizing the latent vector of the encoder with only the comparator loss to decompose the feature into two feature types is limited. The authors should show the visual differences between these two feature types. The expression of the article is very clear, but some basic theories need not be explained in detail (Such in Section 3.4)\nOne more concern : h_id and h_or are both used for reconstruction. It’s best to prove that only using identity feature h_id is better than the overall latent vector h_id + h_or. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "d1w0ZnY1vag", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- It is well presented. The idea of splitting the encoding feature space into task related features and non-task related features is probably not new. But the use of it in estimating rank might be new and intuitively it makes sense to use it. They also propose an extension to the clustering algorithm using a repulsive term and propose MAP estimation algorithm to assign a rank based on the output probabilities of the comparator when the max possible rank is known.\n- Experiments are conducted on 3 data sets. The results show the effectiveness of the approach. The experiments, I feel, are sufficient to show that clustering instances based on non-rank related features will help improve effectiveness of comparison based ranking of new instances. They also show the effectiveness of their proposed MAP estimation rule for assigning a rank.\n- The effectiveness of the repulsive clustering on ranking performance is not clear. The authors discuss that using the repulsive term in the objective for clustering produces more distinct clusters but how does this \"improved\" cluster quality translate to better performance in ranking? As this is one of the key contributions of the paper, a comparison of ranking performances with and without the use of the repulsive term in clustering would be useful.\n- How sensitive/robust is the proposed approach to the number of clusters chosen? How can one choose the right number of clusters to use? A discussion on these would be useful.\n- In each experiment, what was the dimensions of the order-related feature and identity-related feature?\nIn general, I think this paper is above the borderline. But I  would also like to see the comments from other reviewers. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "__26QH1DMNT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary of paper**:\n\nThis paper considers the task ordered learning, making predicting a class label for a point among an ordered graph of classes. The paper proposes a clustering objective that encourages the model to separate data into groups such that classification prediction is easier within each cluster. The method is intuitive, clearly explained and well motivated. The paper indicates state of the art results on a task of estimating ages of individuals from photographs. \n\n**Review summary**: Missing *crucial* discussion on discussion of use cases / broader impact of task of estimating ages from photographs. Otherwise intuitive and effective method for ordered data; effective empirical results; limited novelty / exploration of methodological approach.\n\n**Strengths**: \n\nThe authors describe an intuitive and effective method for making predictions on ordered data. The approach uses a intuitive clustering-based method that groups data into subsets where items are easier to order. The paper is clearly written and explains the approach clearly. The paper shows several examples of predicted output of the method and shows results on two tasks (estimating ages, aesthetic score regression). The method achieves state of the art results on the task of estimating ages and is competitive on the other task. The authors show further results on age transformation. \n\n**Weakness**:\n\n**Broader Impacts of Applications**:  One of the primary applications of the paper is estimating ages of individuals based on their photographs. While this is paper is not the first to focus on such a task, it is very remiss of this paper to not discuss the motivations for this task and the broader impacts and ethical considerations of this task. I would very strongly encourage the authors to add a discussion of the potential uses of their system and the benefits (as well as harms) that come from these uses. I think that it is crucially important to discuss this both in the context of this work as well as previous work on the task. In particular, it would be important to mention how the use of clustering (into groups based on gender/race) in this model factors into potential biases when the model is used. I think it would be necessary to include this discussion in the body of the paper itself rather than an appendix. I greatly believe that this discussion is necessary and the lack of it is one of my top concerns about the paper. \n\n**Distinctions between total ordering and partial ordered / related work**: The presentation of the approach indicates that observations are not directly comparable across clusters. However, the overall model does in fact provide a total ordering -- each point is mapped one of the clusters and then compared within that cluster. I think the presentation would be greatly improved if it were described not in a way that implies a partial ordering (only within each cluster) is there, but instead that the total ordering function is this multi-modal, cluster-based ordering. Further, I think it would important to discuss the relationships between this work and work on partially ordering sets, particularly work on combining partially ordered sets.  It might also be good to consider more related work on ordering, such as, Learning to Order Things  (https://papers.nips.cc/paper/1431-learning-to-order-things.pdf). Also, I think that it is especially important to address other work (such as that in extreme classification) that organizes class labels into groups that are easier to discriminate between (i.e.,  Logarithmic Time One-Against-Some ( https://arxiv.org/abs/1606.04988)).\n\n**Novelty of approach / depth of exploration**: The core novelty of the approach is in the use of clustering to separate the data into groups that are easier to rank. This is a nice idea and appears give strong empirical benefits. I worry that since the clustering component is the core contribution of the paper, that the analysis of the method of clustering is not very deeply explored empirically. The idea is intuitive, but I feel the limited deviation from classic approaches that combine clustering + classification would benefit from additional analysis of the approach, along the dimension of the clustering objective that is selected.  \n\n**Questions for the authors:**\n\n* What are the potential use cases for the system & its applications to age prediction? \n* What are the fairness/ethical/safety concerns of such an application?\n* Were clustering objectives other than the repulsive-based one considered? \n* How does your work connect to papers such as Logarithmic Time One-Against-Some ( https://arxiv.org/abs/1606.04988) which also organize classes into clusters ? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1Zv40MIsqEl", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper considers the problem of order learning, which learns an ordinal classification function. This paper proposes to learn separarted order-relavent and order-irrelavent latent representations to improve the performance of existing methods, which is a very interesting and promising idea. However, the approach lacks novelty and convincing theoretical guarantees, as well as not showing convincing performance even through the insufficient empirical evaluation.\n\nMain concerns:\n- The ORID model structure: The latent representation is separated to h_{or} and h_{id}, and the comparison loss is defined on h_{or}. However, this need not to exclude order-relavent information from h_{id}. Also, it needs to be clarified that to what exent introudcing a discriminator helps, as this turns a minimization problem into an unstable min-max optimization problems. How it works without the discriminator?\n\n- Normalization of h_{id}: Normalizing vectors in a space may result a totally different cluster structure, different clusters may appear to be overlapped with each other by normalization. Euclidean distance can be the natural dissimilarity metric without normalization.\n\n- The DRC algorithm: The idea of encouraging inner-cluster similarity and iter-cluster dissimilarity of Eq. (9) is not new. Also, right after Algorithm 1 in the paper, \"DRC is guaranteed to converge to a local maximum\" is quite suspicious. Is it true that different rules optimiziming the same objective alternatively is guaranteed to converge? At least some references need to be provided as it is a crucial point of the main contribution.\n\n- The decisioin rule: Eq.(15) loops over all y, so what is the point of selecting a y_i in Eq.(13)?\n\n- Experimental results seem to be fine, and authors are honest to report unfavorable results. However, in my humble opinion, results for a sufficient number of repetitions (5 or 10) is needed to achieve a least convincibility.\n\nMinor comments:\n- In Eq.(4), the rightmost inequation should be \\theta(x) - \\theta(y) < -r.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is overall well written and clearly presented. The problem of ordered data clustering is relevant, and the proposed method is effective.

During the discussion, all reviewers agree with the strength of this paper and share the positive impression. The authors successfully addressed reviewers' concerns by the careful author response, which I also acknowledge.
One of the reviewers raised the concern about the broader impacts, while it is also well addressed in the author response.

I therefore recommend acceptance of the paper.
Decision: Accept (Poster)