Deep Repulsive Clustering of Ordered Data Based on Order-Identity Decomposition

We propose the deep repulsive clustering ( DRC ) algorithm of ordered data for effective order learning . First , we develop the order-identity decomposition ( ORID ) network to divide the information of an object instance into an order-related feature and an identity feature . Then , we group object instances into clusters according to their identity features using a repulsive term . Moreover , we estimate the rank of a test instance , by comparing it with references within the same cluster . Experimental results on facial age estimation , aesthetic score regression , and historical color image classification show that the proposed algorithm can cluster ordered data effectively and also yield excellent rank estimation performance . 1 INTRODUCTION . There are various types of ‘ ordered ’ data . For instance , in facial age estimation ( Ricanek & Tesafaye , 2006 ) , face photos are ranked according to the ages . Also , in a video-sharing platform , videos can be sorted according to the numbers of views or likes . In these ordered data , classes , representing ranks or preferences , form an ordered set ( Schröder , 2003 ) . Attempts have been made to estimate the classes of objects , including multi-class classification ( Pan et al. , 2018 ) , ordinal regression ( Frank & Hall , 2001 ) , metric regression ( Fu & Huang , 2008 ) . Recently , a new approach , called order learning ( Lim et al. , 2020 ) , was proposed to solve this problem . Order learning is based on the idea that it is easier to predict ordering relationship between objects than to estimate the absolute classes ( or ranks ) ; telling the older one between two people is easier than estimating their exact ages . Hence , in order learning , the pairwise ordering relationship is learned from training data . Then , the rank of a test object is estimated by comparing it with reference objects with known ranks . However , some objects can not be easily compared . It is less easy to tell the older one between people of different genders than between those of the same gender . Lim et al . ( 2020 ) tried to deal with this issue , by dividing an ordered dataset into disjoint chains . But , the chains were not clearly separated , and no meaningful properties were discovered from the chains . In this paper , we propose a reliable clustering algorithm , called deep repulsive clustering ( DRC ) , of ordered data based on order-identity decomposition ( ORID ) . Figure 1 shows a clustering example of ordered data . Note that some characteristics of objects , such as genders or races in age estimation , are not related to their ranks , and the ranks of objects sharing such characteristics can be compared more reliably . To discover such characteristics without any supervision , the proposed ORID network decomposes the information of an object instance into an order-related feature and an identity feature unrelated to the rank . Then , the proposed DRC clusters object instances using their identity features ; in each cluster , the instances share similar identity features . Furthermore , given a test instance , we decide its cluster based on the nearest neighbor ( NN ) rule , and compare it with reference instances within the cluster to estimate its rank . To this end , we develop a maximum a posteriori ( MAP ) estimation rule . Experimental results on ordered data for facial age estimation , aesthetic score regression ( Kong et al. , 2016 ) , and historical color image classification ( Palermo et al. , 2012 ) demonstrate that the proposed algorithm separates ordered data clearly into meaningful clusters and provides excellent rank estimation performances for unseen test instances . The contributions of this paper can be summarized as follows . • We first propose the notion of identity features of ordered data and develop the ORID network for the order-identity decomposition . • We develop the DRC algorithm to cluster data on a unit sphere effectively using a repulsive term . We also prove the local optimality of the solution . • We propose the MAP decision rule for rank estimation . The proposed algorithm provides the state-of-the-art performances for facial age estimation and aesthetic score regression . 2 RELATED WORK . 2.1 ORDER LEARNING . The notion of order learning was first proposed by Lim et al . ( 2020 ) . It aims to determine the order graph of classes and classify an object into one of the classes . In practice , it trains a pairwise comparator , which is a ternary classifier , to categorize the relationship between two objects into one of three cases : one object is bigger than , similar to , or smaller than the other . Then , it estimates the rank of a test object , by comparing it with reference objects with known ranks . However , not every pair of objects are easily comparable . Although Lim et al . ( 2020 ) attempted to group objects into clusters , in which objects could be more accurately compared , their clustering results were unreliable . Pairwise comparison has been used to estimate object ranks , because relative evaluation is easier than absolute evaluation in general . Saaty ( 1977 ) proposed the scaling method to estimate absolute priorities from relative priorities , which has been applied to various decision processes , including aesthetic score regression ( Lee & Kim , 2019 ) . Also , some learning to rank ( LTR ) algorithms are based on pairwise comparison ( Liu , 2009 ; Cohen et al. , 1998 ; Burges et al. , 2005 ; Tsai et al. , 2007 ) . Order learning attempts to combine ( possibly inconsistent ) pairwise ordering results to determine the rank of each object . Thus , it is closely related to the Cohen et al. ’ s LTR algorithm ( 1998 ) , which learns a pairwise preference function and obtains a total order of a set to maximize agreements among preference judgments of pairs of elements . Also , order learning is related to rank aggregation ( Dwork et al. , 2001 ) , in which partially ordered sets are combined into a linearly ordered set to achieve the maximum consensus among those partial sets . Rank aggregation has been studied in various fields ( Brüggemann et al. , 2004 ) . Since optimal aggregation is NP-hard , Dwork et al . ( 2001 ) proposed an approximate algorithm , called Markov chain ordering . There are many other approximate schemes , such as the local Kemenization , Borda count , and scaled footrule aggregation . 2.2 CLUSTERING . Data clustering is a fundamental problem to partition data into disjoint groups , such that elements in the same group are similar to one another but elements from different groups are dissimilar . Although various clustering algorithms have been proposed ( Hartigan & Wong , 1979 ; Ester et al. , 1996 ; Kohonen , 1990 ; Dhillon & Modha , 2001 ; Reynolds , 2009 ) , conventional algorithms often yield poor performance on high-dimensional data due to the curse of dimensionality and ineffectiveness of similarity metrics . Dimensionality reduction and feature transform methods have been studied to map raw data into a new feature space , in which they are more easily separated . Linear transforms , such as PCA ( Wold et al. , 1987 ) , and non-linear transformations , including kernel methods ( Hofmann et al. , 2008 ) and spectral clustering ( Ng et al. , 2002 ) , have been proposed . Recently , deep neural networks have been adopted effectively as feature embedding functions ( LeCun et al. , 2015 ) , and these deep-learning-based feature embedding functions have been combined with classical clustering algorithms . For instance , Caron et al . ( 2018 ) proposed a deep clustering algorithm based on k-means . It clusters features from a neural network and then trains the network using the cluster assignments as pseudo-labels . This is done iteratively . Also , Yang et al . ( 2016 ) jointly learned feature representations and clustered images , based on agglomerative clustering . Chang et al . ( 2017 ) recast the image clustering task into a binary classification problem to predict whether a pair of images belong to the same cluster or different clusters . Similarly to these algorithms , we use a neural network to determine a feature space in which clustering is done more effectively . However , we consider the clustering of ordered data , and each cluster should consists of elements , whose ranks can be compared more accurately . There are conventional approaches to use clustering ideas to aid in classification or rank estimation . For example , Yan et al . ( 2015 ) developed a hierarchical classifier , which clusters fine categories into coarse category groups and classifies an object into a fine category within its coarse category group . For extreme multiclass classification , Daumé III et al . ( 2017 ) proposed to predict a class label among candidate classes only , which are dynamically selected by the recall tree . It is however noted that the leaves of the recall tree do not partition the set of classes . Also , for age estimation , Li et al . ( 2019 ) proposed a tree-like structure , called bridge-tree , to divide data into overlapping age groups and train a local regressor for each group . The set of local regressors can be more accurate than a global regressor to deal with the entire age range . Whereas these conventional approaches group data in the label dimension to perform their tasks more effectively , the proposed algorithm cluster data in the dimension orthogonal to the label dimension . In other words , we cluster data using identity features , instead of using order features . 3 PROPOSED ALGORITHM . 3.1 PROBLEM DEFINITION . An order is a binary relation , often denoted by ≤ , on a set Θ = { θ1 , θ2 , . . . , θm } ( Schröder , 2003 ) . It should satisfy three properties of reflexivity ( θi ≤ θi for all i ) , antisymmetry ( θi ≤ θj and θj ≤ θi imply θi = θj ) , and transitivity ( θi ≤ θj and θj ≤ θk imply θi ≤ θk ) . Then , Θ is called a partially ordered set . Furthermore , if every pair of elements are comparable ( θi ≤ θj or θj ≤ θi for all i , j ) , Θ is called a chain or linearly ordered set . An order describes ranks or priorities of classes . For example , in age estimation , θi may represent the age class of i-year-olds . Then , θ14 ≤ θ49 represents that 14-year-olds are younger than 49-year-olds . As mentioned previously , it is less easy to tell the older one between people of different genders . An algorithm , hence , may compare a subject with reference subjects of the same gender only . In such a case , each age class θi represents two subclasses θfemalei and θ male i of different types , and the algorithm compares only subjects of the same type . Lim et al . ( 2020 ) assumed that subclasses of different types are incomparable and thus the set of subclasses is the union of k disjoint chains , where k is the number of types . However , in many ranking applications , objects of different types can be compared ( although less easily than those of the same type are ) . Thus , instead of assuming incomparability across chains , we assume that there is a total order on Θ = { θ1 , θ2 , . . . , θm } , in which each class θi consists of k types of subclasses , and that object instances of the same type are more easily compared than those of different types . Suppose that n training instances in X = { x1 , x2 , . . . , xn } are given . Also , suppose that there are m ranks and the ground-truth rank of each instance is known . In this sense , X contains ordered data . The problem is twofold . The first goal is to decompose the whole instances X into k disjoint clusters { Cj } kj=1 in which instances are more easily compared ; X = ⋃k j=1 Cj ( 1 ) where Ci∩Cj = ∅ for i 6= j . In other words , we aim to partition the ordered data inX into k clusters , by grouping them according to their characteristics unrelated to their ranks . These characteristics , which tend to remain the same even when an object experiences rank changes , are referred to as ‘ identity ’ features in this work . For example , in age estimation , genders or races can be identity features . However , we perform the clustering without any supervision for identity features . Notice that instances within a cluster would be compared more easily than those across clusters , since they have similar identity features . The number k of clusters is assumed to be known a priori . Impacts of k on the clustering performance are discussed in Appendix B.7 . The second goal is to assign an unseen test instance into one of the clusters and determine its rank by comparing it with reference instances within the cluster . To achieve these goals , we propose the ORID network and the DRC algorithm .

Summary: This paper considers the problem of order learning, which learns an ordinal classification function. This paper proposes to learn separarted order-relavent and order-irrelavent latent representations to improve the performance of existing methods, which is a very interesting and promising idea. However, the approach lacks novelty and convincing theoretical guarantees, as well as not showing convincing performance even through the insufficient empirical evaluation.