Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper departs from traditional age estimation methods by proposing an ordering/ranking way. A testing face is compared to those anchor faces from each age category to estimate the right category that the testing face belongs to. It is novel to use the MC rule to drive the order learning. What more promising is the unsupervised chains, which could automatically search for a more optimal multi-chain division scheme than the pre-defined data division. The paper is well written.

I have a few concerns as the following:
1.	The geometric ratio (Eq. 10) further enlarges the distance between younger faces and shortens the distance between older faces. This may improve the prediction of younger ages. However, this makes the prediction on older ages more confusion, i.e., too robust to distinguish adjacent categories. The results shown in Table 6 show a better number for Geometric, but it is tested on FG-NET which presents unbalanced age distribution (ages up to 40 years are the most populated). Therefore, the improvement may be caused by the data distribution that is more suitable for geometric ratio, i.e., more younger faces and less older faces. Table 8 also provides such comparison, but please clarify the testing dataset. 
2.	In Algorithm 1, are you using a pre-trained comparator? If not, the T_k obtained from line 3 is still close to random. Such random initialization works for multiple chains with small mutual shift (the distance of references between chains). As indicated in Table 3, more chains will not significantly affect performance, which implies a relatively small mutual shift. However, if encounter a large shift, e.g., other properties would significantly affect the target property that you want to classify (ranking soccer teams where gender makes difference), will the proposed unsupervised learning converge? 
3.	It would be great if the authors could provide more insight to the 2CH unsupervised results. How the two chains are divided by analyzing the trained model? What is the key to divide the data? Maybe, it is like a black-box that is hard to explain but this will not affect the merit of this paper.
 
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper presents an order learning method and applies it to age estimation from facial images. It designs a pairwise comparator that categorizes ordering relationship between two instances into ternary classes of greater than, similar, and smaller than. Instead of directly estimating the class of each instance, it learns pairwise ordering relationship between two instances.

For age estimation from facial images, it uses a Siamese network as a feature extractor of the pairwise comparator, concatenates feature maps of dual inputs (test instance and one of the multiple reference instances that are selected from the training data) and applies it to a ternary classifier. Given the softmax probability scores of the classifier results, it estimates the final class as the one that maximizes a consistency rule computed over indicator functions. The comparator is trained to minimize a comparator loss computed over the softmax probability scores.

The paper also provides an extended version for multiple disjoint chains, where each chain may correspond to a higher-level attribute class, for example, gender or ethnic group. When there is no supervision available, it randomly partitions the training set.  and iteratively updates reliability scores and chain memberships. 

Novelty-wise, I consider that the proposed solution to be satisfactorily innovative and on a different vein than the existing methods. 

One concern about the method is that it imposes the geometric ratio (log distance) between the class distances in age estimation, considering the difference between 5 and 10-year-old instances is easier to detect than 65 to 70-year-old instances as stated in the paper. However, as far as it is understood from the provided discussion compared SOTA methods are not retrained or fine-tuned in the same manner. This raises the question of whether the slight performance improvement is a result of the geometric ratio, in particular when computing the cumulative score CS.   

Another concern is that the performance is not stellar as the presented method underperforms in comparison to DRF (Shen et al. 2018) on MORPH II. The presentation of the results for FG-Net is neither complete nor included in the main paper, and results for CLAP2016 are missing. Discussion of the results Table 8 does not seem fair as it is not clear whether MV is retrained keeping the geometric ratio in mind. It might be the case that even for 15-69, MV might attain lover MAEs. 

The paper also misses the latest SOTA, for instance, BridgeNet [1] is mentioned in a sentence, yet its results are omitted from the Tables. BridgeNet, according to its results, is on par with the proposed method. Also, the numbers reported in this paper and the numbers in [1] have a discrepancy, which causes confusion.

[1] Wanhua Li et al. Bridgenet: A continuity aware probabilistic network for age estimation, CVPR 2019.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper proposes a method for learning partial orders based on learning fuzzy pairwise comparisons (smaller/greater/approximatively equal to), and the retaining of a set of representants in each chain in order to allow consistent result by consistency maximization. The method is applied to the problem of estimating the age based on faces. Extensions proposed are the learning of multiple disjoint chains on this dataset, manually partitioned or learned through an iterative assignment algorithm that has similarities with a soft expectation-maximization principle. Extensive experiments are done on the age estimation problem with comparison to exisiting approaches, and an application to aesthetic assessment is proposed in appendix.

The paper is very well written and contains very extensive and relevant experiments. I only have minor concerns:

1) I couldn't find the explanation for the underlined numbers in Table 2.
2) It would be good to include the details of Appendix C in the text, as is, it looks like an artificial space saving trick.
3) typo in Section 3.2: "we assume that the chain, to which the instance belong, is known" -> "we assume that the chain to which the instance belongs is known"
review_title: Official Blind Review #1

Meta-review: This paper addresses a promising method for order learning and applies the new ideas of multiple-chain learning and anchor selection to age estimation and aesthetic regression. The decision regarding instance class is made by comparing it with anchor instances in the same chain and maximizing the consistency among the comparison results. In a multi-chain setting, each chain may correspond to a higher-level attribute class, for example, gender or ethnic group. Supervised and unsupervised learning of multiple ordered chains is proposed.  As rightly acknowledged by R4: “What more promising is the unsupervised chains, which could automatically search for a more optimal multi-chain division scheme than the pre-defined data division.”
All three reviewers and AC agree that the proposed approach is interesting and shows promising results. There are several potential weaknesses and suggestions to further strengthen this work:
(1) more quantitative results are needed for assessing the benefits of this approach (R3, R4) -- see R3’s request to complete the results for FG-Net, to include the results for CLAP2016 and a comparison with the SOTA method BridgeNet. Pleased to report that the authors have revised the manuscript and have included performance of the arithmetic scheme as well as the geometric scheme for FG-Net. Also the authors have provided some initial evaluations of BridgeNet and promised to report the final results as well as the results for CLAP2016 in the final version. 
(2) R3 and R4 have expressed concerns regarding using the geometric ratio of the class distances in age estimation and that the improvement may be caused by the data distribution that favours it (R4) or because the baseline methods are not fine-tuned in the same manner (R3). The authors have partially addressed this concern in the rebuttal. 
There is a large body of work in computer vision that is focused on relative comparison of samples based on attributes (e.g. age) that is not clearly articulated in the discussions / baseline comparisons (1CH) -- see the seminal work [Relative attributes by Parikh and Grauman, ICCV2011] and the follow up works. 
Considering the author response, the AC decided that the most crucial concerns have been addressed in the revision and that the paper could be accepted, but the authors are strongly urged to include additional results that were promised in the rebuttal for the final revision. 

Decision: Accept (Poster)