Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The novelty of the network structure is marginal. The decomposition way of feature is very common in computer vision. Just utilizing the latent vector of the encoder with only the comparator loss to decompose the feature into two feature types is limited. The authors should show the visual differences between these two feature types. The expression of the article is very clear, but some basic theories need not be explained in detail (Such in Section 3.4)
One more concern : h_id and h_or are both used for reconstruction. It’s best to prove that only using identity feature h_id is better than the overall latent vector h_id + h_or. 

review_title: The proposed method has good indicators and visualization effects. Compared with the previous methods, a better framework is proposed. The expression of the article is very clear, but some basic theories need not be explained in detail. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: - It is well presented. The idea of splitting the encoding feature space into task related features and non-task related features is probably not new. But the use of it in estimating rank might be new and intuitively it makes sense to use it. They also propose an extension to the clustering algorithm using a repulsive term and propose MAP estimation algorithm to assign a rank based on the output probabilities of the comparator when the max possible rank is known.
- Experiments are conducted on 3 data sets. The results show the effectiveness of the approach. The experiments, I feel, are sufficient to show that clustering instances based on non-rank related features will help improve effectiveness of comparison based ranking of new instances. They also show the effectiveness of their proposed MAP estimation rule for assigning a rank.
- The effectiveness of the repulsive clustering on ranking performance is not clear. The authors discuss that using the repulsive term in the objective for clustering produces more distinct clusters but how does this "improved" cluster quality translate to better performance in ranking? As this is one of the key contributions of the paper, a comparison of ranking performances with and without the use of the repulsive term in clustering would be useful.
- How sensitive/robust is the proposed approach to the number of clusters chosen? How can one choose the right number of clusters to use? A discussion on these would be useful.
- In each experiment, what was the dimensions of the order-related feature and identity-related feature?
In general, I think this paper is above the borderline. But I  would also like to see the comments from other reviewers. 
review_title: Well presented paper, some details need to be clarified. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary of paper**:

This paper considers the task ordered learning, making predicting a class label for a point among an ordered graph of classes. The paper proposes a clustering objective that encourages the model to separate data into groups such that classification prediction is easier within each cluster. The method is intuitive, clearly explained and well motivated. The paper indicates state of the art results on a task of estimating ages of individuals from photographs. 

**Review summary**: Missing *crucial* discussion on discussion of use cases / broader impact of task of estimating ages from photographs. Otherwise intuitive and effective method for ordered data; effective empirical results; limited novelty / exploration of methodological approach.

**Strengths**: 

The authors describe an intuitive and effective method for making predictions on ordered data. The approach uses a intuitive clustering-based method that groups data into subsets where items are easier to order. The paper is clearly written and explains the approach clearly. The paper shows several examples of predicted output of the method and shows results on two tasks (estimating ages, aesthetic score regression). The method achieves state of the art results on the task of estimating ages and is competitive on the other task. The authors show further results on age transformation. 

**Weakness**:

**Broader Impacts of Applications**:  One of the primary applications of the paper is estimating ages of individuals based on their photographs. While this is paper is not the first to focus on such a task, it is very remiss of this paper to not discuss the motivations for this task and the broader impacts and ethical considerations of this task. I would very strongly encourage the authors to add a discussion of the potential uses of their system and the benefits (as well as harms) that come from these uses. I think that it is crucially important to discuss this both in the context of this work as well as previous work on the task. In particular, it would be important to mention how the use of clustering (into groups based on gender/race) in this model factors into potential biases when the model is used. I think it would be necessary to include this discussion in the body of the paper itself rather than an appendix. I greatly believe that this discussion is necessary and the lack of it is one of my top concerns about the paper. 

**Distinctions between total ordering and partial ordered / related work**: The presentation of the approach indicates that observations are not directly comparable across clusters. However, the overall model does in fact provide a total ordering -- each point is mapped one of the clusters and then compared within that cluster. I think the presentation would be greatly improved if it were described not in a way that implies a partial ordering (only within each cluster) is there, but instead that the total ordering function is this multi-modal, cluster-based ordering. Further, I think it would important to discuss the relationships between this work and work on partially ordering sets, particularly work on combining partially ordered sets.  It might also be good to consider more related work on ordering, such as, Learning to Order Things  (https://papers.nips.cc/paper/1431-learning-to-order-things.pdf). Also, I think that it is especially important to address other work (such as that in extreme classification) that organizes class labels into groups that are easier to discriminate between (i.e.,  Logarithmic Time One-Against-Some ( https://arxiv.org/abs/1606.04988)).

**Novelty of approach / depth of exploration**: The core novelty of the approach is in the use of clustering to separate the data into groups that are easier to rank. This is a nice idea and appears give strong empirical benefits. I worry that since the clustering component is the core contribution of the paper, that the analysis of the method of clustering is not very deeply explored empirically. The idea is intuitive, but I feel the limited deviation from classic approaches that combine clustering + classification would benefit from additional analysis of the approach, along the dimension of the clustering objective that is selected.  

**Questions for the authors:**

* What are the potential use cases for the system & its applications to age prediction? 
* What are the fairness/ethical/safety concerns of such an application?
* Were clustering objectives other than the repulsive-based one considered? 
* How does your work connect to papers such as Logarithmic Time One-Against-Some ( https://arxiv.org/abs/1606.04988) which also organize classes into clusters ? 

review_title: Missing discussion of broader impacts of application for an intuitive method for ordered data, also missing technical novelty / analysis

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper considers the problem of order learning, which learns an ordinal classification function. This paper proposes to learn separarted order-relavent and order-irrelavent latent representations to improve the performance of existing methods, which is a very interesting and promising idea. However, the approach lacks novelty and convincing theoretical guarantees, as well as not showing convincing performance even through the insufficient empirical evaluation.

Main concerns:
- The ORID model structure: The latent representation is separated to h_{or} and h_{id}, and the comparison loss is defined on h_{or}. However, this need not to exclude order-relavent information from h_{id}. Also, it needs to be clarified that to what exent introudcing a discriminator helps, as this turns a minimization problem into an unstable min-max optimization problems. How it works without the discriminator?

- Normalization of h_{id}: Normalizing vectors in a space may result a totally different cluster structure, different clusters may appear to be overlapped with each other by normalization. Euclidean distance can be the natural dissimilarity metric without normalization.

- The DRC algorithm: The idea of encouraging inner-cluster similarity and iter-cluster dissimilarity of Eq. (9) is not new. Also, right after Algorithm 1 in the paper, "DRC is guaranteed to converge to a local maximum" is quite suspicious. Is it true that different rules optimiziming the same objective alternatively is guaranteed to converge? At least some references need to be provided as it is a crucial point of the main contribution.

- The decisioin rule: Eq.(15) loops over all y, so what is the point of selecting a y_i in Eq.(13)?

- Experimental results seem to be fine, and authors are honest to report unfavorable results. However, in my humble opinion, results for a sufficient number of repetitions (5 or 10) is needed to achieve a least convincibility.

Minor comments:
- In Eq.(4), the rightmost inequation should be \theta(x) - \theta(y) < -r.

review_title: Official Blind Review #4

Meta-review: This paper is overall well written and clearly presented. The problem of ordered data clustering is relevant, and the proposed method is effective.

During the discussion, all reviewers agree with the strength of this paper and share the positive impression. The authors successfully addressed reviewers' concerns by the careful author response, which I also acknowledge.
One of the reviewers raised the concern about the broader impacts, while it is also well addressed in the author response.

I therefore recommend acceptance of the paper.
Decision: Accept (Poster)