Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: A GOL embedding space represents the direction and distance between objects represent order and metric relations between their ranks, by enforcing two geometric constraints 1) the order/rank constraint  and 2) metric constraint reflects rank difference. 

Estimates a test object rank are achieved by kNN, and a metric called discriminative ratio for ranking (DRR) estimates the quality of rank estimation embedding spaces.

Experiments on facial age estimation, historical color image (HCI) classification, and aesthetic score regression demonstrate that GOL constructs effective  embedding spaces and yields rank estimation performances.
strengths_and_weaknesses: It seems GOL is the first attempt to design an embedding space in which the direction and distance between objects represent their order and metric relations. The GOL algorithm performs best in 80% of benchmark datasets for facial age estimation, HCI classification, and aesthetic score regression.

It is not clear how here, the rank and DRR metric are related to traditional (mathematical) methods defining according rank and order geometry, without learning. For example, ranked gradient orientation features are used for head age estimation, classification, with much success, prior to this work.

E.g. please see SIFT-Rank, SIFT being the "scale-invariant feature transform" (the state-of-the-art before GPU-based deep CNN learning), and Rank being the order statistics of the SIFT descriptor (equivalent to uniform orientation sampled gradient filtering).
https://ieeexplore.ieee.org/document/5206849

Toews, M., Wells, W.M. and Zöllei, L., 2012, October. A feature-based developmental model of the infant brain in structural MRI. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 204-211). Springer, Berlin, Heidelberg.


questions: Would be interesting to know the link to traditional ranking theories such as SIFT-Rank.

In Figure 1 c), are we seeing ordered objects arranged in a "band" around a hypersphere?

limitations: There are no obvious limitations, except demonstration on more datasets & tasks.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces geometric order learning (GOL) method for rank estimation by enforcing two geometric constraints: the order constraint and the metric constraint. The order constraint enforces the feature vectors of instances to be arranged according to their ranks, and the metric constraint makes the distance between instances reflect their rank difference. The paper also proposes discriminative ratio ranking metric to assess the quality of embedding spaces for rank estimation. Extensive experiments demonstrate that GOL constructs effective embedding spaces and yields excellent rank estimation performances.
strengths_and_weaknesses: Strengths：
+ A geometric order learning (GOL) method for rank estimation is proposed by enforcing the order constraint and the metric constraint.
+ A discriminative ratio ranking metric is introduced to assess the quality of embedding spaces for rank estimation.
+ Experiments are conducted to demonstrate that GOL constructs effective embedding spaces and yields excellent rank estimation performances.

Weaknesses:
- The proposed GOP method rely on reference points, it is not clear how to select these reference points and how they affect the performance of GOP.
- As the network architectures of different methods in Table 1 are different, so whether the comparison is fair?
questions: - The proposed GOP method rely on reference points, it is not clear how to select these reference points and how they affect the performance of GOP.
- As the network architectures of different methods in Table 1 are different, so whether the comparison is fair?
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: - The paper introduces an algorithm for learning an embedding space using which the rank of an object can be estimated, e.g. age of a person based on face images. 
- The embedding space is constrained such that the ordering and distance between training instances are preserved. 
- The authors introduce a metric for embedding space evaluation by repurposing the B2W metric (inter-class variance/intra-class variance) to account for the rank ordering of the data embeddings. 
strengths_and_weaknesses: - The paper is easy to read. However, parts of the introduction and related work section are repetitive. 
- The order and distance properties listed as preliminaries are common knowledge and seem to add no specific detail to the objective formulation in the paper.
- The central idea of the paper is to combine concepts from order learning and metric learning. The paper presents an objective by combining the two constraints. 
- The visualization presented in the paper shows that the proposed method learns embeddings that are ordered.
- Ablation study shows that each loss objective added helps improve the performance of the model. 

questions: - The order constraint formulation uses $v_b$ which is based on consecutive ranks while the forward uses difference in references corresponding to $x$ and $y$ ranks - Is there a reason for this asymmetry?
- Where does the gain in test runtime come from? I would assume comparing with 5 references/rank would be faster than searching for the k-nearest neighbor from $N$ training examples.
- Also, comparing methods where one uses a compressed representation in the form of references while the other uses the entire training set is a fair setting. The proposed method does learn references during training but seems to ignore these for inference - can the rank estimation be performed with just the references?

Minor: 
- How long is the model trained for? Is there a stopping condition?
- Is there any particular reason for choosing the VGG backbone?

limitations: - The authors show specific failure cases in facial age estimation where the lighting condition, overexposure, and other image perturbations affect the performance of the model - I think this is a general limitation of the dataset and is not specific to the model. A more careful analysis of the proposed method and its limitations might be useful.
- The paper lacks a discussion on why the embedding space has to be ordered - This is crucial for the premise of the method proposed. A method that embeds each rank in distinct spaces will be good at rank estimation (I believe this would explain the performance of other methods compared in the paper).
- The reported results seem to be based on a single run - It would be useful to report results based on multiple random initializations.
review_title: null

Meta-review: metareview: This paper proposes a new approach named geometric order learning (GOL) for rank estimation. Reviewers found that the idea is novel and the paper is well written. The authors have also clearly addressed most questions from reviewers in their responses. Thus, I recommend the acceptance of this paper.
confidence: Certain
Decision: Accept