Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a method to use CLIP for ordinal regression using a combination of soft labels and prefix tuning, with an interpolation scheme added to enforce order between the learnt prompts.
strengths_and_weaknesses: # Strengths
The idea of using context from natural language for ordinal regression tasks is interesting and worth exploring. The proposed interpolation method to allow regressing in between ordinal ranks is also clever. 

# Weaknesses
The main weakness to me is that an obvious baseline is missing from an engineering perspective. The paper uses a VGG-16 network pretrained on ImageNet as a trainable vision encoder for several datasets. The VGG-16 network has 138M parameters, similar to the ViT in the official CLIP release. The natural baseline would then have been to train a linear probe atop the CLIP ViT-B to simply predict the rank as a classification task. Second, the results in Fig 3. are very surprising. If I am reading it correctly, about 35% of the rank prototypes violate the ordinal property. This is a substantial portion, and suggests the proposed method cannot grasp anything on the long tail of the distribution. Third, _why_ do language priors help with this task at all? It's difficult to conceive that CLIP has learned a meaningful representation of say, the number 72. Even if CLIP did have a very meaningful representation of some arbitrary number, the rank embeds are entirely learned. How is the language information being used?  T1 is missing variance numbers for the comparison with CoOp. As evident in T2, CoOp and OrdinalClip are very close, with large variances. The authors should include a statistical test to confirm that the proposed method is indeed superior to CoOp. There are no variance numbers again in Table 7.

# Summary
The technique presented is a somewhat straightforward extension of CoOp, with the main novelty being that the literal rank names are replaced with soft class names. The results are in general very close to that of CoOp, and some tables are missing variance information, while others have variance information. Given how close the results of the method are to CoOp and how high the included variances are, variance results should be included for all tables, and statistical significance tests conducted. An important baseline (linear probing) is missing. Furthermore, it is unclear to me why language information can help with this kind of task at all, I do not see any experiments to explain why language information can help with this kind of task, or provide insight into what the language model is adding here. Finally, the result showing that 35% of the rank embeddings do not obey the ordinal property is troubling, especially since the "broken" rank embeddings are clustered in the long tail. This suggests the error distribution of the model is highly biased and the model does not work well at all for the tail. 
questions: Please see the strengths and weaknesses. 
limitations: Limitations are not discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a language-powered paradigm for ordinal regression tasks by learning rank prompts, named OrdinalCLIP. The OrdinalCLIP can leverage rank categories of language to explicit learning ordinal rank embeddings, which will preserve the order of the language prototypes in the language latent space. In the three regression tasks, including age estimation, historical image dating, and image aesthetics assessment, The experimental results show good performance than other baseline models. In addition, for few-shot learning, the method also gains improvement. The overall structure is well-organised. The paper has a clear motivation and is innovative for the regression field. 
strengths_and_weaknesses: Strength:

1. The innovative language-powered paradigm for ordinal regression uses language prototypes and learned rank prompts, which are interesting and valuable.

2. The good performance shows the effectiveness of OrdinalCLIP.

3. The approvement and experiments of the appendix are detailed.

Weakness:

1. The statement of introduction, related works, and problem statements should narrow down the ordinal regression to the vision-language or CV ordinal regression task cuz there are some pure language ordinal regression tasks.

2.The two loss (image-to-text loss and a text-to-image loss ) should be introduced in detailed, and the reason for using KL.

3. we choose to maintain the order of rank embeddings to preserve the order of the language prototypes. This statement is unclear. How to maintain the order of the languag?
questions: Questions:

1.To leverage the language priors with the text encoder, we treat the rank categories as words.  How to choose a suitable sentence? The sentence of “a person at the age of [rj] is the best? Are there some ablation studies?

2. we directly learn m word embeddings. what's the meaning of m? It's the ordinal number word?

3. See the weakness.

limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a language-powered model for ordinal regression based on CLIP. The language prototypes are constructed from sentences with rank categories via the CLIP paper encoder, and then optimizing the CLIP model by language prototype and image feature matching. To further boost the ordinality, this paper introduces the learnable rank prompts by interpolation from the base rank embeddings. Multiple experiments on age estimation, image aesthetics assessment and historical image dating show that the proposed paradigm surpasses other related methods.
strengths_and_weaknesses: Strengths: 
1. This paper introduces the contrastive language-image pretrained (CLIP) model as a paradigm for ordinal regression is novel to me.
2. The proposed language prototypes and learnable rank prompts are insightful extensions of the CLIP model for the ordinal regression task.
3. The proposed interpolation learning rank prompts contribute to the output of smooth language prototype similarity trends, which represent well learned ordinality.

Weaknesses:
1. The writing of this paper should be improved , including motivation, related work and task description, etc.
2. The motivation for the two proposed interpolations, linear interpolation and inverse-proportion, is unclear and lacks visualization for comparison other than numerical comparison. 

questions: 1. I would suggest that the authors better explain the motivation of the paper and the description of the tasks presented. While the ordinal regression approach proposed by the authors is novel and makes sense, I did not see a description of the task or the challenges.
2. Since this paper uses CoOP as a competitive method for comparison, the paper lacks a description of how to implement CoOP into ordinal regression. Could the authors discuss this further?

limitations: The authors have discussed the limitations and potential negative effects of their work.
review_title: null

Meta-review: metareview: The paper proposes a language-powered model for ordinal regression tasks, based on CLIP. Language prototypes are constructed from sentences with rank categories via the CLIP paper encoder, and then optimizing the CLIP model by language prototype and image feature matching. To further boost the ordinality, this paper introduces the learnable rank prompts by interpolation from the base rank embeddings.

While the proposed approach builds on CoOp, reviewers agree the contribution is significant enough and original enough for NeurIPS. Regarding the experimental section, the paper shows that on three regression tasks (age estimation, historical image dating, and image aesthetics assessment), results show good performance compared to baseline models.

Concerns regarding the writing of the manuscript have been raised [PfAX, RX3e], but seem to have been addressed during the rebuttal phase.
confidence: Certain
Decision: Accept