Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors introduce a relational surrogate loss learning method (ReLoss) for replacing the original losses. The rationale and intuition behind are well-grounded. Experiments on various tasks and ablation studies prove the validity.
main_review: The proposed approach is well motivated and makes sense. The problem study here is also essential and could be of interest to a large audience, as the loss function is fundamental in almost all machine learning tasks. 

Meanwhile, the techniques are simple but effective. Without doubts, the proposed correlation-based optimization that introduces differentiable Spearman's correlation coefficient has looser constraints than approximation-based methods. Also, I believe the gradient penalty is necessary to keep stable convergence and fixed training strategies, since the magnitude of gradients may changes w.r.t. the initialization and randomness.

Experiments are sufficient and convincing. For example, the ReLoss gains noticeable improvements over baselines and outperforms the state-of-the-art method on COCO keypoint dataset.

However, I do have some minor concerns in this paper: 

(a) The surrogate losses can be learned with the universal method (ReLoss) without expertise on loss function design, but it seems to introduce a new problem of confirming the architecture of losses w.r.t the evaluation metrics. 

(b) Approximation errors always exist as the surrogate loss cannot fully recover the evaluation metrics in the whole distribution. In this paper, the authors adopt the original loss as a regularization term to alleviate this problem, but the superiority of ReLoss on rank correlation may be weakened.
summary_of_the_review: The paper proposes a method to learn surrogate losses, compared to related works, the proposed method gains significant improvements on various datasets. The originality and significance are clearly above the bar, though there still remain some minor concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: - This paper proposes a relational surrogate loss learning method (ReLoss) inspired by the fact that the evaluation metric and loss are used to distinguish whether one model is better or worse than another. 

- This paper provides extensive experiments that demonstrate the effectiveness of the proposed method. The performance and efficiency compared to existing surrogate loss methods are significant, and the performance compared to original losses is seem to be significant on various tasks. 

main_review: - The study field of this paper is important. According to the authors’ experiments in Table 1, the original manually-designed losses do not align well with the evaluation metrics on some tasks. The proposed ReLoss can improve those ranking correlations a lot. 

- The proposed method can be widely applied to NLP and CV tasks, and its benefits seem significant according to the experiments. E.g., on human pose estimation task, ReLoss outperforms the state-of-the-art method with only replacement of loss function.  

Minor Weakness: 

- How much additional costs if we train a model using ReLoss? How much time does it cost to train ReLoss? Would ReLoss increase GPU memory or training time in training? This is related to the practical feasibility of the proposed loss.  


summary_of_the_review: I think the proposed method is well-principled and provides meaningful improvements on various tasks. I would like to lean on the positive side for this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a surrogate loss learning method named ReLoss. The ReLoss learned by maximizing the relation between surrogate losses and evaluation metrics is used to replace the original losses. Extensive experiments on computer vision (CV) tasks (image classification, pose estimation, and scene text recognition) and natural language processing (NLP) tasks (machine reading comprehension and translation) are provided, showing the benefits.
main_review: Pros:
* The idea is novel and interesting. Loss functions matter a lot to the performance, while the current hand-craft losses often align poorly to the performance, surrogate loss learning is a way to alleviate this issue. Instead of directly approximating the evaluation metrics as previous methods, this paper proposes a new learning method by revisiting the purpose of loss functions, which is to distinguish the performance of models. Hence, the authors aim to learn the surrogate losses by making the surrogate losses have the same discriminability as the evaluation metrics. The idea is straightforward and is easy to implement by using ranking correlation as an optimization objective.
* To obtain stable gradients, the authors involve a gradient penalty regularization term by enforcing 1-Lipschitz, which is effective according to the experiments in Appendix.
* The experimental results look good. By keeping the same backbones and training strategies, the proposed ReLoss gains improvements on various tasks, including CV and NLP tasks, and outperforms the state-of-the-art methods on human pose estimation and machine reading comprehension tasks.
* The comparisons are sufficient to show the proposed method's superiority. The authors conduct experiments on the synthetic and large-scale benchmark datasets, and gain significant improvements in performance and efficiency compared to existing surrogate loss learning methods.
 
Cons:
* The ReLoss is learned by optimization, which means the qualities of losses may be different in multiple runs. How many times did the authors train the ReLoss to obtain the performance reported in the paper? It would be better to conduct experiments to show the performance of surrogate losses learned in multiple independent runs.

summary_of_the_review: In summary, the reviewer thinks the paper gives an effective way to obtain better losses, the experiments are well-conducted, and even improve the state-of-the-art methods, which would be helpful to the community. I would recommend it for acceptance.
review_title: null

Meta-review: The paper presents an approach to learn the surrogate loss for complex prediction tasks where the task loss is non-differentiable and non-decomposable. The novelty of the approach is to rely on differentiable sorting, optimizing the spearman correlation between the true loss and the surrogate. This leads to a pipeline that is simpler to integrate to existing works than approaches that try to learn a differentiable approximation to the task loss, and to better experimental results.

The paper is well written and the approach clearly presented. The reviewers liked the simplicity of the approach and the promising experimental results on a variety of challenging tasks (human pose estimation and machine reading).
Decision: Accept (Poster)