Relational Surrogate Loss Learning
Authors: Tao Huang
Zekang Li
Hua Lu
Yong Shan
Shusheng Yang
Yang Feng
Fei Wang
Shan You
Chang Xu
Venue: ICLR 2022 Poster (2022)

Abstract: Evaluation metrics in machine learning are often hardly taken as loss functions, as they could be non-differentiable and non-decomposable, e.g., average precision and F1 score. This paper aims to address this problem by revisiting the surrogate loss learning, where a deep neural network is employed to approximate the evaluation metrics. Instead of pursuing an exact recovery of the evaluation metric through a deep neural network, we are reminded of the purpose of the existence of these evaluation metrics, which is to distinguish whether one model is better or worse than another. In this paper, we show that directly maintaining the relation of models between surrogate losses and metrics suffices, and propose a rank correlation-based optimization method to maximize this relation and learn surrogate losses. Compared to previous works, our method is much easier to optimize and enjoys significant efficiency and performance gains. Extensive experiments show that our method achieves improvements on various tasks including image classification and neural machine translation, and even outperforms state-of-the-art methods on human pose estimation and machine reading comprehension tasks. Code is available at: https://github.com/hunto/ReLoss.

Published as a conference paper at ICLR 2022
RELATIONAL SURROGATE LOss LEARNING
Tao Huang Zekang Lii, Hua Lu", Yong Shan', Shusheng Yang',
Yang Feng", Fei Wang², Shan You?; Chang Xu
School of Computer Science, Faculty of Engineering, The University of Sydney
Time Research
"Key Laboratory of Intelligent Information Processing.
Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)
"Huazhong University Science and Technology
University of Science and Technology of China
ABSTRACT
Evaluation metrics in machine learning are often hardly taken as loss functions, as they could be non-differentiable and non-decomposable, e.g. average precision and Fl score. This paper aims to address this problem by revisiting the surro- gate loss learning, where deep neural network employed approximate the evaluation metrics. Instead of pursuing an exact recovery of the evaluation metric through a deep neural network, we are reminded of the purpose of the existence of these evaluation metrics, which is to distinguish whether one model is better or worse than another. In this paper, we show that directly maintaining the rela- tion of models between surrogate losses and metrics suffices, and propose rank correlation-aas optimization method to maximize this relation and learn surro- gate losses. Compared previous works, our method is much easier optimize and enjoys significant efficiency and performance gains. Extensive experiments show that our method achieves improvements on various tasks including image classification and neural machine translation, and even outperforms state-of-the- art methods on human pose estimation and machine reading comprehension tasks.
Code is available at:
com/ hunt o ReLoss
INTRODUCTION
Evaluation metrics matter in machine learning since it depicts how well we want the models to perform. Nevertheless, most of them are non-differentiable and non-decomposable, thus we can not directly optimize them during training but resort to loss functions (or surrogate losses), which serve exactly proxy of task metrics. For example, pose estimation task percentage of correct keypoints (PCK) (Yang & Ramanan 2012) to validate point-wise prediction accuracy, but often adopts mean square error (MSE) as loss function. Neural machine translation task takes the sentence-level metric BLEU (Papineni et al. 2002) to evaluate the quality of predicted sentences, while using word-level cross-entropy loss (CE Loss) in training.
Besides this manual proxy, some works (Grabocka et al. 2019 Patel er al. 2020 propose to learn surrogate losses which approximate the metries using deep neural networks (DNN), so the optimiza- tion of metrics can be relaxed to differentiable space. For example, taking predictions and labels as input, (Grabocka et al. 2019) approximates the outputs of surrogate losses and evaluation metrics by minimizing their L.2 distances. Moreover, recent work even involves the prediction networks into the surrogate loss learning by alternatively updating the loss and predictions, i.e., they train the surrogate losses after every epoch during training, then use the latest optimized losses to train prediction networks in the next epoch. For instance, (Grabocka et al., (019)) mainly focuses on the simple binary classification while LS-ED (Patel et 20200) chooses to adopt the surrogate losses in the post-tuning stage (fine-tuning the models learned by original losses) and achieves promising improvements. However, these methods often suffer from heavy computational consumption and do not perform well on large-scale challenging datasets.
"Correspondence Shan You <youshhn@sennet

---

Published as a conference paper at ICLR 2022
Pnod the same whole stuation is inceliilll time, thing
smaicvo by 1992, one crficcied andric
Ref sangeso, umiil 1992,
Spaccssn'ss coreteii P%)
Ronking vectees:
Ficllos [1. S. 10,
FCELos [6. 1, 8.
52.
FALEU
5,
Spearman's rank corveiations
eisrrioaoo: "BLLE)
PSEPOEL FBLEU) -0.575
Prekition
Evsieation rustiia
(a) Spearman's 's rank correlations
(b) Example neural machine translation task
Figure 1: (a) Our proposed ReLoss significantly improves the ranking correlations between losses and metrics on various tasks. (b) Taking neural machine translation task as an example, we sample 10 sentences from WMT16 RO-EN dataset, then measure the BLEU, cross entropy (CE) loss, and ReLoss with trained network and ground-truth references. Compared to the original CE loss, our ReL.oss obtains a stronger rank correlation.
Both manual and learned surrogate losses follow an exact recovery manner, namely, the surrogate losses should approximate the target metrics rigorously, and optimizing the surrogate loss is sup- posed to improve the evaluation metrics accordingly. However, this assumption does not always hold due to the approximation gap, bringing bias the optimization and leading sub-optimal re- sults. Instead of pursuing an exact recovery of the evaluation metric, we are reminded of the purpose of metrics, which is distinguish the performance of models. If model has smaller loss than the other model, its metric ought to be better. Nevertheless, current surrogate losses usually have weak relation with the evaluation metrics (e.g., CE Loss & BLEU in Figure (b)). Ideally, the surrogate loss should maintain strong relation of evaluation metric to all models.
In this paper, we leverage the ranking correlation as the relation between surrogate losses and eval- uation metrics. Then a natural question raises, if the loss functions only require accurate relative rankings to discriminate the models, why do we need to approximate the metrics exactly? In this way, we propose a method named Relational Surrogate Loss (ReLoss) to maximize this rank correlation directly. Concretely, our ReLoss directly leverages the simple Spearman's 's rank correla- tion (Dodge 2008) as the learning objective. By adopting differentiable ranking method, the ranking correlation coefficient can be maximized through gradient descent. Compared to exactly recovering the metrics, our correlation-basee optimization is much casier to learn, and our ReLoss, which is simply constructed by multi-layer perceptions, aligns well with the metrics and obtains significantly better correlations compared to the original losses. For example, the commonly used loss MSE in pose estimation only has 46.71% % Spearman's 's rank correlation coefficient with the evaluation metric PCK, while our ReLoss enjoys 84. 72% relative improvement (see Table and Figure I.
Our ReLoss generalizes well to various tasks and datasets. We learn ReLoss using randomly gener- ated data and pre-collected network outputs, then the learned losses are integrated into the training of prediction networks as normal loss functions (e.g., cross-entropy loss), without any further fine- tuning. Note that we use the same surrogate losses with the same weights in each task, and we find that it is sufficient to obtain higher performance. Compared to previous works, our method is much easier to optimize and enjoys significant efficiency and performance improvements. Extensive experiments the synthetic dataset and large-scale challenging datasets demonstrate our effective- ness. Moreover, our method outperforms the state-of-the- methods in human pose estimation and machine reading comprehension tasks. For example, human pose estimation task, our ReLoss outperforms the tfate-of-the-as method DARK (Zhang et al. (0200) by 0.2% on COCO test-dev set; on machine reading comprchension task, achieve new state-of-ther of--hh---ttrrrr performance on DuReader 2.0 test set, outperforming all the competitive methods, and even obtain 7.5% better ROUGE-L. compared to human performance.
RELATED WORK
Surrogate loss learning. Since most of the metrics in deep learning tasks are non-differentiable and deoopooomeoeable (e.g., accuracy, F1, AUC, AP, etc.), surrogate losses aim to approximate the metrics to make them differentiable using neural networks. (Grabocka et al. 2019) first proposes to learn surrogate losses by approximating the metrics of tasks through neural network, and the losses
2

---

Published as a conference paper at ICLR 2022
33"
Baak pundatis
www
Mecde wile
200 300 Valdakae insaphor
Zotallica
Domaliana
(b)
Figure 2: Visualization of our toy experiments on the synthetic dataset. (a) Visualization of out- puts of rpproxination-based surrogate loss and evaluation metric on the validation set. (b) Curves of Spearmans' rank correlations between surrogate losses and evaluation metric in the training of losses. (c) Evaluation curves of different losses in training, lower is better.
optimized jointly with the prediction model via bilevel optimization. (Patel et al. 2020 learns the surrogate losses via deep embedding where the Euclidean distance between the prediction and ground truth corresponds to the value of the metric. However, it is hard to obtain precise prediction by directly optimizing the surrogate loss with such strong constraint. We remind that the role of loss functions is to determine which model is better, but with the unavoidable existence of approximation gap, this deterninability does not always hold. In addition, these methods both train the surrogate losses alternately with prediction networks, resulting in noticeable efficiency and generability deduction compared to regular losses. In our paper, instead of only focusing on point-to-point recovery, which ignores the rankings between relative values of metries, we ease the optimization constraint by explicitly learning our ReLoss with rank correlation, and enjoy significant performance and efficiency improvements.
Differentiable sorting & ranking. Differentiable sorting and ranking algorithms (Adams & Zemel 2011 Grover al.) 2018 [londel et al., 2020 Petersen et al.) (02I)) can be used tramiin neural networks with sorting and ranking supervision. Recent approach (Blondel et al. 2020) proposes to construct differentiable sorting and ranking operators as projections onto the permutahedron, i.e. the convex hull of permutations, and using reduction to isotonic optimization. (Petersen et al. (021) proposes diffcrentiable sorting networks by relaxing their pairwise conditional swap operations. In this paper, can any of these differentiable ranking algorithms to generate differentiable ranking vectors, then directly optimize the rank correlation coefficient for the supervision of our surrogate losses. The algorithm in (Petersen et al. 2021) is adopted for better performance.
PRELIMINARIES
For given task with metric function MMy, 9). where and denote the predicted labels and ground-truth labels, respectively, its loss function Lly. 9) can formulated as:
C(y 9) = fly 9).
(1)
where f can be any function with output R
In this paper, we tend use a learned DNN (SDNN) with weights O, surrogate loss, i.c., L ÿ: 0.) = fuNN(y, ÿ; 0:).
The surrogate losses are learned with the networks' outputs and the corresponding metric values M(y i.e.
0 = arg min O,(((y, y: 0,), M(y ())
(3)
0,
where O, is the learning objective of surrogate loss. The prediction networks with weights On are then optimized by descending the learned surrogate losses Cly, ÿ: 0f), i.e.
0 arg min L(y, ÿ 0;)-
e..
Approximation-basee based optimization. To learn a surrogate loss w.r.t. a metric, an intuitive idea to approximate the metric's outputs, iee, learn the surrogate losses by minimizing the distances
3

---

Published as a conference paper at ICLR 2022
between the outputs of surrogate losses and their corresponding metrie values, which is adopted in previous works (Grabocka et al. 2019 Patel et al. 2020), their learning objective O is
O,(C(y, ÿ: , Myy 0) =I1 L(y. y: 0,) My 9) 112.
(5)
we call this optimization as approximation based optimization.
However, it is hard for DNN to fully recover the evaluation metric. We conduct toy experiments using a random weighted DNN with output E as an evaluation metric, and then the surrogate loss is learned using limited observations of the metric. As illustrated in Figure since approximating a random network with random inputs is challenging, the errors between surrogate loss learned by approximation-based optimization and metric values are noticeably large. In order to validate the effectiveness of losses in training. we then train the input data with metric or learned losses, as shown Figure (c), we illustrate the curves of metric values w.r.t. learned input data during training, and directly using metric as loss function obtains best metric value (lower is better), but the performance of input data using approximation based loss is getting worse.
4 LEARNING RELATIONAL SURROGATE LOSS
4.1 RELATION AS RANK CORRELATION
Based on the previous discussion, the prior works adopt an unnecessary constraint by enforcing the surrogate losses to fully recover the evaluation metrics. However, the loss function only needs to have the same ranking relation to the metrics, i.e., we just need to make the surrogate losses have the same ranking metrics. In this paper, obtain the relation between surrogate losses and evaluation metrics by using rank correlation as the learning objective, which we call correlation- based optimization.
The relation between surrogate losses and evaluation metrics is measured by ranking correlation, which is a statistic that measures the relationship between rankings of the same variable. ranking correlation coefficient measures the degree of similarity between two rankings and can be used to assess the relation's 's significance. If the surrogate loss fully correlates to the evaluation metric, the descent of loss value will always obtain better metric values.
Spearman's rank correlation. For optimization of surrogate losses, we use the most commonly used Spearman's 's rank correlation (Dodge) 2008). For two vectors and b with size n, the Spear- man's rank correlation defined as:
( b) = Cov(ra; rb)
(ra; ((ra))(r:: E(ro)) Std(ra)Std(ro)
(6)
Std(ra))Std(r)
where ra is the rank vector of a, Cov(ra, rbb) is the covariance of the rank vectors, Std(ra) denotes the standard derivation of ra-
4.2 LEARNING LOSSES BY MAXIMIZING RANK CORRELATION
Correlation- based optimization. We use Spearman's 's rank correlation as the objective to learn our surrogate losses, since the loss should have negative correlation w.r.t. the metric (higher is better), our objective is to minimize the Spearman's 's rank correlation coefficient, i.e.,
O,((ly ÿ M(y 9)) = (C(Cyy ÿr 0:), M(y ū))
Since the computation of rank vectors Fa and rb Eq.66 is not differentiable, we adopt one of the differentiable ranking methods (Petersen et al. 2021) to obtain differentiable ranking vectors, and empirically find that the errors in differentiable approximation is negligible and our learned correlation be very close the optimal value, i.e., O, -1.
As shown in Figure? 2 compared to approximation- based optimization, the surrogate loss learned by our correlation- based optimization obtains higher rank correlation and faster convergent speed. Besides, optimizing with our correlation-bast loss achieves significantly better performance than approximation--asse optimization, and is more stable than the original loss (evaluation metric).
4

---

Published as a conference paper at ICLR 2022
Learning with Gradient Penalty, In paper, loss to obtain its gradients. However, we find the first-order derivative of the learned ReLoss w.rt. the prediction changes rapidly since only constrain the correlation in Eq.(3), which result in either vanishing or exploding gradients. Nevertheless, in the optimization of networks, we want a loss with smooth gradients to train the networks steadily.
directly backward through the surrogate
Following Gulrajani et al. 2017 Patel et al.) 2020). we now propose an alternative way to smooth the gradients by enforcing the Lipschitz constraint. A differentiable function is I-Lipschitz if and only if it has gradients with norm at most everywhere, so we consider directly constraining the gradient norm of the loss's output w.r.t. its input. i.e.
Cpenttty (ll VyC(y. 9:0) ll2 1))
This penalty of gradients has been shown to enhance the training stability for generative adversarial networks (Gulrajani et al. 2017). Our objective of surrogate loss learning in Eq.(7) becomes
O,(C(y, y; 01), M(y. ū)) (s(((y) 9: M(y, 8)) Aeponatyy
we use À = 10 in our experiments.
4.3 PIPELINE
Now we illustrate how to learn our ReLoss. Different from previous works (Grabocka et al. 2019 Patel et al. 2020) which train the surrogate loss and prediction network alternatively as bilevel optimization, we want our surrogate loss to general as vanilla loss (e.g., cross-entropy loss). Since learn the surrogate loss with much weaker constraint, our surrogate loss can generalize better to the whole distribution of outputs and metric values to train the surrogate loss once for all, without further fine-tuning.
Our training strategy of surrogate loss is summarized in Algorithm [ The training data of surrogate losses is the combination of randomly generated data GR and the outputs of models Gas. Concretely, we design a random generator to produce random outputs and labels uniformly for GR. while for GM, we the intermediate checkpoints of the prediction networks trained by original loss to predict the outputs of train data. Each batch of training data is generated from GR or G with probabilities p and 1 p, respectively.
Algorithm Learning of surrogate losses.
Input: surrogate loss with random weights 0, batch size metric function M, data generators G and GR, sample probability p.
Output: learned surrogate loss with highest correlation.
1: while not converged do
2: = M M Lp =p 0:
3: for 1, I.. do
4:
generate a batch of predictions and ground- -ruth labels (yt, 9.) from GR with probability
or GM with probability 1 p:
5:
compute loss w.r.t. predictions and labels: 1, = L(vi, ū:: 01):
6:
compute metric: m4 MM ÿi
7:
compute Ini = (I) V C(y, g 0i) 2/ 11)2;
8:
L= LU (4) : M M {m,) Lp Lp 14p1
9:
end for
Epeattty + ΣN Lp 11: optimize 0, by descending Vo, (ps(L, M) ) + XCpenaly):
end while
13:
return learned surrogate loss with weights 0
Usage of learned ReLoss. The learned ReLoss can be fixed and then integrated into the training of prediction networks, i.e., we only change the loss function in training, without any modification on training strategy, network architecture, etc. Besides,,,, better performance if combined with the regular loss. In this case, the regular loss might act as a regularization term, and bring decent prior for the ReLoss to enhance the optimization.
emprically find that ReLoss would achieve
5

---

Published as a conference paper at ICLR 2022
Table 1: Rank correlations between loss function (descending order) and metrics (ascending order) in different tasks, higher is better.
Original loss
Spearman's 's (%)
Kendall's 's Tau (%)
Task
Metric
origin
ReLoss (ours)
origin
ReLoss (ours)
Classification
ACC
CE
95.66
98.40 (+2.74)
83.69
89.88 (+6.19)
Human Pose Estimation
PCK
MSE
46.71
86.04 (439.33)
33.04
69.00 (+35.96)
Machine Reading Comprehension
F1
CE
78.68
84.63 (45,95)
61.49
67.96 (46.47)
Neural Machine Translation
BLEU
CE
70.14
75.68 (+5,54)
65.37
70.17 (+4,80)
Table 2: Results on CIFAR- 10, LIFAR-100, and ImageNet datasets.
CE
ReLoss
Datasct
Model
Top-1 (%)
Top-5 (%)
Top-1 (%)
Top-5 (%)
CIFAR-10
ResNet-56
94.32 0.25
94.57 1: 0.08
CIFAR-100
ResNet-56
73.61 1 0.11
74.15 1: 0.14
ResNet-50
76.5
93.0
76.8
93.0
ImageNet
MobileNet V2
71,8
90.3
72.2
90.5
5 EXPERIMENTS
To fully experiment with the effectiveness and generability of our ReLoss, we conduct experiments on both computer vision and natural language processing tasks. In computer vision, we experiment on image classification and human pose estimation tasks, while in natural language processing, we experiment machine reading comprehension and neural machine translation tasks.
We first show the rank correlations original losses and our learned surrogate losses to the metrics in Table [ Spearman's 's (Dodge 2008) and Kendall's Tau (Kendall 1938) are two commonly used coefficients to measure the ranking correlations between two vectors. We can see that in all our ex- perimented tasks, our Rel achieves higher correlations compared the original losses. Notably, even for cross-entropy (CE) loss, which has been shown to align well with the misclassification rate, our surrogate loss still performs better on classification tasks with the metric accuracy (ACC).
5.1 COMPUTER VISION
Image classification. We conduct experiments on three benchmark datasets CIFAR-10, CIFAR- 100 (Krizhevsky al. 2009), and ImageNet (Deng et al., 2009). On CIFAR-10 and CIFAR- -00 datasets, we train ResNet-56 (He et al. 2016) with original CE loss and our surrogate loss and report their accuracies the test set with mean and standard derivation of runs. While on ImageNet dataset, we train ResNet-50 and MobileNet V2 (Sandler et al., 2018), their accuracies on validation set are reported. Notably, all experiments use the same surrogate loss with the same weights.
Table 2 shows the evaluation results. We can see that, though the original CE loss obtains a very high correlation (~ 0.96 in Table by integrating our surrogate loss with higher correlation, the performance can still be improved. Note that we use the same surrogate loss with fixed weights these three datasets, which means that our loss can generalize to different image classification datasets and gain the improvements with negligible additional cost.
Human pose estimation. Human pose estimation (HPE) aims to locate the human body and build body skeleton from images. It is difficult to precisely evaluate the performance of HPE since many features need be considered (e.g., the quality of body parts, the precision of each keypoints). As a result, many metrics are proposed for HPE. Percentage of correct keypoints (PCK) (Yang & Ramanan 2012) and Average Precision (AP) are two of the most commonly used ones. However, current methods usually adopt mean square error (MSE) to minimize the distance between predicted heatmap and target heatmap, which correlates weakly with the evaluation metrics.
In our experiments, we choose to approximate PCK @0.05 since it better reflects the quality of each keypoint, and our ReLoss achieves significant improvement on rank correlation compared to the original MSE loss. We use the most widely used large-scale dataset COCO (Lin et al.) 2014) to evaluate our performance, and the results are summarized in Table 3] We can see that, on validation set, our ReLoss significantly improves the baseline methods, and the AP75 improves the most since

---

Published as a conference paper at ICLR 2022
Table 3: Results of human pose estimation task on COCO dataset.
Method
Backbone
Input size
AP AP APTS AP"
AP" AR PCK@0.05
validation set
SimpleBaseline Xiao
2018
ResNet-50
256 192
70.4 88.6
78.3
67.1
77.2
76.3
85.0
SimpleBaseline
ReLoss
Re
256 x 192
71.9
80.0
68.0
77.9 77.3
86.1
HRNet (un et al.
2019)
HRNet-W32
256 x 192
74.4 90.5
81.9
70.8
81.0 79.8
86.7
HRNet + ReLoss
HRNet- W32
256 x 192
74,8 90.5
82,4
70.9
81.2 79.9
87.3
test-dev set
G-RMI (Papandreou et al.
2017
ResNet-101
353 x 257
64.9 85.5
71.3
62.3
70.0
69.7
SimpleBaseling (Xiao et al.
2018
ResNet-101
384 x 288
73.7
91.9
81.1
70.3
80.0
79.0
HRNet (unn et al.
2019
HRNct W48
384 x 288
75.5
92.5
83.3
71.9
81.5 80.5
DARK Zhang et al.
2020
HRNet-W W48
384 x 288
76.2
92.5
83.6
72.5
82.4
81.1
DARK ReLoss
HRNet-W4R 384 288
76.4 92.7
83.7
72.7
82.5
81.3
our ReLoss aligns PCK for better keypoint localization. On test-dev set, we integrate our ReLoss Loss
into state-of-the-ang method DARK Zhang et al. 2020) and achieve improvements all the metrics.
5.2 NATURAL LANGUAGE PROCESSING
The gaps between loss functions and evaluation metries natural language processing tasks severer since the tasks often use sentence-level evaluation metrics (e-g., BLEU and ROUGE-L) but adopt word-level cross-entropy loss in training.
Machine reading comprehension. The task of machine reading comprehension (MRC) aims to empower machines to answer questions after reading articles. Concretely, with given question, the models are required to locate segment of text from the corresponding reading passage, which most probably the answer. We use Fl score as the evaluation metric learn surrogate loss, and experiment on two typical MRC datasets SQuAD (Rajpurkar et al., 2016) and DuReader (He et al., 2018). SQuAD evaluates performance using F1-score, and DuReader uses ROUGE-L (Lin) 2004) and BLEU-4 (Papineni et al. 2002).
The evaluation results are summarized Table and Table 5 On DuReader 2.0 dataset, our ReLoss gains improvements on dev set, and achieves state-of-the-an performance on test set. On SQuAD 1.1 dataset, we also achieve improvements compared to the baseline method.
Table 4: Results of machine reading comprehension task on DuReader 2.0 dataset. t: reported by (He al. 2018). Method
ROUGE-L BLEU-4
F1
Table 5: Results on SQuAD
dev set
1.1 dataset compared with
MacBERT-hase (Cui et al. 2020)
51.4
50.3
53.9
BERT (Devlin et al.) 2018).
MacBERT base ReLoss
51.8
50.6
54.2
Method
F1
EM
MacBERT large Cui
2020
53.2
51.2
55.5
MacBERT- ReLoss
53.6
51.4
55.9
BERT-base
88.5
80.8
test
BERT-base ReLoss
88.8
81.3
BiDAF!
Seo al.
2016
39.2
31.9
BERT-large
90.9
84.1
Wang al. 2018
44.2
41.0
BERT-lagg ReLoss
91.4
84.6
MCR-Net-large (Peg al.
2021
50.8
49.2
Human Performance
57.4
56.1
MacBERT- large + ReLoss
64.9
61.8
Neural machine translation. Neural machine translation (NMT) aims to translate sentence from the source to the target language with an end-to-end neural model. The evaluation metric of NMT is BLEU (Papineni et al. 2002), which measures the n-gram overlap between the generated translation and the reference. We conduct experiments on the Non- Sonn-utoreggrssivee neural machine Translation (NAT) task, in which the model generates target words independently and simultaneously. Since the

---

Published as a conference paper at ICLR 2022
Table 6: Evaluation results of BLEU on Neural Machine Translation task. We report the perfor-
mance of our methods on the WMT16 EN-RO dataset. Transformer denotes the auto-regressive model. * denotes the performance that we reproduced using the public code.
Original loss
ReLoss EN-RO
ReLoss RO-EN EN-RO
Model
Speed
EN-RO
RO-EN
EN-RO
RO-EN
RO-EN
Transformer Vassann et al.
2017)
1.0x
32.88
33.94
NAT-Base (uu al. 2017)
15.6x
29.24
28.97
30.07 +083
29.68 -0.7
29.93 +0.09
29.61 +064
BoN-L](N22) (Shao et al.
2021
15.6x
30.76
30.46
30.96 40:20
30.74 -0.28
30.88 40.12 30.78 40.32
output of NAT cannot be properly evaluated through word-level cross-entropy loss due the multi- modality problem in language, the correlation between cross-entropy loss and translation quality is weak, limiting the NAT performance.
Table the NAT evaluation results on WM EN-RO and RO-EN datasets. We conduct experiments based on the NAT-base and strong baseline BoN-L; (N-2) (Shao et al.) 202I), which introduce the BoN loss to fine tune NAT-base by modeling the bag of ngrams in the sentence. We
integrate ReLoss into the baseline methods NAT-base (Gulrajani et al. 2017) and BoN-L, (N=2), and the evaluation results show that our ReLoss can improve both of them.
5.3 ABLATION STUDIES
Compare with LS-ED. Prior work LS-ED (Patel et al. 2020) aims to post-tune the scene text recog- nition (STR) model using surrogate loss, which is learned with approximation--asen optimization. In order to compare our method with LS S-ED, we conduct experiments on the same settings. Follow- ing LS-ED, learn the surrogate loss using edit distance, then fine-tune the trained model using our learned loss (without using original loss for fair comparisons). The results in Table 7 show that our ReLoss significantly outperforms the baselines CE and LS-ED. Note that we only train our ReLoss once then integrate it into training, indicating that our loss is more efficient and general.
Table 7: Evaluation results on scene text recognition task comparing with CE and LS-ED, The reported metrics are accuracy (ACC, higher is better), normalized edit distance (NED, higher is better), and total edit distance (TED, lower better).
1ACC (%)
1ED
TED
Test datasct
CE
LS-ED
ReLoss
CE
LS-ED
ReLoss
CE
LS-ED ReLoss
IIT-5K (Mishra et al.
2012)
87,500
87.933
87.700
0.961
0.963
0.961
722
645
667
SVT (Wang al. 2011
87.172
86.708
87.481
0.952
0.954
0.957
180
163
156
ICDAR'SS 03
Lucas et al.
2005
94.302
94.535
94.579
0.979
0.981
0.982
110
99
98
ICDAR'13 Karatzas et al. 2013
92.020
92.299
92.709
0.966
0.979
0.981
137
108
101
ICDAR'15 Karatzas et al. 2015
78.520
78.410
78.355
0.915
0.915
0.916
868
837
845
SVTP (Phan et al.] 20133)
78.605
79.225
80.310
0.912
0.913
0.915
346
333
316
CUTE Risnumawan et al. 2014
73.171
74.216
75.958
0.871
0.875
0.884
224
219
195
Wins
5
6
Average
84.470
84.761
85.299
0.937
0.940
0.943
370
343
340
Transferability of learned ReLoss. In all our experiments, we use the same surrogate loss in each task. If we learn different surrogate losses on specific datasets, would the performance be better? To validate this, we conduct experiments to train ReLoss independently on each dataset, as shown in Table 9 The ReLoss transferred from ImageNet dataset performs similar to the consistent ReLoss learned on corresponding datasets. It might be because we train the ReLoss using predicted and randomly generated data, and it is sufficient to cover different distributions of datasets on image classification. We also experiment EN-RR and RO-EN, and the results using either of them to train the networks are similar, which demonstrates that the learned ReLoss is language- indeeeedennd and can bring similar improvements on the other translation direction.
NMT task. As shown in Table 6 we train ReLoss on both
Comparison of approximation- and our correlation-based optimization. In Figure 2 compare our ReLoss with approximation based methods on the synthetic dataset. Now we further conduct experiments image classification task to show our superiority. Concretely, we learn the
8

---

Published as a conference paper at ICLR 2022
surrogate losses with the same architecture using approximation- based correlation-based opti- mization, then integrate them to train networks on CIFAR datasets, As shown in Table our ReLoss with correlation-base based optimization obtains the highest accuracies compared approximation-base. loss. Note that the standard derivations of accuracies of approximation-base. loss much larger than CE loss and correlation-based loss; this might be because the imprecise rankings and gradients in approximation-base loss weaken the training stability.
the CE loss and
Table 8: Results of different optimization methods on image classification task.
Rank correlation (%)
ACC (%)
Loss function
Spearman's
Kendall's ' Tau
CIFAR-10
CIFAR-100
Cross Entropy
95.66 65.66
83.69
94.32 0.25
73.61 + 0.11
ReLoss (approximation based)
91.71
76.03
94.11 0.42
73.88 0.32
ReLoss (correlation-aase -bsed)
98.40
89.88
94.57 0.08
74.15 = 0.14
Integrating ReLoss with / without regular losses. We empirically find that the prediction networks using our ReLoss converge very fast at the beginning of training, then the performance will increase very slowly even get worse, the experiments on synthetic dataset show the similar trend (see Figure 2 (c)). A possible reason is that there exist some data points that surrogate losses can not predict accurately, making the optimization fall into local minima. For better performance, we use the regular loss as regularization term to help the surrogate losses jump out local minima. We conduct experiments to show the differences by integrating ReLoss with or without regular loss. As summarized in Table 10, the performance drops if not adding regular loss in training, showing that the regular losses can bring a decent prior for ReLoss to achieve better performance.
Table 9:
Comparison of transferred Rel.oss and consistent ReLoss.
Table 10: Evaluation results w/ or w/o regular losses.
ACC (%)
Dataset
ACC (%)
Dataset
w/ regular loss
w/o regular loss
transferred
consistent
CIFAR-IO
94.57 + 0.08
93.82 1: 0.32
CIFAR-10
94.57 1 0.08 94.61 0.12
CIFAR-100
74.15 = 0.14
73.91 = 0.22
CIFAR-100
74.15 + 0.14 74.12 0.09
ImageNet
76.8
75.9
5.4
COMPLEXITY ANALYSIS
Denoting the training iterations of surrogate losses as Ti, the training epochs and iterations in each epoch of prediction networks are F., and T.. respectively, the runtime complexity of our ReLoss O + Fm x T. For comparison, the runtime complexity of regular loss is O Em x Tm)). ) while for previous surrogate loss learning method (Grabocka al. 2019), it trains the surrogate losses after every iteration and has runtime complexity of O x Ti + Em x Tm). Note that our additional cost OTT)) of learning ReLoss costs only 0.5 GPU hour on image classification with a single NVIDIA TITAN Xp GPU, and we only need to train ReLoss once for each task, reducing much computational cost compared to previous works.
6 CONCLUSION
As proxy of the evaluation metric, loss function matters in machine learning since it controls the optimization of networks. However, it is often hard to design a loss function with strong relation to the evaluation metric. In this paper, we aim address this problem by learning surrogate losses using deep neural networks. Unlike previous works that pursue an exact recovery of the evaluation metric, we reminded of the essence of the loss function and evaluation metric, which is to distin- guish the performance of models, and show that directly maximizing the rank correlation between surrogate loss and evaluation metric can learn better loss. How design and learn a more robust and general surrogate loss would be a valuable aspect to improve this work.
ACKNOLEDGEMENTS
Chang Xu was supported by the Australian Research Council under Project DP210101859 and the University of Sydney SOAR Prize.

---

Published as a conference paper at ICLR 2022
REFERENCES
Ryan Prescott Adams and Richard $ Zemel. Ranking via sinkhorn propagation. arXiv preprint arXiv: 1925, 2011.
Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. Fast differentiable sorting and ranking. In International Conference on Machine Learning, pp. 950-959. PMLR, 2020.
Djó Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07233 2015.
MMPose Contributors. Openmmlab pose estimation toolbox and benchmark, www.hnaak..tttpss////gg....
com/ open - www.am/mmmmos.eeemmmmmmmmm 2020.
com/open-
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. Revisiting pre- trained models for chinese natural language processing. arXiv preprini arXiv:2004. 13922, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0488 2018.
Terrance Deeries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint Stt.::1700.045335 2017.
Yadolah Dodge. The concise encyclopedia of statistics. Springer Science & Business Media, 2008.
Josif Grabocka, Randolf Scholz, and Lars Schmidt-Thieme Learning surrogate losses. arXiv preprint urXiv:1905.10100 2019.
Aditya Grover, Eric Wang, Aaron Zweig, and Stefano Ermon. Stochastic optimization of sorting networks via continuous relaxations. In International Conference on Learning Representations, 2018.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. Non- autoregressive neural machine translation. CoRR, abs/1711.0228., 2017. URL http://
arxiv. sexiv.org/abe. wbw.oorg//bbbbb.... 1711. 02281
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 7769-9779,,. 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, et Dureader: a chinese machine reading comprehension dataset from real- world applications. In Proceedings of the Workshop Machine Reading for Question Answering. pp. 37-46, 2018
Geoffrey F. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/ 15s/1503.0253.. 2015. URL htttttttp//atttii
argg/bs.. 1503..... 02531
Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez i Bigorda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, and Lluis Pere De Las Heras. Icdar 2013 robust reading competition. In 2013 12th International Conference Document Analysis and Recognition, pp. 1484-1493. IEEE, 2013.
Dimosthenis Karatzas, Lluis Gomez -igordaa Anguelos Nicolaou, Suman Ghosh, Andrew Bag- danov, Masakazu Iwamura, Jiri Matas, Lukas Neumann. Vijay Ramaseshan Chandrasekhar, Shi- jian Lu, et al. Icdar 2015 competition on robust reading. In 2015 13th International Conference Document Analysis and Recognition (ICDAR), pp. 1156-11 1 160. IEFF, 2015.
10

---

Published as a conference paper at ICLR 2022
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-99) 1938.
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Jian Su, Xavier Carreras, and Kevin Duh (eds.). Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016. Austin, Texas, USA, November 1-4, 2016, pp. 1317- 1327. The Association for Computational Linguistics. 2016. doi: 10.18653/v1/d16-1 10066-11330 1139. URL
https:://oi. wrw/oo.................oo....... 11166//////166111331
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
2009.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Simon M Lucas, Alex Panaretos, Luis Sosa, Anthony Tang, Shirley Wong, Robert Young, Kazuki Ashida, Hiroki Nagai, Masayuki Okamoto, Hiroaki Yamamoto, al. ledar 2003 robust reading competitions: entries, results, and future directions. International Journal of Document Analysis and Recognition (LJDAR), 7(2-3):105-122, 2005.
Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Pro- ceedings of the 18th ACM international conference on Multimedia, pp. 1485-1488, 2010.
Anand Mishra, Karteek Alahari, and CV Jawahar. Scene text recognition using higher order lan- guage priors. In BMVC-British Machine Vision Conference. BMVA, 2012.
George Papandreou. Tyler Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, Chris Bre- gler, and Kevin Murphy. Towards accurate multi pose estimation in the wild. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 4903-4911, 2017.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA pp. 311-318. ACL, 2002. doi: 10.311551073333.177311
URL httpt:tt////m. btttps://aclanttoll.oo
org PO2 -000
Yash Patel, Tomáš Hodañ, and Jifí Matas. Learning surrogates via deep embedding. In European Conference on Computer Vision, pp. 205-221. Springer, 2020.
Wei Peng, Yue Hu, Jing Yu, Luxi Xing, Yuqiang Xie, Zihao Zhu, and Yajing Sun. Mcr-net: A multi-step co-interactive relation network for unanswerable questions on machine reading com- prehension. In ICASSP 202/-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7818-7822. IEEE, 2021.
Felix Petersen, Christian Borgelt, Hilde Kuehne, and Oliver Deussen. Differentiable sorting net- works for scalable sorting and ranking supervision. arXiv preprint arXiv:2 105.0:2055 2021.
Trung Quy Phan, Palaiahnakote Shivakumara, Shangxuan Tian, and Chew Lim Tan. Recogniz- ing text with perspective distortion in natural scenes. In Proceedings of the IEEE International Conference on Computer Vision, pp. 569-576, 2013.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016.
Anhar Risnumawan, Palaiahankote Shivakumara, Chee Seng Chan, and Chew Lim Tan. A robust arbitrary text detection system for natural scene images. Expert Systems with Applications, (18):8027-8048. 2014.
11

---

Published as a conference paper at ICLR 2022
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-45200 2018.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. arXiv preprint 16xiv:6611.0000 2016.
Chenze Shao, Yang Feng, Jinchao Zhang, Fandong Meng, and Jie Zhou. Sequence-level training for non-autoregressive neural machine translation. CoRR, abs/2106.08122 2021. URL https:
//axxiv..rr//bb//21166 08122
Xiu Su, Shan You, Tao Huang, Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. Locally free weight sharing for network width search. In International Conference on Learning Repre- sentations, 2020.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693-5703, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Kai Wang, Boris Babenko, and Serge Belongie. End-to-end scene text recognition. 201 Inter- national Conference Computer Vision, pp. 1457-1464. IEEE, 2011.
Yizhong Wang, Kai Liu, Jing Liu, Wei He, Yajuan Lyu, Hua Wu, Sujian Li, and Haifeng Wang. Multi-passage machine reading comprehension with poss-passage answer verification. arXiv preprint 1805 1805.02220, 2018
Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking.
In Proceedings of the European conference computer vision (ECCV). pp. 466 481, 2018.
Yi Yang and Deva Ramanan. Artieulated human detection with flexible mixtures of parts. 1EEE transactions on pattern analysis and machine intelligence, 35(12):888--2890 2012.
Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1285-1294, 2017.
Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, and Changshui Zhang, Greedynas: Towards fast one-shot nas with greedy supernet. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1999-2008 2020.
Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, and Ce Zhu. Distribution-awrr coordinate repre- sentation for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7093-7102, 2020.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 NEURAL ARCHITECTURES OF OUR SURROGATE LOSSES
Image classification. In order to make the learned surrogate loss generalize to all the classification tasks which take accuracy as the metric, we use the logits ypos with positive labels as the input of our neural network, which is the same as cross-entropy loss, and the outputs of the surrogate loss are simply computed through 4-layer perceptions with intermediate activations, i.e.,
(10)
we use ELU activation CCevvrt et al. 2015) for stable gradients since it is C continuous.
Machine reading comprehension and neural machine translation. Since the evaluation metrics of MRC and NMT are computed by a sequence of texts, based on the architecture in image clas- sification, we use additional self-attention mechanisms (Vaswani et al.] 2017) extract sequential information.
Human pose estimation. Given the prediction heatmap and target heatmap, the original MSE loss is used to minimize the distance between these two heatmaps. Our ReLoss first embeds these two heatmaps into two hidden vectors, then computes the MSE loss between them as the final loss.
A.2 TRAINING STRATEGIES
Surrogate loss. We train the surrogate losses using Adam optimizer with fixed learning rate of 0.01, and the weight decay is set to le-4.
Image classification. The reported models trained using the same code, with the only difference in the loss function. On CIFAR- 10 and CIFAR- 100 datasets, we train ResNet-20 for 200 epochs with an initial learning rate of 0.1, which decays 0.1 100th and 150th epochs, the batch size is set to 128 with cutout Vries & Taylor 2017) data augmentation, we run each experiment 5 times with different random seeds and report their mean accuracy with standard derivation. On ImageNet, we follow the same training strategy as in torchvisior]] (Marcel & Rodriguez 2010). Concretely, we train ResNet-50 for 120 epochs with an initial learning rate of 0.1, step learning rate scheduler which decays 0.1 every 30 epochs is adopted. While for MobileNet V2, we train it for 300 epochs with 4c-5 weight decay, a cosine learning rate scheduler is adopted with an initial learning rate of 0.045, The batch sizes for ResNet-50 and MobileNet V2 are both set to 32, We use SGD optimizer with 0.9 momentum on all datasets. Note that all the experiments use the same surrogate loss with the same weights.
Human pose estimation. We train ResNet-50, HRNet- -32, and DARK-HRNet- W48 following the default configurations in MMPose (Contributors, 2020). Coneretely, the models are trained with Adam optimizer for 210 epochs, and a step learning rate scheduler is adopted with initial value 5e- 4, which decays 0.1 at 170th and 200th epochs. The total batch sizes of 8 GPUs with input size 256 x 192 and 384 x 288 are 512 and 256, respectively.
Machine reading comprehension. We train the networks using Adam optimizer with weight decay 0.01, a linear learning rate strategy which warmups 0.1 epoch and decays 2 epochs is adopted. On DuReader 2.0 dataset, the batch size is set to 32; we train MacBERT-base and MacBERT-large with learning rates 3e-5 and 2c-5, respectively. On SQuAD 1.1 dataset, we train BERT-base and BERT-large with batch sizes and 2, and the learning rates are 5e-5 and le-5, respectively.
Neural machine translation. For WMT16 EN-RO, we use the WMT 2016 corpus, which consists of 610K sentence pairs for training. We take news-dey-201e and news-test-2016 as development and test sets. We learn joint BPE model with 32K operations and share the vocabulary for source and target languages. As knowledge distillation (Hinton et al. 2015 Kim & Rush 2016) has been proven to be crucial for training NAT models, we first train an auto-regressive transformer model (Vaswani ctt al., 2017) as the teacher and then apply sequence-level knowledge distillation to construct the corpus for training NAT models. The NAT-base takes the same architecture as the base transformer model except that we modify the attention mask of the decoder for not masking the future tokens. We use target length predictor predict the length of the target sentence. We golden length
13

---

Published as a conference paper at ICLR 2022
during the training and the predicted length during the inference. For training ReLoss, only use the outputs of the model. For NAT-base, the number of training steps is 200K. We select the checkpoint based on the validation set. We add ReLoss with a factor of I on the CE loss to fine-tune the NAT-base for 10k steps with a batch size of 32 and a fixed learning rate of 1e-5. For BoN-L1 reproduce the results using the public repo R. We combine ReLoss with the BoN Loss to fine-turn the model for 3K steps with a batch size of 512, which keeps the same with the BoN-LI (N=2).
(N=2) (Shao et al. 2021),
A.3 MORE ABLATION STUDIES
Performance of losses with different rank correlations. To show the influence of the rank corre- lations on the performance, we choose the surrogate losses with different rank correlations to train models on CIFAR-10 10 dataset. The results in Figure 3 (a) clearly show that the increase of rank cor- relation boosts the performance, and the losses with lower rank correlations will disturb the training of networks.
Performance of losses with different network capacities. Our surrogate losses are constructed with fully connected layers. To validate the influence of capacities of loss model, conduct exper- iments to learn losses on different capacities on CIFAR-10 dataset. As shown in Figure 3 (b), our original loss model has 33.4K parameters, we adjust the number of layers hidden dimensions change the network capacity. The results show that the performance our ReLoss gets saturated on small number of parameters. With this small network capacity, its computational cost is negligible in the training of prediction networks.
0.56
0.8
Rank cverrolatio
Ancuracy (%)
0.50
0.85
10:3
91
0.5
0.75
31.8
31.6
0.7
0.75
93.4
0.05
Ton dim
41.
0.7
0.0
Bank correlation
Params (xI0F)
(a)
(b)
Figure 3: (a) Evaluation results CIFAR-10 using ReLoss with different rank correlations. (b)
Evaluation results on CIFAR- 10 using different capacities of surrogate losses.
Table 11: Compare with alternate training on image classification.
ACC (%)
Loss function
CIFAR-10
CIFAR-100
Image
Cross Entropy
94.32 = 0.25
73.61 0.11
76.4
ReLoss
94.57 1 0.08
74.15 1 0.14
76.8
ReLoss (alternate training)
94.65 1 0.21
74.18 1 0.31
76.9
The outputs of prediction networks change during the training. For surrogate losses using approximation-baal optimization, it is hard to obtain accurate predictions for all the possible predictions, so the previous works (Grabocka et al. 2019 Patel et al.) 2020) learn surrogate losses alternately with prediction networks during training. In our paper, show that our ReLoss achieves higher performance using only pre-trained surrogate losses. We further conduct experiments to train our ReLoss alternately with prediction networks. As shown in Table the alternate training obtains higher accuracies but gets more unstable on CIFAR datasets. We think this might be because the update of weights in surrogate losses will disturb the gradients,
Compare with alternately learning losses.
14

---

Published as a conference paper at ICLR 2022
while our ReLoss without alternate training provides the same gradients for the same predictions and labels, thus more stable. Since the performance improvements of alternate training are marginal, we use pre- pre-trained losses for better generalization and efficiency.
Effeet of gradient penalty. Our paper aims to stabilize the gradients of surrogate losses by intro- ducing gradient penalty regularization in loss learning. We conduct experiments to show the effec- tiveness of the gradient penalty. As summarized in [T2 we train the models on CIFAR datasets using the surrogate losses with without gradient penalty. The results show that the ReLoss without gradient penalty performs poorly compared to the one with gradient penalty and even the original loss, although obtains a good rank correlation. It indicates that the regularization of gradients of the surrogate losses is necessary and contributes a lot to the performance.
Table 12: Results of ReLoss with or without gradient penalty.
Rank correlation (%)
ACC (%)
Loss function
Spearman's 's
Kendall's 's Tau
CIFAR-10
CIFAR-100
Cross Entropy
95.66
83.69
94.32 1 0.25
73,61 = 0.11
ReLoss (w/o gradient penalty)
98.31
89.56
94.28 + 0.31
73.03 = 0.26
ReLoss (w/ gradient penalty)
98.40
89.88
94.57 0.08
74.15 = 0.14
As for the weight À in Eq. 9. we empirically find that this regularization is casy to achieve since our learning objective of correlation is weak. We have tried different values of X, the term of gradient penalty is always very small (~ (=le)3), so we directly follow previous work (Gulrajani et al. 2017) and use À = 10.
Robustness of performance in multiple independent runs. To validate the robustness of the training of ReLoss, conduct experiments to train the ReLoss multiple times independently. and leverage these learned surrogate losses to train ResNet-56 on CIFAR datasets. As the results summa- rized in Table 13] the accuracies of multiple runs similar (with low standard variance), showing that our ReLoss can obtain stable results. We believe that our ReLoss is easy to learn, and the regu- larization term of gradient penalty could obtain stable gradients of surrogate losses w.r.t. the logits. As result, the performance would be robust.
Table 13: Results of ReLoss CIFAR datasets in multiple independent runs.
Rank correlation (%)
ACC (%)
Number
Spearman's
Kendall's Tau
10
CIFAR-100
1
98.36
89,91
94,49 + 0.06
74.05 = 0,09
2
98.47
89.78
94.51 + 0.08
74.12 = 0.11
3
98.43
89.86
94.43 1 0.10
74.09 0.08
4
98.42
89.75
94.59 + 0.08
74.15 1 0.10
5
98.38
89.71
94.55 + 0.07
74.11 = 0,06
mean + std
98.41 + 0.04
89,80 + 0,08
94.51 = 0.06
74.10 = 0,03
Compare with rank-based classification loss.
Our method adopts differentiable sort algo-
rithms [Blondel et al.) 2020; Petersen et al. 2021) to train the surrogate loss. However, Blondel et al. 2020 proposes rank-based classification loss to directly calculate the L1 error between pre- dicted soft ranks and target ranks of top-1 elements, i.e.
4i = |ppos Nl.
(II)
where Tpos denotes the predicted soft ranks of Ypos and N the number of classes.
We train the above rank-based classification loss (RankLoss) on CIFAR-10, CIFAR-100, and Im- ageNet datasets with the same models and strategies in our paper. From the results summarized in Table 14 we can see that the accuracies obtained by RankLoss are significantly lower than the original CE loss and our ReLoss. Besides, on datasets with more classes (100 and 1000 on CIFAR- 100 and ImageNet, respectively), it only obtains slightly better accuracies than random guess. One possible reason that RankLoss only focuses on the ranks of positive elements of logits, lacking supervision the remained elements. As result, the network receives little information to verge on datasets with large numbers of classes. In contrast, our ReLoss learns from the evaluation metrics and supplies better information for discriminating models, thus achieves better accuracy.
15

---

Published as a conference paper at ICLR 2022
On the other hand, RankLoss is hard to generalize different tasks since it needs to design different loss functions for different metrics. Meanwhile, cannot be applied to regression tasks. As a result, believe it is necessary learn metric-oriented surrogate loss using neural network rather than directly applying differentiable ranking operators as the loss.
Table 14: Compare with rank-based classification loss.
Dataset
Model
CE Loss (%)
RankLoss (%)
ReLoss (%)
CIFAR-10
ResNet-56
94.32
82.77
94.57
CIFAR-100
ResNet-56
73.61
74.15
ImageNet
ResNet-50
76.5
GPU memory and training cost compared original loss. We report the memory consumption and training speed of ResNet-56 CIFAR-10 and CIFAR-I00 datasets in Table 15 Since our ReLoss only has ~ 0.03M parameters, the memory and training time increments are negligible compared to the much larger consumptions of models (e.g., ResNet-50 has 25.6M parameters).
Table 15: Comparisons of GPU memory and training cost.
Training speed (batches / second)
Dataset
Loss function
GPU memory (M)
ACC (%)
CE
618.46
14.29
94.32
CIFAR-10
ReLoss
618.68
14.29
94.57
CE
618.55
14.27
73.61
CIFAR-I00
ReLoss 618.76
14.27
74.15
A.4 VISUALIZATION CONVERGENCE CURVES IN TRAINING
We visualize the convergence curves of CE loss and our surrogate loss in Figure our loss obtains higher validation accuracies over the whole training procedure.
We can see that
Dee
ACC D4)
18
Specia
Epeddo
Spocha
(a) CIFAR-10
(b) CIFAR-100
ImageNet
Figure 4: Convergence curves (validation accuracies) of CE loss and loss on CIFAR-IO, CIFAR- 100, and ImageNet datasets. The data is smoothed using a moving average with factor 0.25. Zoom
up to view better.
16


Reviews:
[{"review_id": "bwsuQFdRst3", "reviewer": "Reviewer_eBZN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The study field of this paper is important. According to the authors’ experiments in Table 1, the original manually-designed losses do not align well with the evaluation metrics on some tasks. The proposed ReLoss can improve those ranking correlations a lot. \n\n- The proposed method can be widely applied to NLP and CV tasks, and its benefits seem significant according to the experiments. E.g., on human pose estimation task, ReLoss outperforms the state-of-the-art method with only replacement of loss function.  \n\nMinor Weakness: \n\n- How much additional costs if we train a model using ReLoss? How much time does it cost to train ReLoss? Would ReLoss increase GPU memory or training time in training? This is related to the practical feasibility of the proposed loss.  \n\n", "summary_of_the_paper": "- This paper proposes a relational surrogate loss learning method (ReLoss) inspired by the fact that the evaluation metric and loss are used to distinguish whether one model is better or worse than another. \n\n- This paper provides extensive experiments that demonstrate the effectiveness of the proposed method. The performance and efficiency compared to existing surrogate loss methods are significant, and the performance compared to original losses is seem to be significant on various tasks. \n", "summary_of_the_review": "I think the proposed method is well-principled and provides meaningful improvements on various tasks. I would like to lean on the positive side for this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Yl_8R4EbmuH", "reviewer": "Reviewer_Y6nu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed approach is well motivated and makes sense. The problem study here is also essential and could be of interest to a large audience, as the loss function is fundamental in almost all machine learning tasks. \n\nMeanwhile, the techniques are simple but effective. Without doubts, the proposed correlation-based optimization that introduces differentiable Spearman's correlation coefficient has looser constraints than approximation-based methods. Also, I believe the gradient penalty is necessary to keep stable convergence and fixed training strategies, since the magnitude of gradients may changes w.r.t. the initialization and randomness.\n\nExperiments are sufficient and convincing. For example, the ReLoss gains noticeable improvements over baselines and outperforms the state-of-the-art method on COCO keypoint dataset.\n\nHowever, I do have some minor concerns in this paper: \n\n(a) The surrogate losses can be learned with the universal method (ReLoss) without expertise on loss function design, but it seems to introduce a new problem of confirming the architecture of losses w.r.t the evaluation metrics. \n\n(b) Approximation errors always exist as the surrogate loss cannot fully recover the evaluation metrics in the whole distribution. In this paper, the authors adopt the original loss as a regularization term to alleviate this problem, but the superiority of ReLoss on rank correlation may be weakened.", "summary_of_the_paper": "The authors introduce a relational surrogate loss learning method (ReLoss) for replacing the original losses. The rationale and intuition behind are well-grounded. Experiments on various tasks and ablation studies prove the validity.", "summary_of_the_review": "The paper proposes a method to learn surrogate losses, compared to related works, the proposed method gains significant improvements on various datasets. The originality and significance are clearly above the bar, though there still remain some minor concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "REtqljCr_xf", "reviewer": "Reviewer_L49G", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* The idea is novel and interesting. Loss functions matter a lot to the performance, while the current hand-craft losses often align poorly to the performance, surrogate loss learning is a way to alleviate this issue. Instead of directly approximating the evaluation metrics as previous methods, this paper proposes a new learning method by revisiting the purpose of loss functions, which is to distinguish the performance of models. Hence, the authors aim to learn the surrogate losses by making the surrogate losses have the same discriminability as the evaluation metrics. The idea is straightforward and is easy to implement by using ranking correlation as an optimization objective.\n* To obtain stable gradients, the authors involve a gradient penalty regularization term by enforcing 1-Lipschitz, which is effective according to the experiments in Appendix.\n* The experimental results look good. By keeping the same backbones and training strategies, the proposed ReLoss gains improvements on various tasks, including CV and NLP tasks, and outperforms the state-of-the-art methods on human pose estimation and machine reading comprehension tasks.\n* The comparisons are sufficient to show the proposed method's superiority. The authors conduct experiments on the synthetic and large-scale benchmark datasets, and gain significant improvements in performance and efficiency compared to existing surrogate loss learning methods.\n \nCons:\n* The ReLoss is learned by optimization, which means the qualities of losses may be different in multiple runs. How many times did the authors train the ReLoss to obtain the performance reported in the paper? It would be better to conduct experiments to show the performance of surrogate losses learned in multiple independent runs.\n", "summary_of_the_paper": "This paper proposes a surrogate loss learning method named ReLoss. The ReLoss learned by maximizing the relation between surrogate losses and evaluation metrics is used to replace the original losses. Extensive experiments on computer vision (CV) tasks (image classification, pose estimation, and scene text recognition) and natural language processing (NLP) tasks (machine reading comprehension and translation) are provided, showing the benefits.", "summary_of_the_review": "In summary, the reviewer thinks the paper gives an effective way to obtain better losses, the experiments are well-conducted, and even improve the state-of-the-art methods, which would be helpful to the community. I would recommend it for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an approach to learn the surrogate loss for complex prediction tasks where the task loss is non-differentiable and non-decomposable. The novelty of the approach is to rely on differentiable sorting, optimizing the spearman correlation between the true loss and the surrogate. This leads to a pipeline that is simpler to integrate to existing works than approaches that try to learn a differentiable approximation to the task loss, and to better experimental results.

The paper is well written and the approach clearly presented. The reviewers liked the simplicity of the approach and the promising experimental results on a variety of challenging tasks (human pose estimation and machine reading).
Decision: Accept (Poster)