Learning Label Encodings for Deep Regression
Authors: Deval Shah
Tor M. Aamodt
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Regression
Label Encoding

Abstract: Deep regression networks are widely used to tackle the problem of predicting a continuous value for a given input. Task-specialized approaches for training regression networks have shown significant improvement over generic approaches, such as direct regression. More recently, a generic approach based on regression by binary classification using binary-encoded labels has shown significant improvement over direct regression. The space of label encodings for regression is large. Lacking heretofore have been automated approaches to find a good label encoding for a given application. This paper introduces Regularized Label Encoding Learning (RLEL) for end-to-end training of an entire network and its label encoding. RLEL provides a generic approach for tackling regression. Underlying RLEL is our observation that the search space of label encodings can be constrained and efficiently explored by using a continuous search space of real-valued label encodings combined with a regularization function designed to encourage encodings with certain properties. These properties balance the probability of classification error in individual bits against error correction capability. Label encodings found by RLEL result in lower or comparable errors to manually designed label encodings. Applying RLEL results in $10.9\%$ and $12.4\%$ improvement in Mean Absolute Error (MAE) over direct regression and multiclass classification, respectively. Our evaluation demonstrates that RLEL can be combined with off-the-shelf feature extractors and is suitable across different architectures, datasets, and tasks. Code is available at \url{https://github.com/ubc-aamodt-group/RLEL_regression}. 

Published as a conference paper at ICLR 2023
LEARNING LABEL ENCODINGS FOR DEEP REGRESSION
Deval Shah & Tor M. Aamodt
Department of Electrical and Computer Engineering University of British Columbia, Vancouver, BC, Canada deeaalsah, aamodt @ece. ca
ABSTRACT
Deep regression networks are widely used to tackle the problem of predicting con- tinuous value for given input. Task-specialized approaches for training regression networks have shown significant improvement over generic approaches, such as direct regression. More recently, a generic approach based on regression by binary classification using binary-encodec labels has shown significant improvement over direct regression. The space of label encodings for regression large. Lacking heretofore have been automated approaches to find a good label encoding for a given application. This paper introduces Regularized Label Encoding Learning (RLEL) for end-to-end training of an entire network and its label encoding. RLEL provides a generic approach for tackling regression. Underlying RLEL is our observation that the search space of label encodings ciently explored by using a continuous search space of real-valued label encodings combined with regularization function designed to encourage encodings with certain properties. These properties balance the probability of classification error in individual bits against error correction capability. Label encodings found by RLEL result in lower or comparable errors to manually designed label encodings. Applying RLEL results in 10.9% and 12.4% improvement in Mean Absolute Error (MAE) over direct regression and multiclass classification, respectively. Our evalu- ation demonstrates that RLEL can be combined with off- off-the-sheef feature extractors and is suitable across different architectures, datasets, and tasks. Code is available at
be constrained and effi-
com /uc- aamodt SrriitggroooRRRRRR RLEL, / ion
INTRODUCTION
Deep regression is an important problem with applications in several fields, including robotics and autonomous vehicles. Recently, neural radiance fields (NeRF) regression networks have shown promising results in novel view synthesis, 3D reconstruction, and scene representation (Liu et al. 2020. Yu et al. 202T). However, a typical generic approach to direct regression, in which the network trained by minimizing the mean squared or absolute error between targets and predictions, performs poorly compared to task-specialized approaches (Yang et al. 2018 Ruiz et al. 2018 Niu et al.) 2016 Fu et al. 2018). Recently, generic approaches based regression by binary classification have shown significant improvement over direct regression using custom-designed label encodings (Shah) al. (2022). In this approach, real-valued label is quantized and converted to an M-bit binary code, and these binary-encoded labels are used to train M binary classifiers. In the prediction phase, the output code of classifiers is converted to real-valued prediction using decoding function. Moreover, binary-encoded labels have been proposed for ordinal regression (Li & Lin 2006 Niu 2016) and multiclass classification (Allwein al., 2001 Cissé et al. 2012). The use of binary-encoded labels for regression has multiple advantages. Additionally, predicting a set of values (e.g., classifiers' output) instead of one value (direct regression) introduces ensemble diversity, which improves accuracy (Song et al., 2021). Furthermore, encoded labels introduce redundancy in the label presentation, which improves error correcting capability and accuracy (Dietterich & Bakiri 1995).
Finding suitable label encoding for given problem is challenging due to the vast design space. Related work on ordinal regression has primarily leveraged unary codes (Li Lin 2006 Niu et al.) Fu et al. 2018). Different approaches for label encoding design, including autoencoder, random search, and simulated annealing, have been proposed to design suitable encoding for multiclass clas-
sification (Cissé al. 2012 Dietterich & Bakiri 1995 Song et al. 202T). However, these encodings

---

Published as a conference paper at ICLR 2023
Disarcee between encodings
500
250
15 99 6
Target Label
150
100
302724211
With regularizer (8-7.0)
Without regularierr (8-0.0)
Difference
labels
Bit
10
12 14
18
Bit
10 12 14 16 18
Without reguirizer RI, 4.98
Without regularizer RZ, MAE
4.13
with regularizer R2, 3.73
With regularizer R1, NME 4.71
Average #bit transitions 3.9
Average #bit transition 2.5
(a) Effect of regularizer R1
(b) Effect of regularizer R2
Figure 1: (a) and (b) demonstrate the effect of proposed regularizers on learned label encodings and the regression (NME/MAE) for FLDI_5 and LFHI benchmarks (Table respectively. (a) Regularizer R1 encourages the distance between learned encodings to be proportional to difference between corresponding label values. (b) Regularizer R2 reduces the number of bit transitions per bit, reducing the complexity of decision boundaries to be learned by binary classifiers. Here blue and white colors represent I and O, respectively.
perform relatively poorly for regression due to differences in task objectives (Section 2 More recently, |Shah et al. (2022) analyzed and proposed properties of suitable encodings for regression. They empirically demonstrated the effectiveness of manually designed encodings guided by these properties. While establishing the benefits exploring the space label encodings for given task, they did not provide automated approach to do so.
In this work, we propose Regularized Label Encoding Learning (RLEL), an end-to-end approach to train the network and label encoding together. Binary-encoded labels have discrete search space. This work proposes to relax the assumption of using discrete search space for label encodings. Label encoding design can be approached by regularized search through continuous space of real-valued label encodings, enabling the use of continuous optimization approaches. Such formulation enables end-to-end learning of the network parameters and label encoding.
We propose two regularization functions to encourage certain properties in the learned label encoding during training. Specifically, while operating on real-valued label encoding, the regularization functions employed by RLEL are designed to encourage properties previously identified as being helpful for binary-valued label encodings (Shah al.) 2022). The first property encourages the distance between learned encoded labels to be proportional to the difference between corresponding label values, which reduces the regression error. Further, each bit label encoding can considered binary classifier. The second property proposes to reduce the complexity of binary classifier's decision boundary by reducing the number of bit transitions (0 - and - 0 transitions in the classifier's target over the range of labels) in the corresponding bit in binarized label encoding.
Figure demonstrates the effect of proposed regularizers on the learned label encodings and regression errors. Figure Ta plots the I.1 distance between learned encodings for different target labels versus the difference between corresponding label values. The I11 distance between encodings for distant targets is low without the regularizer. In contrast, the proposed regularizer encourages the learned label encoding to follow the first design property. Figure Lb plots learned label encoding (binarized representation for clarity). Each row represents encoding for target value, and each column represents a classifier's output over the range of target labels. The use of regularizer R2 reduces the number of bit-transitions (i.e., - and 0 + transitions in a column) to enforce the second design property and consequently reduces the regression error.
We demonstrate that the regularization approach employed by RLEL encourages the desired properties in the label encodings. We evaluate the proposed approach on 11 benchmarks, covering diverse datasets, network architectures, and regression tasks, such as head pose estimation, facial landmark detection, age estimation, and autonomous driving. Label encodings found by RLEL result in lower or comparable errors to manually designed codes and outperform generic encoding design approaches (Gamal et al. 1987 Cissé et al.] 2012 Shah et al.] 2022). Further, RLEL results in lower error than direct regression and multiclass classification by 10.9% and 12.4%. respectively, and even outperforms several task-specialized approaches. We make the following contributions to this work:
2

---

Published as a conference paper at ICLR 2023
Target Label
Renery--ncdde
Target Label
Rinany-encoded
Jarget Label
Snuar--nnodddd
Label
Emur Pribubiity
D
B
B
B
B
B
BP
B
B
B
B
D
B
B
D
Target
0
8
8
1
8
8
0
1
8
1
2
D
8
0
0
0
1
8
1
8
0
1
6
(b) Hadamard encoding (c) Johnson encoding (d) Error distribution Figure 2: (a-c) Examples of label encodings. Each row represents the binary-encoder target label for given target. (d) represents the error probability distribution the classifier-1 for diffcrent target values in Johnson encoding. Green lines represent the bit transitions of classifier.
(a) Unary encoding
We provide efficient label encoding design approach by combining regularizers with continuous search space of label encodings.
We analyze properties of suitable encodings in the continuous search space and propose regularization functions for end-to-end learning network parameters and label encoding.
We evaluate the proposed approach 11 benchmarks and show significant improvement over different encoding design methods and generic regression approaches.
BACKGROUND AND RELATED WORK
This section summarizes relevant background information on regression by binary classification approach and different code design approaches. Task-specific regression approaches are summarized in Appendix A.3 However, a generic regression approach applicable to all tasks is desirable.
2.1 REGRESSION BY BINARY CLASSIFICATION
regression problem can be converted a of binary classification subproblems. Prior works proposed to use N binary classifiers for scaled and quantized target labels E: 21,2.... N) (Niu et al. 2016 Fu et al. 2018). Here, classifier-k target output is 1 the target label is greater than k. else Figure 2a represents the target output of binary classifiers for this setup. (Shah et al. (2022) proposed Binary-encoded Labels (BEL.), generalized framework for regression by binary classification. proposed approach, real-valued target label is quantized and converted to binary code B of length M using an encoding function M binary classifiers are trained using binary-encoded target labels B E 10, 13MM During inference, the output of binary classifiers, i.e., predicted code, is converted to the real-valued prediction using a decoding function D.
the
Using encoded labels introduces crror-correction capability, i.c., tolerance classification error. Hamming distance between two codes (number of differing bits) gives a mcasure of error- cororccornttion capability. Error-correcting codes, such Hadamard codes (Figure 2b have been proposed to encode labels in multiclass classification (Dietterich & Bakiri 1995 Verma & Swami 2019). BEL showed that such codes are not suitable for regression due differences in task objectives and classifiers' error probability distribution, and proposed properties of suitable codes for regression.
The first property suggests trade-off between classification errors and error correction properties. Each classifier learns decision boundary for bil Iransitions from - 0 and ( - in the classifier's target bit over the numeric range of labels (green lines within a column in Figure 2}. For example, in Johnson encoding (Libaw & Craig 1953 (Figure 2ch, the classifier for bit 8 learns two decision boundaries for bit transitions in intervals (2,3) and (6, 7). The number of intervals for which the classifier has to learn separate decision boundary increases with bit transitions, increasing its complexity. Hadamard codes have excellent error-correction properties but have several bit transitions (Figure 2by; this increases the complexity of a classifier's 's decision boundary and reduces its classification accuracy compared to unary and Johnson codes (Figure 2a and Figure 2c). Rahaman et al. (2019) introduced the term spectral bias and demonstrated that neural networks prioritize learning fow-freuenccy functions (i.c., lower local fluctuations). The spectral bias of neural networks provides insights into accuracy improvement with the reduction in the number of bit transitions.
3

---

Published as a conference paper at ICLR 2023
Second, Hamming distance between two codes should increase with the difference between corre- sponding label values to reduce the probability of making large absolute errors between predicted and target labels. The probability that erroneous predicted code for label X will be decoded as Y decreases as the hamming distance between codes for values and Y increases. Thus the above rule reduces the regression error. Last, the encoding design should also consider the error probability of classifiers. BEL shows that the error probability of classifiers is not uniform for regression and increases near bit transitions, as shown for classifier B in Figure 2d Here, the probability of predicting 8 for target label 1 is very low, as the bit differing between corresponding codes (B' has very low classification error probability. BEL shows that this property can exploited design better codes for regression. These three factors significantly affect the suitability of encodings. BEL demonstrates that simple codes sampled based on these properties, such as unary or Johnson code, result in lower errors than widely used error-correcting Hadamard code.
2.2 ENCODING DESIGN
Encoding design is a well-studied problem with applications in several fields. Iterative approaches. such simulated annealing random walk, have been proposed for code design (Dietterich & Bakiri 1995 Song et al. 2021). However, iterative approaches are computationally expensive as each iteration requires full/partial training of the network measure the error for sample encodings. Works on multiclass classification using binary classifiers have demonstrated the effectiveness of error-correcting codes such as Hadamard or random codes (Verma & Swami] 2019 Dietterich & approach to design compact codes for mu ticlass classification problems with a large number of classes. However, these approaches do not consider the task objective and classifiers' nonuniform error probability distribution for regression.
Bakiri 1 1995). Cissé et al. (2012) proposed an autoencoder
Deep hashing approaches aim to find binary hashing codes for given inputs such that the hashing codes preserve the similarities in the inputs space (L.uo et al., 2022 Wang et al. 2018 Xia et al. 2014 et al., [019 Liu al., 2016). Deep supervised hashing approaches use the label information to design the loss function. In deep hashing, loss functions are designed to decrease the hamming distance between binary codes for similar images (c.g., same label). contrast, label encoding design for regression aims to reduce the error between decoded output codes and target labels. Further, deep hashing approaches are designed for classification datasets and do not account for the nonuniform error probability distribution of classifiers observed in regression. As shown in prior work (Shah) et al. 2022). classifiers' nonuniform probability significantly affects the design of suitable codes for regression. Thus, naive adaptation of deep hashing approaches for regression problems performs poorly compared to codes designed by the proposed approach RLEL (Section A.1.5).
3 REGULARIZED LABEL ENCODING LEARNING
Regression aims to minimize the error between target labels y; and predictions d for a set of training samples i. In regression by binary classification, the network learns M-bit binary -ncodee labels
B, {0, M. During inference, the predicted code B, is decoded to real-valued label 9.- We propose to relax label encodings' search space from a discrete ((0, 1 1 to a continuous space (IR enabling the use of traditional continuous optimization methods. We propose regularizers to enable efficient search through this space. This work automates the search for label encoding using an end-to-end training approach that learns the network parameters and label encoding together.
This section explains the proposed label encoding learning approach RLEL. First, we explain the regression by binary classification formulation used in this work for end-to-end training of network parameters and label encoding. Further, we introduce properties of suitable label encodings in continuous space. Lastly, we explain the proposed regularizers and loss function that accelerate the search for label encodings by encouraging learned label encoding to exhibit the proposed properties.
3.1 LABEL ENCODING LEARNING
represents the formulation used in this work for label encoding learning. I, and y represent the input and the real-valued target label for sample respectively. We assume y: E N] for simplicity as the real-valued targets with any arbitrary numeric range can be scaled and shifted to this range. Q; E (1,2,.., N} represents the quantized target label. The input æ;; is
Preliminaries: Figure 3
4

---

Published as a conference paper at ICLR 2023
Correlation Vector C
Feature Vecter
Output codc
Z
D
2
Feature Extractor
Decoding Matris M
9. ΣS x
M
Fully Connected
Layer
SoftMax
Figure 3: The flow for combined training of feature extractor and label encoding.
Table Summary of notations used in this work
Notation
Description
S4. Nh. Q.
Input, real-valued target label, and quantized target label training example M, [1. N) and Q, (1, N}
N
The range of target labels Number of quantization levels for Q2:
M
Number of bits/values for label encoding
B,
B,
Target and prodicted binary-oncoded labels (used for hand-crafted label encoding
2.
Predicted real-valued encodings; activation values of the output code layer in Figure
E
Learned label encoding through RLEL; calculated from 2, for all training examples using equation
1
C2
Output correlation vector of length N. Here C gives measure of the prohability that predicted label value is equal
D
Decoding marrix that converts the predicted encodings to correlation vector Cis
passed through a feature extractor and fully connected (FC) layers to generate the predicted encoding 2, Rad ①.. Here, an FC layer of size 0 (0 < M) is added between the feature vector and output code. This layer reduces the number of parameters in FC layers and improves accuracy, as shown by previous work (Shah al., 2022). Each neuron of the output code is a binary classifier, and the magnitude 2,/ gives a measure of the confidence of the classifier-k (Allwein et al. 2001). The output code and decoding matrix D R are multiplied @2, and the output is passed through softmax function to give correlation vector C₁ R c where the value of C represents the probability that the predicted label ÿ: k. This correlation vector is then converted to real-valued prediction by taking the expected value ④ Table summarizes the notations used in this work.
Prior works use custom custom-designes designed label encoding in this formulation. For example, Shah et al. (2022) proposed a series of suitable label encodings B, {Q2)) E (0, M. The network can be trained using binary cross-entropy loss between B, and 2, and these encodings are used as columns of the decoding matrix (D.. E())). However, is desirable to automatically find suitable encodings B; and decoding matrix D without searching through a set of hand-designed encodings.
The search space of binary label encodings is discrete and hence challenging search using traditional continuous optimization methods (Darabi et al.] 2018). Hence, we relax the assumption of binarized label encodings and use continuous search space. This relaxation, coupled with the proposed formulation, enables the use of traditional optimizers to learn label encoding and the decoding matrix D with the entire network by optimizing the loss between targets and prediction. Let S,, represent the set of training samples with quantized target Q; n, and E' E X matrix, where each row En,; is the encoding for target Q2 n. E is defined as:
xM represent label encoding
En.:
Σ 2
S
ICS
However, training the network solely with the loss between ÿ and yi does not constrain the search space of label encodings (E). In regression, the label encoding (E) significantly impacts the accuracy, and label encodings that follow specific properties result in lower error (Shah et al. 2022). The following section explains desirable characteristics of output codes for regression and how these properties can be encouraged in learned label encoding using regularization functions.
3.2 LABEL ENCODING LEARNING WITH REGULARIZERS
Section 2.1 summarizes the properties of suitable binary label encodings for regression proposed by prior works. These properties constrain the vast search space of label encodings. We further propose two properties applicable to real-valued label encodings narrow its search space.
5

---

Published as a conference paper at ICLR 2023
R1 Distance between encodings: A binary classifier's real-valued output represents its confidence (i.e., error probability). The L1 distance between real-valued predicted encodings gives more weight to classifiers that are more confident (i.e., higher value). Thus, by considering the LI distance between real-valued codes instead of the hamming distance between binary codes, we can combine the second and third design properties of binary label encodings (Section 2.Tp into single rule for real-valued label encodings. This gives the first regularization rule: LI distance between encodings for two labels should increase with the difference between two labels, i.e., Er, E2 -
The number of bit transitions in a bit-position of label encoding gives a measure of the binary classifier's decision boundary's complexity. There are no 0 1 or 1 + 0 transitions in real-valued label encodings. Thus, we approximate the number of bit transitions by measuring the [.1 distance hetween encodings for adjacent label values Q₂ = and Q: n The number of bit transitions for real-valued label encoding E can be approximated as:
R2- Regularizing bit transitions:
B3 Ejt1,l
This leads the second regularization rule: The L. distance between encodings for adjacent target label values should be regularized to find a balance between the complexity of the decision boundary and the error-correction capability of designed codes for a given benchmark.
3.3 Loss FUNCTION FORMULATION
We propose two regularizers applicable to learned label encoding (E) to limit its search space. E is
measured from the output codes 2; over the complete training dataset (Equation] []. However, deep neural networks are trained using mini- batches, where each batch consists of K training examples sampled randomly from the (typically shuffled) training set. We extend the proposed regularization rules to apply to a minibatch-based loss function.
R1: Regularizer RI can be approximated as the following for batch with K training examples:
K
L1
m0 ly: リo -1 2/1l1)
The above regularization considers K2 pairs in minibatch of K samples, and penalizes a pair of training samples i and j if 1.1 distance between encodings 2; and 2, is less than twice the difference between corresponding label values. The scaling parameter is set to two as it encourages at least one bit difference between two binary codes. This encourages the L1 distance between encodings to be approximately proportional to the difference between corresponding target values.
R2: Regularizer R2 minimizes the LI distance between encodings of adjacent label values. In randomly formed minibatch consisting of only subset of training examples, adjacent target labels might not be present. Hence it is nontrivial to apply this regularizer to the label encoding, However, we find that imposing this regularizer on the decoding matrix also helps with regularizing the bit transitions in the learned label encoding and can be added the loss function irrespective of the batch formulation approach. Theoretical analysis and empirical verification for this approach are provided in Section A.1.7 and Section 4.4 Equation represents the proposed regularizer.
M N
L
Di, Distal
(4)
Loss Function: We use the cross-entropy loss between Ć and soft target labels. Here, each bit of label encoding resembles binary classifier. However, identifying the predicted label corresponding to the multi-bit label encoding can be treated as a multiclass classification problem. Soft target labels are probability distributions generated using the distance between different classes. Soft target labels can be used with cross-entropy loss and have shown improvement over typical classification loss between the correlation vector C and quantized target label Q, or regression loss between the
expected prediction i and target label y: for ordinal regression (iaz & Marathe, 2019). We use

---

Published as a conference paper at ICLR 2023
Table 2: Benchmarks used for evaluation
Feature
Label rangef
Task
Dataset
Benchmark
Extractor
Quantization levels
300LP Zhu al.
Landmurk--fre free
ResNe150
/AFI 2000 7hu el.
1.HH
0-201/200
2D head pose
at
2016
estimation
2016
BTW (Fnnell
2013
D112
0: 0-150V150
10
COFW (Burgss-Artizz et
Facial
HRNetV2-
FLDI/FLDI (100%/100% training dataset)
0-256/256
10
2013)
Landmark
(Wana
300W Sagonaa el 2013
FLDZ/FLD2 (1098/10% (109/
training dalaset)
0-256/256
Delection
el al.
2020
WFLW Wir
PLOMPLDS x (10%//0%% training dalasel)
0 252522566
10
MORPH-II [Kicanok
ResNet50/
AEI
0-64/64
10
Age estimation
Tesafaye 2000
ResNe134
APAD (Nil et
20T6)
AF2
0-32732
10
End-to-ond
sutenomous
Net Bojarsk
PilotNet
PN
0-670/670
10
driving
el
2017)
this loss function for RLEL and multiclass classification. Complete loss function with regularizers (Equation and can be written as:
N-1 L ((:)) Σ
K
K
Disti +8
max 2x 1y: リí -112: 201li,,
where (v:)) = e e-bi-vil
(5)
Σn e In
Here, the first term is the loss between target and predicted labels. Φ; represents the target probability distribution generated from target V/- The second and third terms are for regularizer RI (Equation and regularizer R2 (Equation 4). respectively.
A trade-off exists between the proposed desirable properties of label encodings: Encouraging one design property comes the cost of relaxing constraints imposed by other design properties. As demonstrated by [Shah et al. (2022), finding the right balance between these properties for given benchmark is crucial to finding the best label encoding for given problem. Thus, these design properties can be naturally applied as regularizers, and the search for balance between different properties can be seen as tuning the regularization parameters a and 3.
EVALUATION
This section first provides the experimental setup used evaluate the proposed approach, then compare RLEL with different label encoding design methods. We also compare RLEL with different regression approaches demonstrate its effectiveness as generic regression approach. Last, provide an ablation study to show the impact of proposed regularizers.
4.1 EXPERIMENTAL SETUP
Table summarizes the regression tasks, feature extractors architecture (Figure 3 and datasets for benchmarks used for evaluation. Selected benchmarks cover different tasks, datasets, and network architectures and have been used by prior works on regression due to the complexity of the task (Diaz & Marathe 2019 [hah et al. (022). We also evaluated RLEL on facial landmark detection tasks with smaller datasets to demonstrate its generalization capability. this setup, a subset of training samples is used for training, whereas the complete test dataset is used to measure the test error.
Landmark-free 2D head pose estimation (LFHH takes 2D image as input and directly finds the pose of human head with three angles: yaw, pitch, and roll. The facial landmark detection task focuses on finding (z, y) coordinates key points in a face image. The age estimation task is used to find person's is age from the given face image. In end-to-end autonomous driving, the car's steering wheel's angle to be predicted for a given image of the road. Normalized Mean Error (NME) or Mean Absolute Error (MAE) with respect to real-valued labels used as the evaluation metrics.
We compare with other encoding design approaches, including simulated annealing, autoencoder (summarized in Appendix A.2) and manually designed codes (Shah et al., 2022). We also compare
7

---

Published as a conference paper at ICLR 2023
Table Comparison of with different label encoding design approaches. The bold and underlined numbers represent the first and second best errors, respectively.
Error (MAE or NME)
Approach
LFHI
LF112
FLDI
FLDI
FLD2
FLD2 s
Simulated anncaling
4.32:t0.12
5.03=00.8
3.55+0001
6.52++0.05
3.59+0..0
5.35+0.01
Autoencoder
3.38.100.1
4.84.0..2
3.39.10.01
4.85
3.39110.00
4.20.10.05
LWL regularizers)
4.03:10.15
4.9610008
3.36.10.01
1..07
3.39:10.01
4.28.0.05
BEL(Manually designed)
3.56:F0111
4.77=0005
3.34:0.01
4.63+0.03
3.40:F0.02
4.15+0.01
RLEL
3.55.10.10
4.77
3.36.10.01
4.7110..0
3.37.10.02
4.15.10.05
Approach
FLD33
FLD3_S
AEI
AE2
PN
Simulated annealing
4.52:10.02
6.3810001
2.3310.01
3.1710.01
4.2510.01
Autoencoder
4.36:F0.01
5.62+0.01
2.29:0.00
3.1940.01
4.49
LWL regularizers)
4.35:1:0.2
5.6810.04
2.30.10.01
3,1740.01
3,22:10.02
BELnManyllyy designed)
4.36100.02
5.6210.00
2.271
3.1110.00
3.1110.01
RLEL
4.35
5.58
2.20
3.1440.01
3.01
RLEL with generic regression approaches, such as direct regression and multiclass classification. For direct regression, LI or L2 loss functions with L2 regularization are used. Label value scaling (hyperparameter) is used to change the numeric range of labels. For multiclass classification, we use cross-entropy loss between the softmax output and target labels.
The feature extractor and regressor are trained end-to-end for all approaches. The feature extractor architecture, data augmentation, and the number of training iterations are kept uniform across different approaches for a given benchmark. There is no notable difference between the training time for all approaches. The training dataset is divided into 70% training and 30% validation sets for tuning hyperparameters. The network is trained using the full dataset after hyperparameter tuning. We use the same values for quantization levels prior work (Shah al., (002)). An average of five training runs with an error margin of 95% confidence interval is reported. Appendix A.3 provides details on datasets, training parameters, related work (task-specific approaches), and evaluation metrics.
4.2 COMPARISON RLEL WITH ENCODING DESIGN APPROACHES
Table 3 compares different encoding design approaches. RLEL results in lower error than simulated annealing and autoencoder-bate approaches for most benchmarks. Both approaches are widely used for code design. However, for regression tasks, the suitability of label encoding depends upon the problem, including the task, network architecture, and dataset (Shah et al., (022). Simulated annealing or autoencoder-based approaches do not optimize the encodings end-to-end with the regression problem., resulting in higher error. Furthermore, the gap between the error of learned label encoding with and without regularizers (RLEL and LEL) increases for smaller datasets, which suggests that RLEL-learned codes generalize better.
RLEL can not be used with binary-crose entropy loss for training. We observe that for some benchmarks, the autoencoder based approach outperforms (c.g.. LFH1) as it can used with binary- cross entropy loss. The main objective of RLEL is to automatically learn label encoding that can reach the accuracies of manually designed codes (BEL), as using such codes is time and resource- consuming. Hyperparameter search for RLEL can be performed by off-the-she the--hhhhhffff hy hyperarrameter tuners/libraries without manual efforts (Li et al., 2017 Falkner et al. 2018). In contrast, hand- design codes. Also, multiple training runs still
designed codes need human intervention required to find suitable codes for a given benchmark from a set of hand designed codes. As shown in Table RLEL results in lower or comparable errors dendddese codes.
4.3 COMPARISON RLEL WITH REGRESSION APPROACHES
RLEL is a generic regression approach that focuses regression by binary classification and proposes a label encoding learning approach. We compare RLEL with other generic regression approaches, including direct regression and multiclass classification shown in Table 4 RLEL problem formulation introduces more fully-comnected layers after the feature extractor; hence, we also perform ablation study on increasing the number of fully connected layers in Appendix A.1.4
8

---

Published as a conference paper at ICLR 2023
Table 4: Comparison of RLEL.wit with different regression approaches and state-of-the-art task-specialized ap-
proaches (more details in Appendix A.3). "/x:M" represents the model size.
Multiclass
Direct regression
RLEL
Task-specialized approach
classification
LFHI
1.22+5.13/23.5M
4.49:0.24/242M
3.30:0000/69..8 8M
LFH2
5,32.10.12/23..55
5.3110.05/24.8M
4.77 6M
3.90.10.03/69.M 8M
FLDI
3.60=0.02/10.2M
3,48:0.00/25.66
3.36:0000/11..6M
3.34:0.02//0.6M
FLDI_S S 32.70.11.37/1..2M
5.36.10.03/25.66M
4.7110.04/10.6M
FLD2
3.54+0.03/10.2M 3,46+0.00/44.2NM
3,37+0002/11.22
3.07/25.10
IM
FLD2_s
5.04=0.02/10.2M 4.50:00004/5.220 2M
4.15:0005/11.22
FLD3
4.6410.03/10.2MM 4.46.10.01/61.3M
4.3510.01/11.77M
4.32/-
FLD3_S
6.35:00.0711..2M 6.05=#0.0166..3M
5.58:0.01/11.77
AEI
2.37:0.001/33.5M
2.75.0..03/2..M
2.28.10011/23.6M
1.96/3.7M
AE2
3.1610.002/2355M
3.38.0.005/2:88M
3.1410.01/23.6
3.47/21.
PN
45/40.
5.54:0.03/25.66
3.01=0.33/0..6M
4.24/10.2M
"This uses different network architecture, data mugmentation, and training process.
Table 5: Effect of regularization R2 on bit-transitions in binarized and real-valued label encodings.
Proposed regularizer using Decoding matrix (Equation
ABit transitions in hinarized label encoding
Approximated hit transitions label encoding (Equation X
a value
6816.1
5097
391 88
215.3
3596
0.5
130.8
3180
104,19
RLEL consistently lowers the error compared to direct regression and multiclass classification with 10.9% and 12.4% improvement on average.
4.4 ABLATION STUDY
Figure la demonstrates that the use of regularizer RI encourages the L1 distance between encodings to be proportional to the difference between target values. The second regularizer R2 is introduced
to regularize the number of bit transitions in encodings. As mentioned in Section 3.3 we apply the regularization on the decoding matrix as it is nontrivial to apply this regularization on the output codes for randomly formed batches. Table 5 summarizes the effect of a (i.e., the weight of R2) on the number of bit transitions in the decoding matrix and binarized/real--alleed label encoding. The second column is the number of bit transitions in the decoding matrix (Equation 4 which is used as the regularization function. The third and fourth columns are the total number of bit transitions in binarized and real-valued encodings (Equation 2 The table shows that the proposed regularizer on the decoding matrix also encourages fewer bit transitions in the label encoding. Figure Tb shows the impact of regularizer R2 on learned binarized label encoding.
As pointed out by prior works (Shah et al. 2022). there is trade-off between the error probability and error correction capability of classifiers for regression. Hence, depending upon the benchmarks, more bit transitions can be added as the advantage of increased error correction outweighs the increase in classification error. We observe a similar trend, where adding R2 does not improve error for some benchmarks (FLDI_s, FLD2_s), as it constrains the number of bit transitions.
CONCLUSION
This work proposes an end-to-end approach, Regularized Label Encoding Learning, learn label encodings for regression by binary classification setup. We propose combination of continuous approximation of binarized label encodings and regularization functions. This combination enables an efficient and automated search of suitable label encoding for a given benchmark using traditional continuous optimization approaches. The proposed regularization functions encourage label encoding learning with properties suitable for regression, and the learned label encodings generalize better, specifically for smaller datasets. Label encodings designed by the proposed approach outperform simulated amnealing- and autoencoder-designee label encodings by 12.6% and 2. 1%, respectively. RLEL-designed codes show lower or comparable errors to hand-designed codes. RLEL reduces error on average by 10.9% and 12.4% over direct regression and multiclass classification.

---

Published as a conference paper at ICLR 2023
6 ACKNOWLEDGEMENTS
This research has been funded in part by the National Sciences and Engineering Research Council of Canada (NSERC) through the NSERC strategic network on Computing Hardware for Emerging Intelligent Sensory Applications (COHESA) and through an NSERC Strategic Project Grant. Tor M. Aamodt serves as a consultant for Huawei Technologies Canada Co. Ltd and recently served as consultant for Intel Corp.
Reproducibility: We have provided details on training hyperparameters, experimental setup, and network architectures in Appendix A..]] Code is available at https: / / thub. com/ We have provided the training and inference code
ubc-aamodt- REEEEEEerrrrrrr___ with trained models.
Autonomous robotics and vehicles are major applications of deep regression networks. Thus improvement of regression tasks can accelerate the progress of these fields, which may lead to some negative societal impacts such as loss of jobs, privacy, and ethical concerns.
Code of Ethics:
REFERENCES
Erin L. Allwein, Robert E. Schapire, and Yoram Singer. Reducing multiclass to binary: A unifying approach for margin classifiers. J. Mach. Learn. Res., 1:113-141, September 2001. doi: 10.1162/ 15324430152733135
Ardhendu Behera, Zachary Wharton, Pradeep Hewage, and Swagat Kumar. Rotation axis focused attention network (rafa-net) for estimating head pose. In Computer Vision ACCV 2020. 2021.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to End Learning for Self-Driving Cars. arXiv: I604.03366 2016.
Mariusz Bojarski, Philip Yeres, Anna Choromanaska, Krzysztof Choromanski, Bernhard Firner, Lawrence Jackel, and Urs Muller. Explaining how deep neural network trained with end-to-end learning steers a car. arXiv:/704.000 170.07991, 2017.
Adrian Bulat and Georgios Tzimiropoulos. Human Pose Estimation via Convolutional Part Heatmap Regression. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 717-732, 2016.
Xavier Burgos-Artizzu, Pietro Perona, and Piotr Dollár. Robust Face Landmark Estimation under Occlusion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1513- 1520, 12 2013. doi: 10. 00.100//CC2.2013.111.1
Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. Rank consistent ordinal regression for neural networks with application to age estimation. Pattern Recognition Letters, 140:325-331, 2020. 2020. doi: https://doi.org/10.. 1016////00011/...... http::/aaarrcc22...1111
Sully Chen. Driving-datasets.
M. Cissé, T. Artières, and Patrick Gallinari. Learning Compact Class Codes for Fast Inference in Large Multi Class Classification. In Peter A. Flach, Tijl De Bie, and Nello Cristianini (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 506-520. Springer Berlin Hcidelberg, 2012.
Sajad Darabi, Mouloud Belbahri, Matthieu Courbariaux, and Vahid Partovi Nia. Regularized binary network training. arXiv preprint 1rXiv:1812.118000 2018.
T. G. Dietterich and G. Bakiri. Solving Multiclass Learning Problems via Error-Correcting Output Codes. Journal of Artificial Intelligence Research, 2:263-286, 1995. doi: 10.1613/jair.103 105.
Raál Díaz and Amit Marathe. Soft labels for ordinal regression. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi: 10. 10.11009CVP..2 1900087
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimiza- tion scale. In ICML., 2018.
10

---

Published as a conference paper at ICLR 2023
Gabriele Fanelli, Matthias Dantone, Juergen Gall, Andrea Fossati, and Luc Gool. Random forests for real time 3d face analysis. International Journal of Computer Vision, 101(3):437 458, February 2013.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep Ordinal Regression Network for Monocular Depth Estimation. Proceedings the IEEE Computer Society Conference on Computer Vision and Puttern Recognition, pp. 2002-2011, 2018.
A.E. Gamal, L.. Hemachandra, I. Shperling, and V. Wei. Using simulated annealing design good codes. IEEE Transactions on Information Theory, pp. 116-123, 1987. doi: 10.1 199. 1057277.
Bin-Bin Gao, Hong- Yu Zhou, Jianxin Wu, and Xin Geng. Age estimation using expectation of label distribution learning. In Proceedings of the Twenty-Sevnnth International Joint Conference Artificial Intelligence, IJCAI-18, pp. 712-718, 2018. doi: 10.24963/jjcai22118999
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Heng- Wei Hsu, Tung-Ya Yu Wu, Sheng Wan, Wing Hung Wong, and Chen- Yi Lee. Quatnet: Quaternion- based head pose estimation with multiregression loss. IEEE Transactions on Multimedia, 21(4): 1035-1046, 2019. doi: 10.1 109900M..0........ 2.266770.
Lu Jin, Xiangbo Shu, Kai Li, Zechao Li, Guo-Jun Qi, and Jinhui Tang. Deep ordinal hashing with spatial attention. Transaction on Image Processing, 28(5):2173-218 2019.
Marek Kowalski, Jacek Naruniec, and Tomasz Trzcinski. Deep alignment network: A convolutional neural network for robust face alignment. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2034 2043, 2017.
doi: 10. N0.11099CVPR.:2017.2544
Abhinav Kumar, Tim K. Marks, Wenxuan Mou, Ye Wang. Michael Jones, Anoop Cherian, Toshiaki Koike-Akino, Xiaoming Liu, and Chen Feng. LUVLi face alignment: Estimating Landmarks' loca- tion, uncertainty, and visibility likelihood. Proceedings of the IEEE Computer Society Conference Computer Vision and Puttern Recognition, 2020. doi: 10.1 2,.10///VPR422002200000000
H. Lai, Pan, Ye Liu, and S. Yan. Simultaneous feature learning and hash coding with deep neural networks. In 2015 ICEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Ling Li and Hsuan Tien Lin. Ordinal regression by extended binary classification. In Proceedings of the 19th International Conference Neural Information Processing Systems, pp. 865-872, 2006.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn, Res, 18(1): 6765-68 6816, jan 2017. ISSN 1532-4435.
William Libaw and Leonard Craig. A photoelectric decimal-coded shaft digitizer. Electronic Computers, Transactions of the L.R.E. E. Professional Group on, EC-2:1 -4, 10 1953.
Haomiao Liu, Ruiping Wang, Shiguang Shan, and Xilin Chen. Deep supervised hashing for fast image retrieval. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33:15651-15663. 2020.
Xiao Luo, Haixin Wang, Daqing Wu, Chong Chen, Minghua Deng, Jianqiang Huang, and Xian- Sheng Hua. A survey deep hashing methods. ACM Trans. Knowl. Discov. Data, 2022. ISSN 1556-4681. doi: 10. 105/344/352222
Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. Ordinal regression with multiple output CNN for age estimation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016.

---

Published as a conference paper at ICLR 2023
Mohammad Norouzi, David J. Fleet, and Ruslan Salakhutdinov. Hamming distance metric learning. In Proceedings of the 25th International Conference on Neural Information Processing Systems- Volume 1, 2012.
Hongyu Pan, Hu Han, Shiguang Shan, and Xilin Chen. Mean-variance loss for deep age estimation from face. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5285-5294, 2018.
doi: 10. 2000//CCP...18..0000
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Proceedings of the 36th International Conference Machine Learning, pp- 5301-5310, 2019.
Sebastian Raschka. MLxtend: Providing machine learning and data science utilitics and extensions to Python's scientific computing stack. Journal of Open Source Software, 3(24);638, April 2018. doi: 10.21 105/joss.00638.
K. Ricanek and T. Tesafaye. Morph: longitudinal image database of normal adult age- age-progression. In 7th International Conference on Automatic Face and Gesture Recognition (FGR06), pp. 341-345, 2006. doi: 10. 109//GRR2006.7777
Nataniel Ruiz, Eunji Chong, and James M. Rehg. Fine-grained head pose estimation without key- points. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
Olga Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma. Zhiheng Huang, A. Karpathy, A. Khosla, M. Bernstein, A. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-252. 2015.
C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic. 300 faces in-the-wild challenge: The first facial landmark localization challenge. In 2013 IEEE International Conference on Computer Vision Workshops, pp. 197-403 2013.
Deval Shah, Zi Yu Xue, and Tor M. Aamodt. Label encoding for regression networks. In International Conference on Learning Representations, April 2022. URL httphttts/://pppeeeeeeew. /teet [pdf:id=kkawWddKLLI
Yang Song, Qiyu Kang, and Wee Peng Tay. Error-Correcting Output Codes with Ensemble Diversity for Robust Learning in Neural Networks. AAAI, 2021.
Gunjan Verma and Ananthram Swami. Error correcting output codes improve probability estimation and adversarial robustness of deep neural networks. Advances in Neural Information Processing Systems, 32(Neu,, 2019.
J. Wang. T. Zhang. J. Song, N. Sebe, and H. Shen. A survey learning hash. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(04), 2018.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolute representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence, PP, April 2020.
Xinyao Wang, Liefeng Bo. and Fuxin Li. Adaptive wing loss for robust face alignment via heatmap regression. In 2019 JEEE International Conference on Computer Vision (ICCV), pp. 2019. doi: 10. 109/1CC..201.00000000
W. Wu, Chen Qian, S. Yang, Q. Wang, Y. Cai, and Qiang Zhou. Look at boundary: A boundary- aware face alignment algorithm. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 21 2129-2138, 2018.
Rongkai Xia, Yan Pan, Hanjiang Lai, Cong Liu, and Shuicheng Yan. Supervised hashing for image retrieval via image representation learning. In Proceedings of the Ehenty-Eighte AAAI Conference Artificial Intelligence, AAAI'4 2014.
12

---

Published as a conference paper at ICLR 2023
Zixuan Xu, Banghuai Li, Miao Geng, Ye Yuan, and Gang Yu. Anchorface: An anchor- facial landmark detector across large poses. ArXiv:2007.03221, 2020.
Tsun-Yi Yang, Yi-Hsuan Huang, Yen-Yu Lin, Pi-Cheng Hsiu, and Yung-Yo Yu Chuang. Ssr-net: A compact soft stagewise regression network for age estimation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 18, pp. 1078-1084, AAAI Press, 2018. ISBN 9780999241127.
Tsun-Y Yi Yang. Yi-Ting Chen, Yen- Yu Lin, and Yung- Yu Chuang. Fsa-net: Learning fine-grained structure aggregation for head pose estimation from a single image. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one few images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4578-4587, 2021.
Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and Stan Li. Face alignment across large poses: 3d solution. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 146-155, 06 2016.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
This supplemental material provides additional results and ablation studies (Section A. methodol- ogy for baseline encodings design approaches (Section A.2), and related work on task ase-speeia approaches and experimental setup ASection|A.3) for RLEL. Code is available at athhtpps: /gthhbb
com/ubc- aamodt gro.p regresssioon
A.1 ABLATION STUDY
provide an ablation study and supporting data on impact of proposed regularization functions and hyperparameters on label encoding learning. Section A.1.4 covers an ablation study on the impact of the number of fully connected layers in direct regression and multiclass classification. Section A.1.5 explains and compares deep hashing approaches (adapted for regression) with RLEL. Section A.1.6 provides results for geometric mean and Pearson coefficient as evaluation metrics.
Section A.I.J Section A.I.2, and Section A.1.3
A.1.1
IMPACT OF REGULARIZER RI
200
32
20
40
90
60
16
56
and Sww.S
88
anaand
(a) Without regularization: 8 == Figure 4: (a) and (b) show the LI distance between pairs of encodings for FLDI_ s benchmark for B = and 3 = 5.0, respectively. Each cell (i.j) in this matrix represents the L1 distance between learned encodings for label and i.e., E.;- Ezalli-
(b) With regularization: 5.0
21
200
500
40
200
1M8
37A
315
150
40
122
141
133
375
3 A
(a) Without regularization: / 3=0
(b) With regularization: 5.0
Figure 5: (a) and (b) show the LI distance between pairs of encodings for FLD2_s s benchmark for B = and 3 = 5.0, respectively. Each cell (i.j) in this matrix represents the LI distance between Iearned encodings for label and i.c.. E.;- Ej
We proposed regularization function RI to encourage the LI distance between encodings to be proportional to the difference between corresponding label values. Figure 4a and 4 represent the LI distance between pairs of learned encodings for FLD1 s benchmark without and with regularization, respectively. The -axs and Y -axis represent the label values. Here, some columns and rows are replaced by white lines, as these label values are not present in the training dataset. The data point coordinates (i, j) represent the L1 distance between encodings for label and j. i.e.,
14

---

Published as a conference paper at ICLR 2023
Table Effect the scaling parameter on error FL.D1, and FLD22, benchmarks.
Value of the scaling parameter
NME (FLD1,)
NME (FLD2,,
4.89
4.15
4.71
4. 15
4.83
4.20
4.97
4.27
4.95
4.28
5.06
4.41
| - E5all. For example, in Figure 4a the LI distance between encodings for label values 0 and 97 120 (light-blue coloured point coordinate (0, 97)). In Figure (4b the L1 distance between encodings for label values 4 and 96 is ~ 170 (red coloured point at coordinate (4, 96)).
The first design property (Section 3) states that the L1 distance between encodings should increase with the difference between corresponding label values. The difference between label values for pairs of encodings increases with the distance from the diagonal of this plot. Thus, the value of data points (i.e., the I.1 distance between encodings) should increase with the distance from the diagonal of this plot. As shown in Figure 4a without regularization, the distance between encodings less for faraway label values (blue-colored data points away from diagonal), which shows that learned encodings do not follow the proposed design property. As shown Figure 4b the introduction of regularization function R2 remedies this and increases the L1 distance between encodings for faraway labels. Similar observations are made for FLD2_s benchmarks, as shown Figure 5a and Figure 5b
Distance between encodings
Distance between encodings
300
400
R1 regularizer (8-5.0) Without regularizer ($=0.0)
250
300
200
150
200
100
100
50
With regularizer (8=7.0)
Without regularizer (8=0.0)
40
60 80
100
120
140
50
100
150
200
250
Difference between target labels
Difference between target labels
(a) FLD benchmark
(b) FLD benchmark
Figure 6: (a) and (b) plot the LI distance between pairs of encodings versus distance between corresponding label values for FLDI_s and FLD2_s benchmarks.
plotts the LI distance between encodings versus the difference between corresponding label values for benchmarks FLD1 s and FLD2_s s. For both the benchmarks, the proposed regularizer RI helps enforce the first design property for real-valued label encodings and results in better label encodings with lower error (Table 3)
Efffet of the scaling parameter in equation 3 We use the scaling parameter in equation 3 Our intuition behind using the scaling parameter 2 is based on binary-encoded labels. For two adjacent labels (i.e., ly; 3/ = 1), the loss function encourages || - 2 to greater than Here, 2 is the output encodings. In the case of binarized label encoding (-1 if Z < 0 and +1 if Z > 0), I|2,- 11 = signifies that two encodings differ in at least one bit.
We also analyzed the effect of changing this parameter for two benchmarks. Table 6 shows the impact changing this scaling parameter for two benchmarks. We observe that the error is higher if the scaling parameter is too low, encodings for two adjacent labels can not be discriminated against. this parameter is set too high, the encoding space more constrained and consequently the performance is degraded.
Based on this intuition and empirical verification on two benchmarks, we use the value for all benchmarks.
15

---

Published as a conference paper at ICLR 2023
A.1.2 IMPACT OF REGULARIZER R2
The regularization function R2 is proposed reduce the number of bit transitions in the learned label encoding. Figure 7 compares the label encodings learned for LFHI benchmark for different values of a, where is the weight of regularization function R2 (Equation 5). Each row & is the encoding for label value k. Each column i represents the output of the encoding position l for different label values. The regularization function proposed decrease the transitions in an encoding bit (blue->red and red red-bblue) over the range of label values. Section 4.4 provided quantitative results to demonstrate that increasing the value of reduces the number bit transitions. We observe similar trends in the plots of learned label encodings shown in Figure 7 increasing the value of a decreases bit transitions in the learned label encodings and improves MAE,
2 22
(a) Without regularization: 0, MAE= 4.13
(b) 0.1, MAE= 3.76
-3
-28
aneans
(c) 0.5, MAE= 3.73
(d) 1.0. MAE= 3.56
Figure 7: (a)-d)) represent the label encodings learned by RLEL for different values of weight a for regularizer R2 (Equation 5)
A.1.3 EFFECT OF HYPERPARAMETER IN RLEL
The RLEL approach introduces two hyperparameters. We first evaluate the sensitivity to these hyperparameters to determine the complexity of hyperparameter tuning. Figure(8shoon the NME for FLDI benchmark for different values of a and 3 values in Equation As shown in the figure, the error is not sensitive to small changes in these hyperparameters values, suggesting that a sparse search in the hyperparameter space suffices. Furthermore, several approaches have been proposed for efficient hyperparameter search (Li et al.) 2017: Falkner et al. 2018), and any off-the-shelf hyperparameter tuners/libraries can be used automatically find these values without manual efforts. In contrast, hand-designed codes need human intervention to design codes. Also, multiple training runs still required find suitable codes for given benchmark from set of hand-designed codes. On the other hand, RLEL provides an end-to-end automated approach for label encoding learning.
A.1.4 IMPACT OF THE NUMBER OF
LAYERS:
For RLEL, we use an extra fully connected bottleneck layer in the regressor as proposed by the prior work regression by binary classification (Shah et al. 2022). We provide ablation study
16

---

Published as a conference paper at ICLR 2023
Normalized Mean Emor (NME)
4.35
4.2
3.6
4.30
8 from Equation
3.0
2.4
4.25
1.8
1.2
0.6
4.20
0.0
0.3
0.9
1.5
2.1
2.7
a from Equation (5)
Figure 8: Impact of hyperparameters and from Equation on NME for FL.D1_ henchmark.
Table 7: Impact of the number of fully-connected layers in direct regression and multiclass classification on the error (MAE or NME). This table is reproduced from (Shah et al. 2022).
Direct regression
Multiclass classilication
Benchmark
KC layer
FFC layers
layer
2 KC Iyyrrs
LFHI
4.49
LPH2
5,65
5.59
5.31
5.42
FLDI
3.60
3.63
3,58
3.56
FLD2
3.54
3.58
3.51
3.62
FLD3
4.50
FLD4
1.51
1,51
1.56
1,33
2.44
2.35
2.75
2.81
3.38
PN
4.24
4.35
4.56
5.74
(reproduced from (Shah et al. 2022) to show the impact of additional fully connected layers in direct regression and multiclass classification. Table 7 provides the error (MAE NME) for direct regression and multiclass classification with one or two fully connected layers after the feature extractor. As shown in the table, increasing the number of fully connected layers in direct regression and multiclass classification does not reduce the eтrоr for most benchmarks (possibly due overparameterization).
A.5. COMPARISON WITH DEEP HASHING APPROACHES
Deep supervised hashing approaches use neural networks as a hash function and learn hash codes in end-to-end manner. The loss function for deep supervised hashing is designed to preserve the similarity between inputs in the hashing space. Often, these approaches use the label information determine the similarity between images (i.e., same label) (Liu al.) 2016)] Xia et al., 2014). Some deep hashing approaches have proposed to augment the loss function with classification loss improve the performance. We adapt two widely used deep-hashing approaches regression and compare RLEL with deep hashing approaches.
Liu et al. 2016) proposed deep supervised hashing (DSH) approach with loss function based on the pairwise similarity between images. The proposed approach introduces a loss function to preserve the similarity between output codes for similar training images (e.g., same class) and maximize discriminability between output codes for different training images (e.g., different class). Further, they propose using relaxation on the binary output and a regularizer to encourage the output code to be close to discrete values +1/ 1. The hamming distance between output codes can be computed for binary-like outputs using the L2 norm. We use DSH for regression with some modifications (DSH-reg). We used the quantized label to determine the class of training sample.
Lai et al. (2015) proposed triplet ranking loss to learn a hash function that preserves relative similarities between images (SFLH). For images (I, I+, where I is closer to I+ than I-, the loss function designed to encourage higher hamming distance between codes for (I, I-) than
(T, T+). For classification datasets, triplets typically formed using two images from the same class
17

---

Published as a conference paper at ICLR 2023
and one from different class (Norouzi et al. 2012). They proposed to piece-wise threshold function to encourage binary- outputs.
We the above approach (SFLH) for regression with few modifications (SFLH-reg). To generate triplets, we pick sets three images from a given batch and determine the similarity between images using differences between the label values. We K triplets for minibatch of K training samples.
Further, for both DSH-reg and SFLH-reg, we augment the loss function with regression loss. We add fully- connecte layer between the output code and prediction. The MSE loss between the final outputs and target labels is added to the loss function (DSH-reg-La) SFL.-ree-LL2).
Table Comparison of RLELwith different deep hashing approaches adapted for regression.
Method
MAE
DSH-reg
71.3
DSH-reg-LJ L2
4.11
SF-.H-reg
69.8
SFLH-reg-La L2
4.73
RLEL ( only R1
3.93
RLEL RI + R2)
3.55
Table compares the modified deep hashing approaches with RLEL. The gap between loss functions with and without regression loss is significant, which shows that a loss function that only aims to preserve the similarity between output codes is sufficient and needs account for the error between decoded output and target (i.e., regression loss). RIEEL results in a lower error as it is designed for regression problems that account for classifiers' nonuniform error probability distribution.
Regularizer R1 encourages the distance between output codes for images to be proportional to the difference between label values, similar to pairwise or ranking-based loss functions proposed by deep hashing. However, deep hashing approaches use the hamming distance between binary outputs. As we show in Section 3.2 the hamming distance between codes does not account for the error probability of classifiers. Thus the confidence of the classifiers. R1 does not use regularizer or nonlinear activation on the output codes encourage binary-like outputs, typically done in deep hashing approaches. In contrast, show that suitable regression codes can be learned by not using this constraint. Thus RLEL with only RI regularizer results in lower error than deep hashing approaches.
the L1 distance between the real-valued outputs to account for
1. EVALUATION
Table 9: Comparison of RLELwith different regression approaches using Geometric mean and Pearson coefficient as evaluation metrics.
RLEL
Direct Regression
Multiclass Classification
GeoMean
Pearson Coeff.
GeoMean Pearson Coeff.
GeoMean Pearson Coelf.
LFHI
1.95
97.68
2.91
97.10
2.30
94.60
LFH2
2.09
92.22
2.49
91.06
2.40
88.76
FLDI
0.96
99.94
1.07
99.93
99.93
FLDI s
1.31
99.87
6.38
99.81
1.81
99,80
FLD2
1.92
99.97
2.12
99.97
2.07
99.97
FLD2
2.44
99.96
3.03
99.98
3.22
99.94
FLD3
0.96
99.99
1.05
99,99
1.01
99.97
FLD3_s
1.21
99.98
1.56
99.97
1.37
99.97
Table 9 compares RLEL with direct regression and multiclass classification using geometric mean and Pearson coefficient as evaluation metrics. The geometric mean represents the geometric mean of absolute enror for the test dataset. The Pearson coefficient represents the correlation between the target and predicted labels for the test dataset. As shown the table, RLEL results in significant reduction in the error compared to other generie regression approaches.
18

---

Published as a conference paper at ICLR 2023
10.0
Mean=8.195 Veriance=0.066
Frequenc
7.5
Frequency
Mean=8.180 vanance-0.06)
5.0
2.5
0.8.0
7.0
7.5
8.0
8.5
9.0
9.5
10.0
8.0
6.5
7.0
7.5
8.0
8.5
9.0
10.0
[|0..all
[|P...lI
(a) LFH2 benchmark Pitch label
(b) LFH2 benchmark Roll label
10.0
Frequency
7.5
Mean=1.549
Variance=0.594
5.0
2.5
0.0
0.5
1.0
||D
(c) LFHI benchmark Roll label
Figure 9: and (b) plot the distribution of |D...|| |I for LFH2 benchmark. (c) plots the distribution of || for LFHI benchmark. Here variance is very low, which suggests that the assumption |. ll || x
[1, .]. E N] are for label values with very few (or even zero) training examples.
NJ is valid. For the LFHI benchmark, the variance higher than LFH2. However, all outliers
A.1.7 THEORETICAL ANALYSIS OF PROPOSED REGULARIZATION FUNCTIONS
Regularization function R2:
We used matrix D instead of label encoding F. to apply regularizer R2 in equation 4 We insight into this decision as follows. First, note the output encodings are multiplied with D to generate the correlation vector C; (Figure 3). We use the multiclass classification loss between C2 and the target labels for training. Due to this, label encoding E and decoding matrix D are related, and use of matrix D proves to be effective for regularizer R2. We further explain this in detail below.
Let E represent an encoding matrix of size N x M. Each row EA:: represents the encoding output when the label k. D is the decoding matrix of size M x N. Let C₁ represent the output correlation row vector of size x N when the target label is k. Here, C% is obtained by multiplying Ek;; with D (Figure 3)
C2 = ER.
(6)
Since we apply softmax on the output vector to find the predicted label (Figure 3 ideally, Ot should have the highest value as the target label value is k.
Ok Ög, where, + k x E (1,
D... Ex..D..e where, # K, [1,2,..,NN) (Using equation 6). Let 6R rep- resent the angle between row vector Ft.: and column vector D This leads to the below equation:
|| III... ||Be__llDD,,ss|o((((..))) (cos(O..x))
D...//os).... where, x + k, (1,2,
Shah al. (2022) used a hand-crafted decoding matrix D with an equal number Is and Os each column for -enodee labels. Hence the L2 norm of each column is the same. In label encoding learning, parameters of matrix D are learned during training and are not constrained to have the same L2 norm for each column. However,
observe similar trend empirically. Figure Plots the distribution D..==l || for different benchmarks. As shown in the figure, for most benchmarks, observe small variance in the distribution of DD.. ||- Based this intuition and empirical
19

---

Published as a conference paper at ICLR 2023
validation, assume that ||D..=l| = ||D for [1 and y E [1 to simplify the analysis.
Using this assumption in equation 7 leads to the following inequality:
cos(O cos(0),)), where, k (1,2,..,,N) N)
Thus the cosine similarity between ER.: and D., should be the highest to predict the label k. The optimization process to reduce the loss between the target and prediction will try to maximize this cosine similarity. the best case, the angle between Ex,: and D.,k will be zero, and both vectors are parallel.
This simplification leads to the following relation between E and D.
Ek.- = tDikk where & > 0
Similarly, Es+1,: t' 1 where t'
Since L and /' both are positive values, reducing Di,e DI,K-1 also reduces Ex,i E
Regularizer rule R2 proposes to regularize the number of decision boundaries by regulariz- ing Ei Ej=1 Egi Ejtial as per equation 2 Based on the analysis above, regularizing EM Σソ= Dis Di.jta helps with the above goal as Ej,- Ej+1, reduces with Di,- Dij+t1-
Regularization function RI:
The first property suggests |i;;; - Ej,ll1 & - jl.
So ideally, ||E;; Ej:111 = Ali -
Since F..; is average of 2; for samples with label value (equation 1. the above condition leads
11 2y = Aly Mi
(8)
Based on this requirement, we add regularization function max(0, Alys - リリパ - -112,-22 11. which
penalizes the encodings if || 2,11 Aly; yリy- It does strictly impose equation 8 However,
approximately imposes the constraint as per shown in empirical verification Section A.1.1
Our intuition behind using the scaling parameter is based on binary-encoded labels. For two adjacent labels (i.e., | - ツy = 1), the loss function encourages || - 22 1 to be greater than 2. Here, 2 is the output encodings. In the case of binarized label encoding (-1 1 if Z < 0 and +1 if > 0), ||2, Z₁ = signifies that two encodings differ at least one bit.
A.1.8 IMPACT OF THE NUMBER OF QUANTIZATION LEVELS (N)
The number of quantization buckets is treated as design parameter for binary-encoded labels. Shah et al. (2022) showed that the error changes with the number quantization levels. Fewer levels introduce quantization error, and more levels increase the number of bits in the encoding. They showed a trade-off between these two factors to decide the number of quantization levels.
Our work focuses the design space of encoding and decoding functions. Hence use the same values for the quantization levels (NV as BEL Shah et al.) (2022). Parameter tuning can be integrated into hyperparameter tuning included in the optimization process.
We further analyze the effect of the number quantization levels for RLEL. Table 10 shows the NME (Normalized Mean Error) for different values of N for FLD1 benchmark.
This suggests that the proposed method RLEL is less sensitive to the number of quantization levels for higher values. For RLEL,, the decoding matrix that converts the encodings the predicted label
20

---

Published as a conference paper at ICLR 2023
Table Impact the number of quantization levels on error for FLD1 benchmark
Quantization levels (N)
NME
32
3.49
64
3,36
128
3.36
256
3.36
384
3.37
512
3.37
Table 11: Effect of dataset size the error for FL.D1 benchmark.
%ataaset used
RLEL
BEL
Difference (RLEL-BEI) BEL))
100
3.36
3.35
0.01
80
3.43
3.42
0.01
60
3.53
3.47
0.06
40
3.77
3.72
0.05
20
4.08
4.04
0.04
10
4.71
4.63
0.08
also learned during the training (Figure 3). This matrix of size M x N, where each column represents the weight parameters for one quantization level. One possible reason for the above results that matrix learns the number of quantization levels suitable for this problem.
There is potential learn the number of quantization levels and non-uniform quantization using the proposed RLEL framework. For example, Figure 3- step (4), fixed parameters are used to scale the correlation vector O% and find the expected prediction ÿ.- These parameters represent quantization levels. One possible approach to learning the quantization levels is to make these parameters trainable. In this case, 1.1/1. loss between the expected prediction ÿ; and target labels y; can be used to train the network.
A.1.9 IMPACT OF DATASET SIZE ON ERROR FOR RLEL AND BEL
In order compare the effect of dataset size on encoding design, we run BEL and RLEL approaches with the same training loss function (cross entropy loss in equation 5) We take the dataset FLD1 and use fraction of dataset for training. The entire test dataset is used for testing here. Table TT summarizes the error achieved by RLEL and BEL for different fractions of the training dataset. The evaluation shows that the gap between the performance of RLEL and BEL decreases with the increase in dataset size, which suggests that RLEL might be able to achieve lower error for larger datasets.
A.1.10 COMPARISON OF LEARNED AND MANUALLY DESIGNED ENCODINGS
We visually compare the encoding learned by RLEL with BEL manually designed code for one benchmark. Figure TO shows the learned and manually designed encodings. Here, row k represents an encoding for label k. Column represents the bit values for classifier-k over the numeric range of labels. We notice some common characteristics between both encodings. For example, the codes for nearby labels differ by fewer bits than faraway labels. Both the codes also have fewer bit transitions (0 - 1 and 1 + O transitions in column). These characteristics in the learned encodings are encouraged by the proposed regularizers RI and R2. There are a few differences between learned and hand-crafted encodings. In contrast to hand-crafted labels, encodings for adjacent labels do not differ in some cases, where hand-crafted encoding assures at least one or two bits of difference between adjacent labels.
A.2 LABEL ENCODING DESIGN
We evaluate different label encoding design approaches, including simulated annealing and autoen- coder. These approaches have been used to design encodings for multiclass classification by prior works (Song al. 2021 Cissé et al. 2012). We adapt these approaches design encodings for
21

---

Published as a conference paper at ICLR 2023
222
C000OAA 8
40 20 200 2437 1
N
11 1311111433 1617 20
Cooe
(b) Hand-crafted encoding (BEL) Figure 10: (a) and (b) give examples of learned and hand-crafted encodings. Here, row l represents an encoding for label k. Column represents the bit values for classifier-k over the numeric range of labels.
(a) Learned encoding (RLEL)
regression tasks and compare RLEL with these code design techniques. This section provides the methodology for simulated annealing and autoencoder-basse label encoding design.
A.2.1 SIMULATED ANNEALING
Simulated annealing is probabilistic approach to find global optimum of a given function. It is often used for combinatorial optimization, where the search space is discrete. Algorithm| I represents
Algorithm Simulated annealing for encodings design
Input: Kmax, T, M. N; Output: Ce {0, 13 MXN;
1: C0 (0 where Pr(CO.., 0) Pr(COs,j 1)
2: T=T
3: for k Kmxx do
4:
Cnew = Move(C)
5:
D D(C E(C)
6:
if or Random(O,1) then
7:
C= Cnew
8:
end if
9:
t= T=T/(k+
10: end for
the pseudo-code for label encoding design using simulated annealing. This algorithm takes two hyperparameters. Kmax (number of iterations) and T (initial temperature). It designs a code matrix of size N x M, where N is the number of values and M is the number of bits. Fach row A: in this code matrix represents encoding for value Code matrix initialized with a random matrix of and [L.
For each iteration, a new code matrix Cnew is sampled from the current code matrix C using a Move function 4h. For example, move function can be designed to randomly flip a few bits in C. The difference between the errors of the current and new code matrix is measured (Line 5). The error of a code matrix, i.e., expected regression error for this problem, is measured using function E. For example, E can be replaced by training a regression network for a given code matrix to measure the regression emror. Finally, the current code matrix is updated with the matrix Crew
22

---

Published as a conference paper at ICLR 2023
Analytical model's MAE
8
10
Trained network's MAE
Figure 11: Comparison of Mean Absolute Error (MAE) approximated the proposed analytical model and trained network for different code matrices, Each point this plot is different code matrix.
probabilistically. The probability is determined using the decrease in regression enror and current temperature (Line The current temperature is updated for each iteration (Line D).
There are mainly two design parameters in the above algorithm: the error measurement function and the move function Move. We further explain the design of these functions.
Error measurement:
We used the expected absolute error between targets and decoded predictions for a given code matrix its error, the goal is design code matrix that results in lowest regression error. However, training regression network for each sample code matrix to measure its regression error is computationally expensive and time-consuming (~~ 200 training runs). Hence we use an analytical model to estimate the regression error for given code matrix.
Regression error is the absolute error between targets Q; and decoded predictions Q. For given target Qi and target code B; CQ.... the predicted code (B,) will be erroneous due to classification emros. This erroneous predicted code is decoded to predicted value (Q:). The following equation is used predict Q in appcteddcorrrelatio--alee based decoding (Shah al. 2022).
e B.
EG (Bi,
kak, where
E CB.
The regression error can be estimated given sufficient samples of B; and B,
Shah al. (2022) provided an approximate model of classification errors. They showed that for each classifter, its error probability distribution can be approximated using a combination of Gaussian distributions, where p the number of bit transitions. Each Gaussian distribution is centered around bit transition. For example, for bit-k in unary code with bit transition between Q = k and Q = & + 1, error probability of the classifier-k for different target labels Q can approximated as:
e (Qi) rfN((ne.))) (Q:), where, μı = 0.5
(10)
B, can be sampled for the given Q: and C' using the above error proo-pprbbbbilty model. Equation 9 then used to find the decoded prediction Q.. We measure the expected absolute error between Q and Q: using 100 N samples.
We further verify the validity of this analytical model by finding the correlation between regression error measured by this model and trained networks. Figure II plots the analytical regression error versus actual regression error for FLD_1 benchmarks. Here, each point is for a different code matrix. The -axii represents the absolute emorr approximated by the proposed analytical model. The X-axis
23

---

Published as a conference paper at ICLR 2023
Random flip
10
Error-based flip
Approximate error
2000
4000
6000
8000
10000
Iteration
Figure 12: Comparison of convergence of random-fih flip and proposed error-based flip functions.
S
Fully Connected Layer
TanH- Activation
Figure 13: Network architecture for autoencoder-based encodings design.
represents the absolute test error of trained network for given code matrix. The figure shows that the proposed analytical model for error measurement approximates error with significant speedup.
The move function flips some bits in the current code matrix to sample a new one. naive approach would be to randomly flip bits. We further optimize the move function to consider the regression task objective. For given code matrix, using the proposed analytical model, find matrix F of size N x N, where Ft, j jl Thus, each element represents a pair (Ci.., Cj.) of encodings' contribution to expected error. We select top-b pairs from this matrix. For each pair of encodings, find bit-positions that have equal bit-values between two encodings, and randomly selected bit-position from this list is flipped in encoding Ct... This procedure increases the hamming distance between pairs of encodings that contribute the highest to the regression error. Figure 12 compares the convergence of the proposed move function and random-fit random-flip-asse based move function. Here the Yaxii represents the approximated error for the current code matrix, and Xaxxis represents the iteration identifier. The figure shows that the proposed error-based move function results in faster convergence and lower error.
Move function:
(Roound(D((B... C)) jlQ B, Ct.).
We use the proposed move function with the analytical model to approximate regression error in Algorithm to design label encoding for regression using simulated annealing.
A.2.2 AUTOENCODER
Cissé al. Cissé al. 2012) proposed an autoencoder-based approach to design encodings for multiclass classification problem. Figure represents the network architecture used for encodings design. Input S; is an N -ddimensional vector for class i. Here, each element S.Di] represents the similarity between class and j. The output of the bottleneck layer C, is the designed encodings for class
24

---

Published as a conference paper at ICLR 2023
For regression problems, network. The network is trained using SGD optimization, where each batch consists of randomly sampled and The following loss function is used for training:
St[p] l j|. Let W represent the weight parameters of the
= || - 5.11 + ||š; -,1l² + 30ax(0, b |IC; - Cjlli) + 1
(11)
Here, the first and second terms represent reconstruction losses for inputs S, and S,. The third term encourages a minimum distance of b between any pair of encodings to yield unique encodings for different classes. The fourth term is an L2-regularizer.
Once the network is trained, the real-valued encodings C; are converted to binary encodings such that has equal numbers of Os and Is. This formulation introduces three hyperparameters. We determine the number of bit transitions in the designed label encodings and select hyperparameters that result in the lowest number of bit transitions.
Note that this autoencoder network is decoupled from the regression network and design codes agnostic to classifiers' characteristics for given regression problem.
A.3 EXPERIMENTAL METHODOLOGY
We use 11 benchmarks covering four different regression tasks for evaluation. This section summa- rizes the experimental setup, including datasets, evaluation metrics, hyperparameters, and related work for each these tasks. also report the training time using an NVIDIA RTX 2080 Ti GPU with 1IGB of memory for each benchmark.
A.3.1 HEAD POSE ESTIMATION
In landmark-free 2D head pose estimation, for given 21) image, the head pose of a human is directly estimated terms of three angles: yaw, pitch, and roll. We use loose cropping around the center with random flipping for data augmentation. We use the ResNet50 network as the feature extractor. This network is initialized using pre-trained parameters for ImageNet (Russakovsky et al. 2015) dataset. During the training for RLEL the entire network, including the feature extractor, is trained.
Datasets: We use the evaluation methodology followed by prior works (Ruiz et al. 2018 Yang al. 2019). Two protocols are used for evaluation.
Protocol I (LFHI): This protocol uses the BIWI (Fanelli et al. 2013) dataset for training and evalua-
tion. This dataset consists of 15, 128 frames of 20 subjects. Random 70% - 30% splits are used for training and evaluation. The ranges of yaw, pitch, and roll angles are [-75", 7, 65% 885), and
[-55%, 45%), respectively.
Protocol 2 (LFH2): In this protocol, the network is trained using the 300W-LP (Zhu et al. 2016) dataset consisting of 122, 450 samples. AFLW2000 (Zhu et al. 2016) dataset is used for evaluation. The range of all labels is [-990, 99%] in this setting.
Evaluation metrics: We report the Mean Absolute Error (MAE) between the targets (14) and predictions (ft). Let N represent the number of samples, and P represent the number of labels (three head pose estimation). The MAE defined as:
N
MAE
Uh.j
(12)
Network architecture and training parameters: Table 12 summarizes the hyperparameters used for RLEL. The learning rate of the decoding matrix D kept 10x higher than the learning rate of the feature extractor. L2 regularization with weight of 0.0001 is used for direct regression.
Related work Head pose estimation is a widely studied problem. Existing task- ssscippecil ap- proaches propose different loss formulations feature extractors to improve the error. HopeNet (Ruiz et al. 2018) proposed a combination of regression and classification loss. SSR-Net (Yang et al. 2018 and FSA-Net (Yang et al. 2019) proposed stage-wise soft regression. QuatNet (Hsu et al.
25

---

Published as a conference paper at ICLR 2023
Table 12: Training parameters for LFHI.
Training rime hours)
Appronch
Label range/ Quantization lovels
Optimizer
Epochs
Batch size
Learning rate
Learning rate schodule
Yaw: [-75 755 /150,
Adam, weight decay=D, momentum
LPHI
Pitch: -65" 85" /150
50
0.0001
1/10 after 30 Egochs
0.5
1.0
2
Roll:
5550 ,45" /000
Adam, weight decay=0,
LFH2
1.99,,
1000,909)2000
20
16
0.00001
1/10 after 10 Epochs
2.0
0.0
momentum
2019) proposed to use MSE loss with custom ordinal regression loss, RAFA-Net (Behera et al. 2021) proposed an attention-based feature extractor architecture. Table 13 and Table compare the performance of RLEL with related work.
Table 13: Landmark-froe free 2D Head poses estimation evaluation for protocol (HPE1 and HPE3).
Appruach
Festure Extractor
#Params (M)
Yaw
MAE
SSR-Net-MD Yang et al. 2018
SeR-Net
1.1
4.24
4.35
4.19
4.26
regression)
FSA-Cape- Fusion (Ynng
2019 (Soft regression)
FeA-Net
2.89
RAFA.Net (Rehern
RAFA-Net
69.8
3.07
4.30
2.82
3.40
(Dircct Regression)
Direct regression (L2 loss)
ReaNer50
23.3
3.80
4.03
4,28g
4.22 = 0.35
(Shah 2023
ResNer50
3.80
RLEL
ResNet50
23.6
3.41
3.20
3.97
3.55 1: 0.10
Table Landmark-free 2D Head poses estimation evaluation protocol 2 (HPE2 and HPE4).
Feature Extractor
#Params (M)
Approach
Pitch
Roll
SSR-Net-MD Yang et 2018
SSR-Net
(Solt regression)
FSA-Caps-Fusion Yang
2019 (Soft regression)
PSA-Net
4.50
4.64
5.07
KAFA-Not Behhra al. 2021
RAFA-Net (HPE4)
69.8
3.60
4.92
3.88
4.13
(Direct Regression)
HopeNet* (ax = Ruiz
2018 (classification
ResNet50
23,9
5.44
regression loss)
Direct regression loss)
ResNet50
4.18
5.32
BEI Shahh al. 2021
ResNe150
23.6
4.54
5.76
3.96
4.77 0.05
R...I.
ResNet50
23.6
4.69
5.79
3,86
4.77 + 0.05
A.3.2 FACIAL LANDMARK DETECTION
Facial landmark detection focuses on finding the (x, y) coordinates of facial keypoints for given 2D image.
Evaluation metrics:
We report the Normalized Mean Error (NME) between the targets yı and predictions ői- Inter-ocular distance normalization is used for all datasets. For N test samples, P facial landmarks, and L normalization factor, the NME is defined as:
P
1 N
NME
Σ
Vi.j
(13)
P
L
Datasets: We use three datasets widely used for facial landmark detection: COFW (Burgos-Artizzu ct al. 2013), 300W (Sagonas et al.) (2013), and WFLW (Wu et al. 2018). HRNetV2 network architecture for feature extraction (Wang al. 2020) and the modified regressor architecture for
26

---

Published as a conference paper at ICLR 2023
label encoding proposed by BEL (Shah et al. 2022) are used in this work. Random flipping, scaling (0.75 1.25), and rotation (=30) are used for data augmentation. The COFW dataset consists of 1, 345 training and 507 testing images annotated with 29 landmarks. The training set of the 300W dataset has 3, 148 images annotated with 68 facial landmarks. This dataset provides four test sets: full test set, common subset, challenging subset, and the official test set with 300 indoor and 300 outdoor images. WFLW dataset is a comparatively larger dataset with 7, 500 training and 500 testing images. Each image is annotated with 98 facial landmarks. The test set is divided into six subsets: large pose, expression, illumination, make-up, occlusion, and blur.
Training parameters: Table 15 provides a summary of all the training parameters. The learning rate of the decoding matrix D is kept 20x higher than the learning rate of the feature extractor. The HRNetV2-W 18 network is initialized with pretrained weight parameters for the ImageNet dataset. We refer to HRNetV2-W18 evaluated COFW as FLDI/FLD1 s, 300W as FLD2/FLD2 s, and on WFLW as FLD3/FLD3
Table 15: Training parameters for facial landmark detection for HRNe1V2-WIL W18 feature extractor.
Learning
(BEL/Direct
Dataset/ Benchmark
Learning schedule
Training time
Optimizer
Epachs
Batch size
regres- sion/Multiclass classification)
CK
(GPU hours)
Adam, weight decay=O,
COFW/ FLD1
60
0. 0.0005/0.0000
1/10 after 30 and 50 Epochs
3.0
0.0
1
0.0003
momennumm 0
Adam, weight decay=U,
COFW/ FLDI_s
0.0005/0.0002
1/10 after 30 and 50 Epochs
0.0003
momentum = 0
Adam, weight
0.0007/0.0000
1/10 after 30 and 50 Epochs
300W/ FLD2
docay=0,
60
5.0
1.0
3
0.0003
momentum 0
300W/
Adam, weight decay=0,
0.0007/0.00033
1/10 after 30 and 50 Epochs
60
5.0
0.05
3
HLD2_s
0.0003
momentum 0
Adam, weight decay=0,
0.0003/1.0003/
1/10 after 30 and 50 Epochs
WFLW/ FLD3
60
8
0.0
0.1
5
0,0003
momentum 0
WFLW/ FLD3_s
Adam, weight docay=0,
0.0003/0.00m 0003//
1/10 after 50 Epochs
60
8
5.0
0.1
5
0,0003
momentum
Related work Facial landmark detection is a widely studied problem. A common approach is to use heatmap regression, where the target heatmaps are generated by assuming Gaussian distribution around the ground truth landmark location. Prior works proposed the use of binary heatmaps with pixel-wise binary cross-entropy loss (Bulat & Tzimiropoulos (0166). HRNet (Wang) ct al. 2020) proposed a feature extractor that maintains high-resolution representations and uses heatmap regression. AWing (Wang al. 2019) proposed modified heatmap regression loss function with adapted wing loss. AnchorFace (Xu et al. 2020) used anchoring of facial landmarks on templates. LUVLi (Kumar al. 2020 proposed a landmark's location, uncertainty, and visibility
likelihood-based loss. Table 16 18 compare RLEL with related work.
Table 16: Facial landmark detection results COFW dataset (FLDI).
#Params/ GPocps
Approach
Feature Extractor
Test NME
FRo
LAB (w B) at 2018)
Hourglass
25.1/19.1
3.92
0.39
AWing (Wang et
2019
Hourglass
25.1/19.1
HRNerV2-W W 18 (Wang et al 2020
W11
9.6/4.6
3.45
0.19
(eestmap regression)
Direct regression (12 Joss)
HRNeIV2-W18
10.244.7
3.96
4 0.02
0.29
Direct regression (L loss)
HRNetV2-W1B
10.2/47
3.60
4 0.02
0.29
BEL
HRNerV2 W18
10.6/4.6
1
R...
HRNerV2.W18
10.6/4.6
3.36 + 0.01
0.20
Uses dillerent data augmentation for the training
27

---

Published as a conference paper at ICLR 2023
Table 17: Facial landmark detection results on 300W dataset (FI.D2..
#Params/ GPlops
Approach
Feature Extractor
Test
Common
Challenging
Full
DAN (Kawalskii el 2017
3.19
5.24
3.59
LAB (w BT Wu et
2018
Hourglass
25.1/19.1
2,98
5.19
3.49
FnchorFace (Xu et al.
2020
ShullleNet-V. V2
3.12
6.19
3.72
AWing Wang
2019)
Hourglass
25.1/19.1
2.72
4.52
3.07
LUVLi [Kumar al
2480
CU.Neet
2.76
5.16
3.23
HR W18 [Wang
2020
(Heatmap regression)
HRNaVV2-I88
9.644.6
2.87
5.15
3.32
Direct regression (1.2 loss)
HRNeVV2 W18
10.2/4.7
4.40
3.25
5.65
3.71 1: 0.05
Direct repression loss)
HRNaV22WW1S
10.2/4.7
3.10
::
BEL Shahh al. 2022
VR WI8
11,2/4.6
4.09
2.91
5.50
3.40 1. 0.02
RLEL
HRNaV22-118
11,2/4.6
4.03
2.90
5.39
3.37 : 0.02
=Uses different augmentation the training
Table 18: Facial landmark detection results (NME) on WFLW test (FLD3) and 6 subsets: pose, expression (expr.). illumination (illu.), make-up (mu.), occlusion (occu.) and blur. = 10 is used for BEL.
Parramo/ Gillos
Appreach
Feature Extractor
Test
Pose
Expr.
Ill..
MU
Occu.
Blur
LAB (w B) Wa cl.
Hourglass
25.1/19.1
5.27
10:24
5.51
5.23
5.15
6.79
6.32
AnchorFace Xu Auettal
HRNaWV2W11
-/5.3
4.32
7.51
4.69
4.20
4.11
4.82
2020
AWing Wang
2019 m
Hourglass
25.1/19.1
4.36
7.38
4.58
4.32
4.27
5.19
4.96
LUVLi (Kumar et al.
CU-Net
4.37
2020
HRNNet22 18 (Wang
2020 (Heatmap
HRNaV2-WIS
9.6/4.6
4,60
7.94
4.85
4.55
4.29
5.44
5.42
regression)
Direct regression loss)
HRNaIV2-I18
10.24.7
4.64 = 0.03
8.13
4.96
4.49
4.45
5.41
5.25
REL. (Shah al. 2022
HRNaV2 WI8
11.7/4.6
4.36 + 0.02
7.53
4.64
4.28
4.19
5.19
5,05
RLEL.
HRNeAV2-WI WIS
11.74.6
4.35 1 0.01
7.57
4.57
4.36
4.19
5.25
5.07
Uses different augmentation the training
A.3.3 AGE ESTIMATION
This task focuses on predicting person's age from a given 2D image. MAE (Equation 12p and Cumulative Score (CS) are used as the evaluation metrics, and ResNet50 (He et al.) (016) is used as the feature extractor. CSO the percentage of test samples with absolute error less than years.
Datasets MORPH-II (Ricanek Tesafaye 2006) and AFAD (Niu al. 2016) datasets are used for evaluation. We follow the protocols for preprocessing and data augmentation of datasets as per prior works (Shah et al. 2022, Raschka 2018). MORPH-II dataset consists 55, 608 images with random split of 39, 617 training, 4, 398 validation, and 11, 001 test images. The AFAD dataset consists of 164, 432 images with random split of 118, 492 training, 13, 166 validation, and 32, 763 test images.
Training parameters: Table 19 summarizes the training parameters for A AE1 (MORPH-II) and
AE2 (AFAD) benchmarks. The learning rate of the decoding matrix D is kept 10x higher than the learning rate of the feature extractor. 1.2 regularization with weight of 0.001 used for direct regression. Training for AEI and AE2 consumes 2 and 7 hours, respectively.
Table 19: Training parameters for age estimation using MORPH-II and AFAD datasct
Bench- mark
Optimizer
Epochs
Batch size
Learning rale
Learning schedule
9
a
Adam, weight
AE1
decay=0,
50
64
0.0001
0.0
2.0
momentum=0)
Adam, weight decay=0.
AE2
64
0.0001
0.0
5.0
momentum=()
28

---

Published as a conference paper at ICLR 2023
Related work Different approaches including ordinal regression Niu al.
2016 Cao al.
2020; Pan et al. 2018; Gao et al. 2018), soft stage-wise wise regression Yang et al. 2018 2019 soft
Gao
labels (Diaz& & Marathe 2019) have been proposed for age estimation. OR-CNN (Niu et al. 2016
and CORAL-CNN (Cao et al. 2020) proposed ordinal regression by binary classification with threshold-based encodings (i.e., unary codes). DLDI (Gao al. 2018) augmented the loss function with KL-divergence between softmax output and soft target probability distributions. MV-Loss (Pan
al., 2018) proposed to penalize the prediction based on the variance of the age distribution. We
compare CLL with related work in Table 20 and Table 21
Table 20: Age estimation results MORPH-II dataset (AEI).
MORPH-11
Approach
Feature extractor
Parrameters (M)
MORPH-I (MAB)
DR-CNN NNi al
2016
(Ordinal
1.0
2.58
0,71
regression binary
classincation)
MV Loss Pun et
2018
(Direct
VGG-16
rogressioo)
DLDL (Gao 2018) (Ordinal
ThinAgeNet
3.7
1,96*
Pegression with multi-class classilication)
CORAL-CNN (Cao et (2120] (Ordinal
ResNe144
21.3
2.49
binary classiication)
Dircet Regreasion (12 Loss)
ReaNe100
23.1
2.37 + 0.01
0,903 1 0,002
BEL
ResNe150
0.001
RLEL
ResNet50
23.1
2.28 + 0.01
0.901 + 0.002
Table 21: Age estimation results AFAD dataset (AE2).
Approach
Feature extractor
#Parameters (M)
AFAD (MAE)
AFAD (CS9 5)
OR-CNN NNu et al 2016 (Ordinal
3.51
0.74
rogression bonury
CORAL-CNN (Cao ct
2020 (Ordinal
ResNe134
regression by binary classilication)
Direct Regression (L2 Loss)
ResNe150
23,1
3.16 + 0.02
0.810 4 0.02
BEL (Shah et al. 2022
ResNer50
23.1
3.11 1. 0.01
0.23 1 0.001
RLEL
ResNe150
23.1
3.14 = 0.01
80.78
0.002
A.3.4 END-TO-END SELF DRIVING
For the regression task of end-to-end autonomous driving, we use the NVIDIA PilotNet dataset, and PilotNet model (Bojarski et al. 2016). In this task, for a given image of the road, the angle of the steering wheel that should be taken next is predicted. MAE (Equation 12) is used as the evaluation metric.
Dataset The PilotNet driving dataset consists of 45, 500 images taken around Rancho Palos Verdes and San Pedro, California (Chen). We use the data augmentation technique used by prior works (Bo- jarski al. 2016).
Training parameters Table 22 summarizes the training parameters. The learning rate of the decoding matrix D is kept 10x higher than the learning rate of the feature extractor.
Table 22: Training parameters for end-to-end autonomous driving using PilotNet.
Learning rite scbedule
Optimizer
Epochs
Batch size
Learning rate
/
weight decay=le-5,
50
64
at spachs
0.0
momentum=0
Related work End-to-end autonomous driving is an interesting task with increasing attention. PilotNet (Bojarski et al., 2017) used small, application-speeife network. We compare RLEL with the baseline PilotNet architecture in Table 23
29

---

Published as a conference paper at ICLR 2023
Table 23: End-to-end autonomous driving results on PilotNet dataset (PN) and architecture (Bojarski et al. 2017
2016).
Approach
Feature extractor
#Parameters (M)
MAE
PilotNet Bojarski
PilotNet
4.24
BEL (Shah et 2022)
PilotNet
1.8
3.11
0.01
RI RIFFI.
PilotNer
1.8
2.94
0.01
30


Reviews:
[{"review_id": "wCXM97Jo2y", "reviewer": "Reviewer_TMxF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on deep regression by using label encoding. Specifically, this paper proposes Regularized Label Encoding Learning (RLEL) for end-to-end training of an entire network and its label encodings. RLEL contains two regularization terms, designed to encourage encodings with certain properties. The first one encourages the rule that the L1 distance between label encodings should increase as the difference between two labels increases. The second one encourages the rule that the L1 distance between label encodings for adjacent target label values should be minimized. The effectiveness of the proposed method is demonstrated by a lot of experiments.", "summary_of_the_review": "Overall, this paper proposes a novel label encoding method that is reasonable to solve the deep regression problem. There are also some shortcomings in this paper as mentioned above. The authors should carefully check the manuscript repeatedly and provide necessary details and discussions to further polish this paper.", "strength_and_weaknesses": "Strengths:\n- This paper proposes a novel label encoding method to solve regression and seems can alleviate the disadvantage of the previous label encoding method, the space of label encodings for regression is large.\n- This paper introduces two simple yet meaningful rules for label encoding and designs corresponding regularization terms to encourage label encodings to satisfy the rules.\n- Comprehensive experimental results on various datasets clearly demonstrate the effectiveness of the proposed method.\n\nWeaknesses:\n- According to Eq. (5), the generation of soft target labels is affected by the batch size K. It seems impossible. So, I guess it should be the number of target labels N rather than K in the generation of soft target labels. \n- The authors should use unified representation. For example, the correlation vector is denoted by $\\hat{C}_i$ in most cases. However, $C_i$ is used in Eq. (5) and Fig. 3.\n- The authors assume $y_i \\in [1, N]$ for simplicity as the real-valued targets with any arbitrary numeric range can be scaled and shifted to this range. However, they do not provide details and discussions of N. Obviously, different N would cause different performance.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "vbJpw9QZTR5", "reviewer": "Reviewer_XTEa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces Regularized Label Encoding Learning (RLEL) for end-to-end training of an entire network and its label encoding, by combining continuous label encodings space with regularizers. The proposed method is extensively demonstrated on 11 benchmarks. ", "summary_of_the_review": "The authors propose a simple yet effective end-to-end training to simultaneously learn the label encoding and features. The experiments are solid and extensively demonstrate the effectiveness of the method. Therefore, I recommend borderline acceptance. ", "strength_and_weaknesses": "strength:\n\n(1) The authors tackle the problem of deep regression through the lens of binary classification. It is also interesting and technically sound to extend the conventional hand-crafted encoding method to be differentiable. Overall, I like this simple yet useful idea. \n\n(2) The experiments are extensive, and the proposed end-to-end encoding method has competitive performance compared to hand-crafted. The ablation study also demonstrates the effectiveness of two proposed regularizers.\n\nWeakness: \n\n(1) In the experiments, manually designed BEL outperforms RLEL in many cases. I'm curious why an end-to-end training method seemingly achieves a suboptimal solution, compared to a hand-crafted method. Can the authors provide some intuition on it? Also, how different is it between the learned encodings and (better performed) hand-crafted encodings? \n\n(2) I encourage the authors to discuss the efficiency of the method. What is the time cost to incorporate the label encoding learning with regularizers, compared to other hand-crafted encodings? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rDCXvCnloLh", "reviewer": "Reviewer_vuKX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the deep regression problem and proposes a new automated label encoding learning framework along with two regularizers. In specific, the authors relax the assumption of binarized label encodings of the BEL method and propose to search for continuous label embeddings. Moreover, two regularizers are introduced. The first restricts the learned label encoding to preserve the distance of the bits. The second is to enrich the bit transitions. Experiments show the proposed method obtains competitive results to manually designed label encodings.", "summary_of_the_review": "See above. ", "strength_and_weaknesses": "Pros:\n\n1. The handcrafted label encoding, though effective, increases the difficulty of the practical deployment of deep regression models. This work proposes an automated label encoding learning method that is much more practical. \n\n2. The experimental results are promising as the proposed method achieves competitive or better performance than manually designed label encodings. \n\n3. The paper is well-organized and mostly written clearly. \n\nCons:\n\n1. While the presentation is generally good, the methodology is hard to follow. Some notations are not clearly defined. For example, the authors used B, Z, E to denote the label encodings. What are their differences? Why E is the average of E? What is \\mathcal{E}(Q_i) on page 5?\n\n2. The 2.1 section (and Figure 2), while important and informative, isn't really clear to me. Why is the bit transition from 1->0 and 0->1 important for label encoding? Also, how does the bit position measures the binary classifier's decision boundary? Personally, I recommend the authors polish this section for a clear explanation of the important properties of label encodings.\n\n3. I noticed that the softmax operator and the cross-entropy loss are used. It suggests the regression network learns a multi-class classifier. While this paper frequently mentioned the binary classifier, I would like to see the difference between these two designs. In particular, it seems a multi-class classifier may reduce the complexity of the encoding space.\n\n4. The authors propose to regularize the matrix D instead of the original label encodings in Eq. (4). While empirically effective, there are no theoretical insights on why it works. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "1wVnI721VaU", "reviewer": "Reviewer_ZxJ5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an end-to-end automated approach to find label encodings for deep regression problem. In my understanding, these tasks are usually using images as input and are outputting a continuous value. Due to the nature of deep regression, a general approach to regress directly does not show good results compared to task-specific approaches. Their work is largely based on a prior work [Shah 2022], in which 1) binary-encoded labels were introduced to encode quantized regression values 2) three desirable properties of suitable encoding/decoding functions were identified. This paper extends [Shah 2022]'s work by not using manually identified binary codes, but learn the continuous label encodings from the data directly. Their empirical experiments show that their learned-encodings and the new model result in lower or comparable errors to [Shah 2022] work.\n\n[Shah 2022] Deval Shah, Zi Yu Xue & Tor M. Aamodt. LABEL ENCODING For REGRESSION NETWORKS. ICLR 2022.\n\n", "summary_of_the_review": "Weak accept. The paper proposes an effective way to automatically learn continuous encoding to help improve deep regression model performance. If the author could give more theoretical analysis on why the proposed model encourages the expected encoder properties, I would be happy to increase my scores. ", "strength_and_weaknesses": "Strength: clarity and quality of writing. Extensive experiments.\n\nThe paper is well written with a pretty clear introduction of background and related work. I also like how the author motivates the underlying work by showing the challenges in the design space. By giving a literature review on the binary encoding's usage in regressor, I was able to understand why the properties of suitable encodings (identified by prior works) are important in such a work. By following this guidance to the properties, the author started to introduce the regularization functions to encourage such properties when using real-valued label encodings. The loss function is then introduced based on the regularizer design. The experiments use almost the same datasets (from Shah 2022) so that a fair comparison with its predecessor can be easily conducted. The author also compares with a few other label encoding design approaches and regression approach to show that their work is a generic regression approach and generalizes better with even a small dataset.\n\nWeakness: not strong originality.\n\nMost of the originality in this paper is to propose a learned continuous label encoding (compared to existing work on using manually designed binary encoding). From the regularizer design, the author identifies effective regularization functions to encourage the expected property within a continuous encoder. The proposed method seems effective but incremental in nature. \n\nQuestions to the author:\n1. How would you decide the number of quantization buckets ($M$ in the continuous predicted encoding $Z_i$)? Is there an optimal dimension size for various dataset?\n2. Computationally (during training or inference), is there any significant difference between RLEL and BEL?\n3. In equation (3), can you give more explanation that why you choose the magic number 2?\n4. It seems to me (by looking at Table2) BEL is reasonably performing well though it is manually designed. I kindly expect that adaptively learned encoder shall perform even better if more data is given during training. I am not familiar with the scale of the experimental dataset you have used. How many examples are there in each of them? Were you able to test your method in a significantly large-scaled problem (e.g., with many more training examples) to show the superiority of your model)?\n5. Does RLEL use uniform quantization (similar to BEL)? Is there something that can be done to adaptively select quantization level and non-uniform quantization?  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%