Label Encoding for Regression Networks
Authors: Deval Shah
Zi Yu Xue
Tor Aamodt
Venue: ICLR 2022 Spotlight (2022)
Keywords: Regression
Label encoding
Output codes

Abstract: Deep neural networks are used for a wide range of regression problems. However, there exists a significant gap in accuracy between specialized approaches and generic direct regression in which a network is trained by minimizing the squared or absolute error of output labels. Prior work has shown that solving a regression problem with a set of binary classifiers can improve accuracy by utilizing well-studied binary classification algorithms. We introduce binary-encoded labels (BEL), which generalizes the application of binary classification to regression by providing a framework for considering arbitrary multi-bit values when encoding target values. We identify desirable properties of suitable encoding and decoding functions used for the conversion between real-valued and binary-encoded labels based on theoretical and empirical study. These properties highlight a tradeoff between classification error probability and error-correction capabilities of label encodings. BEL can be combined with off-the-shelf task-specific feature extractors and trained end-to-end. We propose a series of sample encoding, decoding, and training loss functions for BEL and demonstrate they result in lower error than direct regression and specialized approaches while being suitable for a diverse set of regression problems, network architectures, and evaluation metrics. BEL achieves state-of-the-art accuracies for several regression benchmarks. Code is available at https://github.com/ubc-aamodt-group/BEL_regression.


Published as a conference paper at ICLR 2022
LABEL ENCODING FOR REGRESSION NETWORKS
Deval Shah, Zi Yu Xue & Tor M. Aamodt
Department of Electrical and Computer Engineering University of British Columbia, Vancouver, BC, Canada [devalshah, fzyxue, aamodt (èce. ubc. ca
ABSTRACT
Deep neural networks are used for wide range of regression problems. However, there exists a significant gap in accuracy between specialized approaches and generie direct regression which a network is trained by minimizing the squared or absolute error of output labels. Prior work has shown that solving a regression problem with set of binary classifiers can improve accuracy by utilizing well- studied binary classification algorithms. We introduce binary-encoded labels (BEL), which generalizes the application of binary classification to regression by providing framework for considering arbitrary multi-bit values when encoding target values. We identify desirable properties of suitable encoding and decoding functions used for the conversion between real-valued and binary-encoded labels based on theoretical and empirical study. These properties highlight a tradeoff between classification error probability and error-correction capabilities of label encodings. BEL can be combined with off-the-shelf task-specific feature extractors and trained end-to-end. We propose a series of sample encoding, decoding, and training loss functions for BEL and demonstrate they result in lower error than direct regression and specialized approaches while being suitable for a diverse set of regression problems, network architectures, and evaluation metrics. BEL achieves state-of-the-art accuracies for several regression benchmarks. Code available at Ittps: thutbb.. com hte//uucaaaae aabbo- -group BEI, B regression
INTRODUCTION
Deep regression networks, which a continuous output predicted for given input, are traditionally trained by minimizing squared/absolute error of output labels, which we refer to as direct regression. However, there a significant gap in accuracy between direct regression and recent task-specializedd approaches for regression problems including head pose estimation, age estimation, and facial landmark estimation. Given the increasing importance of deep regression networks, developing generic approaches to improving their accuracy desirable.
regression problem can be posed as a set of binary classification problems. A similar approach has been applied other domains such as ordinal regression (Li & Lin 2006) and multiclass classification (Dietterich & Bakiri 1995). Such a formulation allows the use of well-studied binary classification approaches. Further, new generalization bounds for ordinal regression or multiclass classification can be derived from the known generalization bounds of binary classification. This reduces the efforts for design, implementation, and theoretical analysis significantly (Li & Lin 2006). Dietterich & Bakiri (1995) demonstrated that posing multiclass classification as a set of binary classification problems can increase eтror tolerance and improve accuracy. However, the proposed approaches for multiclass classification do not apply to regression due to the differences task objective and properties of the classifiers' error probability distribution (Section 2). On the other hand, prior works ordinal regression have explored the application of binary classifiers in a more restricted way which limits its application to a wide range of complex regression problems (Section D. There exists lack of generic framework that unifies possible formulations for using binary classification to solve regression.
In this work, we propose binary-encoder labels (BEL) which improves accuracy by generalizing application of binary classification to regression. In BEL, a target label is quantized and converted to binary code of length M, and M binary classifiers are then used to learn these binary-encoded

---

Published as a conference paper at ICLR 2022
labels. An encoding function is introduced to convert the target label to binary code, and decoding function is introduced to decode the output of binary classifiers to a real-valued prediction. BEL allows using an adjustable number of binary classifiers depending upon the quantization, encoding, and decoding functions. BEL opens possible avenues to improve the accuracy of regression problems with large design space spanning quantization, encoding, decoding, and loss functions.
We focus on the encoding and decoding functions and theoretically study the relations between the absolute error of label and binary classifiers' errors for sample encoding and decoding functions. This analysis demonstrates the impact of binary classifiers' error distribution over the numeric range of target labels on the suitability of different encoding and decoding functions. Based on our analysis and empirically observed binary classifiers' error distribution, we propose properties of suitable encoding functions for regression and explore various encoding functions on wide range of tasks. We also propose an expected correlation-baser decoding function for regression that can effectively reduce the quantization error introduced by the use of classification.
A deep regression network consists of feature extractor and regressor and is trained end-to-end. regressor is typically the last fully connected layer with one output logit for direct regression. Our proposed regression approach (BEL) can be combined with off-the-shelf task-specific feature extractors by increasing the regressor's 's output logits. Further, we find that the correlation between multiple binary classifiers' outputs can be exploited to reduce the size of the feature vector and consequently reduce the number of parameters in the regressor. We explore the use of different decoding functions for training loss formulation and evaluate binary cross-entropy, cross-entropy, and squared/absolute error loss functions for BEL. We evaluate BEL on four complex regression problems: head pose estimation, facial landmark detection, age estimation, and end-to-end autonomous driving. We make the following contributions in this work:
We propose binary-encodee labels for regression and introduce a general framework and taxonomy for the design aspects of regression by binary classification. We propose desirable properties of encoding and decoding functions suitable for regression problems.
We present a series of suitable encoding, decoding, and loss functions for regression with BEL. We present an end-to-end learning approach and regression layer architecture for BEL. We combine BEL with task-specific feature extractors for four tasks and evaluate multiple encoding, decoding, and loss functions. BEL outperforms direct regression for all the problems and specialized approaches for several tasks.
We theoretically and empirically demonstrate the effect of different design parameters on the accuracy, how it varies across different tasks, datasets, and network architectures, and provide preliminary insights and motivation for further study.
2 RELATED WORK
Binary classification for regression: Prior works have proposed binary classification-bact -baed approaches for ordinal regression (Crammer & Singer] 200T] Chu & Keerthi 2005 Li & Lin (006).
Ordinal regression is class of supervised Ieanningedccarnng problems, where the samples are labeled by a rank that belongs to an ordinal scale. Ordinal regression approaches can be applied to regression by discretizing the numeric range of the real-valued labels (Fu et al., 2018 Berg et al. 2021). In used for target labels E 21, .... N}. where classifier-k predicts if the label is greater than k or not for given input. Li & Lin (2066) provided reduction framework and generalization bound for the same. However, the proposed binary classification formulation is restricted. It requires several binary classifiers if the numeric range of output is extensive, whereas reducing the number of classifiers by using fewer quantization levels increases quantization error. Thus, more generalized approach for using binary classification for regression desirable to allow flexibility the design of classifiers. Binary classification for multiclass classification: Dietterich & Bakiri (1995) proposed the use of error-correcting output codes (ECOC) to convert a multiclass classification to a set of binary classification problems. This improves accuracy it introduces tolerance to binary classifiers' errors depending upon the hamming distance (i.e., number of bits changed between two binary strings) between two codes. [Allwei et al. (2001) provided a unifying framework and multiclass loss bounds in terms of binary classification loss. More recent works have also used Hadamard code, a widely used error-correcting code (Song al. 2021 Verma & Swami 2019. Other works have focused on
the existing works ordinal regression by binary classification, N binary classifiers
2

---

Published as a conference paper at ICLR 2022
My Tareet label
Quantisstion
Q. E ..,
Enoodding g
10, Targel code
'raining of binary- tlastifeess (C*, CC, ...., C....kk)
%
(5)
B, E (0, 13
4
Sk R.R. Predicted label
Docoding 2
Predicted code
M biuary claesiliers (C', (AM)
2, Predicted contidrecc
x2
Figure 1: The training (top) and inference (bottom) flow of binary-encoded labels (BEL) for regression networks. Red colored blacks represent design aspects we focus on.
the use and design of compact codes that exhibit a sublinear increase in the length of codes with the number of classes for extreme classification problems with a large number of classes (Cissé et al.
2012, Evron et al., 2018). However, the proposed encoding and decoding approaches do not consider the task objective and labels' ordinality for regression. Further, the binary classifiers possess distinct error probability distribution properties for regression problems as observed empirically (Section 3.1 which can be exploited to design codes suitable for regression.
Multiclass classification and ordinal regression by binary classification can be viewed as special cases falling under the BEL framework. As shown in Section other BEL designs yield improvements in accuracy over these approaches. Task-specific regression techniques are well explored as summarized D). While effective, task task-specific approaches lack generality by design.
below (see also Appendix
Head pose estimation: SSR-Net (Yang et al. 2018 and FSA-Net (fsa 2019) used a soft stagewise regression approach. HopeNet (Ruiz et al. [201 8) used a combination of classification and regression loss. Hsu et al.] (2019) used a combination of regression and ordinal regression loss.
Facial landmark defection: |Wang et al. (2020) minimize 1.2 loss between predicted and target 2D heatmaps with the latter formed using small variance Gaussians centered on ground truth landmarks. AWing (Wang et al./ (219)) modified loss for different pixels in the heatmap. LUVLi (Kumar et al.) 2020) proposed a landmark's ' location, uncertainty, and visibility likelihood-based loss. Bulat & Tzimiropoulos (0166) used binary heatmaps with pixel-wise binary cross- coos-ey loss.
Age estimation: OR-CNN (Niu et al. (2166) and CORAL-CNN (Cao et al. (020) used ordinal regression via binary classification. MV-Loss (Pan et al.] (0018) proposed to penalize the model output based on the age distribution's 's variance, while Gao et al. (2018) proposed to use the KI--ivergence
between the softmax output and a generated label distribution for training.
BINARY-ENCODED LABELS FOR REGRESSION (BEL)
We consider regression problems where the goal is to minimize the error between real-valuce target labels y; and predicted labels ú. over a set of training samples i. We transform this problem set of binary classification sub-problems by converting a real-valued label to a binary code.
Figure D shows the training and inference flow for BEL. The red-colored blocks highlight functions that vary under BEL. A real-valued label y. E RR is quantized to level Q2 E (1. N) ①. The quantized label converted to a binary vector B, E (0, 1]MM, that we call binary-encoded label,
using encoding function & ② There are (2%) possible encoding functions large number. The binary-encoded labels B, are used to train M classifiers 3 During inference the M classifiers predict a binary code B3 E {0, 1IM for input v; ( The predicted code (ß,) or the predictions' magnitude (2.), which indicates its confidence, is then decoded to predicted label S E R using a decoding function 5 We explore decoding functions that yield either quantized or continuous predicted outputs. The latter avoids quantization error by employing expected correlation (Section 3.3).
quantization, encoding, decoding, regressor network architecture, and training loss formulation. In this work we consider only uniform quantization while leaving nonuniform quantization (Fu et al.) 2018) to future work. Section 3.2) and 3.3 explore the characteristics of suitable encoding, decoding, and loss functions. Section 3.4 explores the impact of regressor network architecture. We find varying any of these aspects can improve accuracy. While BEL provides framework, and some design choices appear generally better than others, the most suitable BEL parameters to employ vary across task, dataset, and network architecture,
BEL contains five major design parameters resulting a large design space:
show both theoretically (Section 3.1) and empirically (Section 4)
3

---

Published as a conference paper at ICLR 2022
M bA
Q b
Q
Q
0000
0808
1080
80
11
56
6
66
60300008
60996008
0000000
0
81
0001
164
1106
880 61
18
91
08998699
69366891
88
11
60200000
60996611
1 0.0 8
8 1110
10
4
0111
1111
88
11
16
15
66996600
10386600
1111
82
16
32
68890011
69996609
2
1110
81
9
1118
14
B611
83 11
14
1.0
11
33
60996011
6938660]
T1
8601
63386611
1.00
10
1000
16
9686
8 8
06
1.0
96
35
23
60590011
60566.11
(a) (b) Unary (c) Johnson (d) B1JDIn
(c) BIJDJ
(l) B2IDJ
(g) HEXI
Figure 2: Examples of BEL codes. Part (a) represents the quantized values of the labels for Unary and Johnson
endes shown in Parts (b) and (c). Part (d) shows B1JDJ code without reflected binary; Parts (e) and (f) show BIJDJ and B2IDJ codes for targets in the range 1 to 16. Part (g) shows quantized and encoded values for HEXJ code (space added differentiate between base and displacement. or digits). Red lines represent bit transitions. These BEL codes described in Section 3.2
3.1 ANALYSIS OF ENCODING/DECODIE FUNCTIONS
This section analyzes the potential impact of encoding/decoding functions on regression error assuming empirically observed error distributions for the underlying classifiers. We compare Unary and Johnson codes (Figure 2b and 2c to determine when each is preferable. With this analytical study, we aim obtain insight into ordinal label classifier impact on regression error when employing simple encoding and decoding functions (E, D). Based upon this analysis we identify desirable properties for these functions. The design of the codes and intuition for trying them are discussed in Section 3.2] We divide our analysis into three parts: First, the expected error of predicted labels is derived in terms of classifiers' errors for two () functions. Next, we propose an approximate classifier's 's error probability distribution over the numeric range of target labels for regression based on empirical study. Last, we compare the expected error of sample {) functions based on our analysis. We use labels y E [1 N 1]. with quantization levels Q2 E {1,2, N 1}. Quantization error is not included as it is not affected by {E, D) functions.
Expected absolute error bounds terms of classification error: First, we analyze the unary code (BEL-U). The encoding function EBEL. converts Q; to B, = b]. b,V-2, bf.
where by 1 for k < Qi. else O. In this case, good choice of decoding function turns out to be simply counting the number of outputs across all N classifiers since error a single classifier changes the prediction by only one quantization level. Adding one since Q 1 is encoded by all zeros gives:
N-
DRELUU( (6).
2N-
8
+1
Let (z) be the error probability of classifier for target quantized label Q, = n. For a uniform distribution of y in the range [1,N- - 1] the expected error for BEL-U can be shown (see Appendix B to be bounded as follows:
BIG vl)
en())
N
similar analysis of expected error can be applied to binary encoded labels constructed yield
Johnson codes (BEL-J), in which Q, is encoded using B, = b. 82 N/2 where, b/: = 1 for N
Q < l: - < N - Q. else 0 (see Equation 27 in Appendix
2
Error probability of classifiers: To use Equation we need to determine (n) classifier's target output is 0 or 1. For BEI,, the target labels of a given classifier will have one or more hir transitions from 0 + 1 or 1 - as the target value of the regression network's output varies. For example, for the unary code (Figure 2b), the target output of classifier b² has bit transition from 0 to 1 going from Q; = to Q; = 3. The classifier should learn a decision boundary in (2, Each BEL classifier is tasked with learning decision boundaries for all bit transitions. As the difficulty of this task varies with the number of bit transitions it varies with different encoding functions. Moreover, the misclassification probability of classifier tends to increase the target label is closer to the
4

---

Published as a conference paper at ICLR 2022
Torget output of the clasiifeer
Target output of the classiffer
expected
0.6
0.8
SEL comagreed SEL]
60
Empliical
Empirical
4.5
Gauszian approximation
20.
Gaussian approximation
[.-3.3, s=2.4)
[n=3.6, d=2.2)
0.4
2.5
20
0.3
0.3
2.5
0
0.2
0.2
1.5
0.0
0
50
150 Target Label
100
200
250
150 Target Label
100
200
250
0.5g
20
30
40
30
50
-00
(a) Classifier A
(b) Classifier B
(c) Expected erre comparison
Figure 3: Part (a) and (b): classification error probability vs. target output for two classifiers. Target output 1 where blue and 0 elsewhere. Part (c): expected error increase of BEL-U versus BEL-J based on Equation 2 Equation 4 (blank means that combination of and results in error probability greater than one).
classifier's decision boundaries (Cardoso & Pinto da Costa 2007). Thus, we approximate ex for a classifier k with bit transitions as a linear combination of Gaussian distributions. Here, each Gaussian term centered around bit transition. Let (N(+.=)) (v) denote the probability density of normal distribution with mean µ and variance o², Each classifier for BEL-U encoding has one bit transition, whereas, each classifier for BEL-J encoding has two bit transitions (except the first and last classifiers). es (y) of classifier i: for BEL-U and BEL-J encoding is approximated as:
BEL-U (y) = =fN((..)))))) (v), where, j4 EK
0.5
66
BE) = - + [Nimze. (v), where, pikk
+ + M2k
1.5
(4)
Here, r is scaling factor. Figure 3a and 3b compares Equation 3 and against empirically observed error distributions for two classifiers using an HRNerV2-W18 (Wang et al. 2020) feature extractor (backbone) trained with the COFW facial landmark detection dataset (Burgos-Artizzu et al. 2013).
Comparison of expected absolute error for BEL-U and BEL-J: Based on the above analysis, compare the expected absolute errors of BEL-U and BEL-J. Figure represssent the percentage increase in absolute error for BEL-U compared to BEL-J for valid values of standard deviation o axia and scaling factor (xi) used in Equation and Here, BEL-J has lower error in the red-colored region (Sincrease>> 0), whereas BEL-U has a lower error in the blue-colored region (%inerease < 0). The figure shows that whether BEL-J or BEL-U has lower error depends upon the values of a and r. This suggests that the best {E, function will depend upon the classifier error probability distribution. The classifier error distribution in turn may depend upon the task, dataset, label distribution, network architecture, and optimization approach. Derivation of expected error for BEL-U and BEL-J and classifiers' empirical error probability distributions for different architectures, datasets, and encodings are provided in Appendix B to
3.2 DESIGN OF ENCODING FUNCTIONS
Based on the above analysis and further empirical observation we identify three principles for selecting BEL codes for regression as minimize error. First, individual classifiers should require fewer bit transitions as this makes them easier to train. Second, a desirable property for BEL encoding function is that the hamming distance between codes (number of bits that differ) should be proportional to the difference between the target values they encode. However, hamming distance weighs all bit changes equally. Thus, hamming distance based code design provides equal error protection capability to all bits (Wu. 2018 Xie et al., (002)) and does not account for which classifiers are more likely to mispredict for a given input. This matters because the misclassification probability of BEL classifiers is not uniform, but rather increases the closer the target value of an input is to bit transition (e.g.. Figure| and 3b}. These observations yield a third important consideration: For a given target value classifiers closer to bit transition are more likely to incur error.
The principles above highlight a tradeoff between classification error probability and error corre-ooreetion properties when selecting BEL codes. To evaluate the trade-offs, we empirically evaluate encodings that, to greater or lesser extent, satisfy one or more of the principles while focusing on reducing the number of classifiers (bits) to avoid increasing model parameters. Development of algorithms
5

---

Published as a conference paper at ICLR 2022
that might systematically optimize encoding functions is left to future work. Specifically, explore the following codes:
Unary code (U): Unary codes (Figure 2bb have only one bit transition per classifier and thus require
M = - 1 bits to encode values. Unary codes satisfy the first two principles and prior work on
ordinal regression by binary classification (Li & Lin 2006 Niu et al. 2016) uses similar codes.
Johnson code (J): The Johnson code sequence (Figure 2c is based on Libaw- Libaw-Craig code (Libaw & Craig 1953). We select this code as it has well-separated bit transitions and requires M -/4 bits compared to N required for unary code. This code exemplifies the impact of considering non-uniform classifier error probabilities (third principle). For example the hamming distance between 1 and 8 is just one. However, the bit transition for the differing bit, for classifier 6', is far from 1 or 8. Assuming equal error probability distributions centered on each bit transition for each classifier (as in Equation bb is less likely to mispredict than b2, b or ba for inputs with target values near 1 or 8.
Base+ daspplaceememmet based code (BIJDD//2JJJJ): We further reduce the number of bits using a baseetdiplaccement-be based representation. In this representation, a value is represented in base-k using a base-term b and displacement d via b * d. b and are represented using Johnson codes. Further, to improve the distance between two remote codes, we adapt reflected binary codes for term (Gray 1953). We evaluate base-2 (B1JDJ - Figure 2eb and base-4 codes (B2JDJ Figure 20
Binary coded hex Johnson code (HEXJ): In HEXJ (Figure2g) each digit (O-F) of the hexadecimal representation of a number is converted to an 8-bit binary code using Johnson code. For example, for the decimal number 47 (i.e., 2F in hex), (4X Concetanate((ohnsonn)) 16-bit HEXJ code can represent numbers in the range of 00 to FF (a total of 256). The number of bits increases sublinearly with the number of quantization levels for HEXJ, making it suitable for regression problems with many quantization levels.
Johnson(F)). A
widely used as error- correcting codes and have been used for multiclass classification (Dietterich & Bakiri 1995] Verma & Swami 2019). They require M N bits to encode N values. However, Hadamard codes violate all three BEL code selection principles: First, each classifier has many bit transitions. Second, as each code is equidistant (hamming distance of 2 ), the difference between target values is ignored. Finally, they protect all bits equally so do not take advantage of non-uniform error probabilities. We verify empirically Hadamard codes are unsuitable for regression (Section A..
Hadamard code (HAD): Hadamard codes (Bose & Shrikhande, 1959)
3.3 DESIGN OF DECODING FUNCTIONS
We explore three decoding functions: custom decoding, correlation-based decoding, and expected correlation-based decoding. Custom decoding functions are specific to the encoding function, and are only evaluated for unary and Johnson codes. In contrast, correlation-basse
decoding, first explored in prior work studying ECOC for multiclass classification (Allwein et al., 200T), can be applicd to all codes. For quantized labels in {1, 2, .... N), we define code matrix C of size N x M, where M is the number of bits/classifiers used for the binary-encoded label. Each row Ck in this matrix represents the binary code for label Q k. For example, Figure 2b can be considered code matrix, where the first row represents code for label Q, = Let 2, E R.M denote the output logit values of the classifiers. For decoding, the row with the highest correlation with the output 2, is selected as the decoded label. Here, real-valued output 2, is used instead of output binary code B, to find the correlation as it uses the confidence of classifier to make a more accurate prediction. For target quantized labels Q E {1, ..., N), the decoding function defined as:
DGFN (2))
arginax KE (1.2...... (2,- .C...)
However, DGEN outputs a quantized prediction, introducing quantization error. To remedy this concern and demonstrate the potential of more sophisticated decoding rules, we propose and evaluate an expected correlation- based decoding function, which allows prediction of real-valued label ÿ.. For target labels ys E [1, N1, the decoding function is defined as:
Ce.
DGNN-EX (2.) = Σ kog, where dx = N C2.
(6)
6

---

Published as a conference paper at ICLR 2022
Chiput
Decoding
Output
Feanu Extnetor
Extrettr
Ossput
Decoding fanction
+ Dutgut P
Laye vector
Regressor
Festure veddoe
(a) Direct regression
(b) Binary-encoded labels
Figure 4: Network architecture for direct and BEL regression; only the regressor architecture modified, but the entire network is trained end to end. P' is the number of dimensions of the regression network output.
Table 1: Benehmarks used for ovaluation
Feature
Label range!
Task
Extractor
Specialized Approach
Daraset
Renchmark
Quantization
ResNer50
Regression+clasifee
BIWI
HPE1
100-100/200
10
Landmark-liee head
Ruiz 2018)
30OLP/AFLW200
HPE2
100-1000000
pose estimation
RAFANet
Direct regresston
BIWT
TIPE3
180 1800360
50
Rehern
HPE4
-800
COFW
FLDI
0-256/256
10
Heatmap regression
Facial Landmark Delection
HRNeAV2-
Wang et al 2200 Xu etal.
300W
FLD2
0-256/256
W18
WTLW
FLD3
01236/266
10
AFLW
FLD4
0-256/256
30
Age estimation
ResNet134
ResNet50
Cao et al. 2020)
reuresston
MORPIE
AFAD
AF2
0-32232
10
End-to-end autonomous
driving
PilotNet
regression (Bojarski
et 2017
PilotNet
PN
0-670/670
10
A deep neural network with multiple output binary classifiers can be trained using the binary cross-entropy (BCE) loss Lucu (Z,, [(Q)). However, this loss minimizes the mismatch between predicted and target code but does not directly minimize the error between the target and predicted values. Decoding functions DOEN and DGEN-EX can be used to calculate the loss and minimize the mismatch between decoded predictions and target values directly. Decoding func- tion G finds the correlation between each row of the code matrix (Ck..) and the output Z. C.2 gives the correlation vector, and the index with the highest correlation is used as the predicted label. In this case, cross- cooss-yntropy loss LCR (C2,, Q:) can be used to train the network. Similarly, for decoding function DENN-EX which predicts a continuous value, L1 or L2 loss Luna (DGEN-EX (2,). ). Vi) can also be used for training. We evaluate multiple combinations of decoding and loss functions in Section
Training loss functtins:
3.4 REGRESSION NETWORK ARCHITECTURE FOR BEL
A regression network typically consists of feature extractor and regressor. The regressor consists of a fully connected layer between the feature extractor's s output (i.e., feature vector) and output logits for direct regression shown in Figure 4a In BEL, the number of output logits is increased to the number of classifiers (bits) used. When R P, with > 1, the required number of output significantly increase the
logits-P x M assuming M-bit BEL encoding per output dimension- size of the regression layer. However, empirically, we find small feature vectors suffice as the output logits are highly correlated for the explored encoding functions. Adding fully connected bottleneck layer to reduce feature vector size to 0 reduces the number of parameters and provides trade-off between the model size and accuracy. Figure/4b shows the modified network architecture for BEL.
EVALUATION
Table 1 summarizes the tasks, datasets, and network architectures used for the evaluation of BEL. These tasks are commonly used for evaluation of regression approaches by prior works due to the complexity of problem and network architectures (Diaz & Marathe 2019). Landmark-free 2D head pose estimation (HPE) aims to find a human head's pose in terms of three angles: yaw, pitch, and roll from 2D image without landmarks. Facial landmark detection (FLD) is problem of detecting the (x,, y) coordinates of keypoints in a given face image. Age estimation aims to predict the age of person from an image. In end-to-end autonomous driving, the steering wheel's next angle is predicted
7

---

Published as a conference paper at ICLR 2022
Error (MAEAMME)
HPE1
5.4
HPE2
15
HPE3
15
HPE4
3.7
FLD1
3.7
FLD2
3.6
3.6
5.2
6.0
10
3.5
3.5
5.0
3.4
4.0
3.4
4.8
D1
D2
D3
D4
D5
D2
D3
D4
0
D2
D3
D4
D5
D2
D3
D4
D2
D3
D4
D5
D1
D2
D3
D4
D5
Enro (MAE/NME)
4.6
FLD3
FLD4
2.8
AE1
3.5
#
PN
U
3.4
1.52
:
2.6
4.5
1.50
3.3
B2.D]
4.4
3.2
HEX]
11.4
3.1
D2
D3
D4
D5
D2
D3
D4
D5
D
D4
D5
D
D1
D2
D3
D4
D5
Figure 5: Error (MAE or NME) for different encoding, decoding, and loss functions for BEL. DI-D5 represents different combinations of decoding and loss functions: DI (BCE loss with SEL-U/BEL-J/GEN decoding for U/Jlothers), D2 (CE/GEN-EX), D3 (CE/GEN), D4 (L1 L2/GEN-EX) -X)), and D5 (BCE/GEN-
from an image of the road. Normalized Mean Error (NME) and Mean Absolute Error (MAE) with respect to raw real-valus valued labels are used as the evaluation metric for FLD and the rest, respectively.
We also evaluate direct regression and multiclass classification baseline regression approaches. For direct regression, LI or L2 loss functions are used. Label values are scaled to reduce the range of labels. The loss function and the scaling factors classification-be based regression, the target values are quantized and converted to a class. The network trained using cross-entropy loss in this case. In evaluation, the entire network (i.e., feature extractor and regressor) are trained end-to-end for direct regression, multiclass classification, and BEL. The feature extractor, data augmentation, evaluation protocols, and the number of training iterations are kept uniform across different methods for each benchmark. We report average of five training runs and eтrоr margin of 95% confidence interval. Details datasets, training parameters, related work for specific tasks, and other evaluation metrics are provided in Appendix
using hyperparameter tuning. In the multiclass
BEL introduces several design parameters for regression by binary classification. We evaluate differ- ent encoding (E), decoding (D), and training loss (C) functions for BEL across all the benchmarks and study the extent and nature of the impact of these design parameters on accuracy.
Encoding function (E): Figure 5 plots error (MAE or NME) using different encodings. We do not show results for Hadamard codes here as it results in significantly higher error than other encodings (Appendix(A) (A). On average, Hadamard codes result in 60% higher eтror than J encoding, which shows that these codes are unsuitable for regression. The results show the encoding function significantly affects the accuracy and the best-performing encoding function varies across tasks, datasets, and network architectures (e.g.. HPE1 and HPE3 are trained on the same dataset and different architecture). In Section 3.1 we observed that which encoding/decoding functions result lower error depends upon the classifiers' error distribution. For decoding functions used for comparison in Section 3.I, J does better than U for HPE3, FLD1, and AE1; attribute this to misclassification errors occurring more frequently near bit transitions based on the analytical study.
The encoding function impacts the number of classifiers and the complexity of the function to be learned by classifier. We observe a trade-off between these two parameters. For some benchmarks, the availability of sufficient training data and network capacity facilitates the learning of complex classifiers such as B2JDJ. In such case, a reduced number of classifiers compared to U, J, or BIJDJ codes results in lower error. We provide empirical results for the same in Appendix A
Decoding (D) and training loss (L) functions: We explore three decoding and three training loss functions (Section 3.3). However, not all the combinations of decoding and loss functions (D/L) perform well. For example, CE, LI, or L2 losses do not use decodings BELII or BELLU Therefore, optimizing the network for these losses does not directly minimize the absolute emror between targets
8

---

Published as a conference paper at ICLR 2022
Table Comparison of BEL with different regression approaches. "Specialized approach" described in Table
Error (MAE or NME) Model size HPE2
Appraach
HPFI
HPE3
HPF4
Specialized approach
3.40/
4.14/69.88
Direct regression
4.76 53.55
5.65 10.13/23.5M
3.40
10026 69.8M
4.14 1.011/ 8M
Multiclass classiication
4.49 10.24/24:2M
5.31 10005 / 24.8M
4,54 0.04 / 72.0M
5.14 1:000 72.0M
BEL
3.56 0.01 23.6M
4,77 10.0 /3.6M
1.00
89.8M
3,90 10.03769.8M
BEL 2/D/2 functions
U/GN-EEXLL2
U/GEN-ERE BOEN-EXXBCE
BUDJIGEN EUDIIEEN-EXIICE
URGEN-EXICCE
Appraahh
FL.DI
FL.D2
FLD3
FL.4
Specialized approach
9.6M
3.32/ 9.6M
4.32/ 9.6M
1.57/ 9.6M
Direct regression
3.60 1:002// 10.2M
3.54 1:003 / 10.2M
4.64 +0003/ 10.2M
1.51
1:01 10.2M
Multiclass classiication
3.58 10.03 / 25.4M
3.51 1:002/ / 45.2M
4,50
0.01 / 661.5
1.56
10.01
/0.IM
20.1M
BEL
3.34 10.02/ 10
3 10.0 10.02/1..22 /1.2M
4.36
0.02/ 11.M
1.47
10.00/
1088
BEL 2/D/2 functions
HEXI/GEN-EXCC
UIGEN-EXCCE
TUDUGGENEEXCE
BUDDIGEN-EXCEE
Appraach
AEI
AE2
PN
Specialized approach
2.49 21
3.47/ 21.3M
4.24/ 1.8M
Direct regression
2.44 1:001 /
3,21 +0002 / 23.1
4.24 +0.45/ L8M
Multiclass classiication
2.75 10.03 / 23.1M
3.38 10.05 / 23.1M
5,54 0.00 1.9
BEL
2.27 1.0.01 /23.IM
3.11 1..00/23000 23.1M
3.11 20.01/IN
BEL 2/D/2 functions
J/BEL-1BBB
BUDII
JIGENCC
and decoded predictions. We present results for five out of nine DIC combinations. comprees error (MAE or NME) achieved by different DIL combinations and highlights the range of error variations. DGEN-EX results in the lowest error for the majority of the benchmarks as reduces quantization error and also utilizes the output logit confidence values. DGEN EX consistently perform better than DENN function that has been used for multiclass classification by prior works (Allwein al., 2001). The use of CE L1/L2 loss results in lower error with DGNN-EX for most benchmarks as the training loss function directly minimizes the error between targets and decoded predictions.
Comparison of BEL with regression approaches: Table 2 compares BEL with other approaches for different benchmarks (Table We explore and evaluate multiple combinations of encoding (E), decoding (D), and loss (C) functions for BEL in this work. In these experiments 20% of the training set is used as validation set and the validation error is used to choose the best BEL, approach. An ablation study for using more fully connected layers for direct regression and multiclass classification Appendix A BEL results lower error than direct regression and multiclass classification and even outperforms task-specific regression approaches for several benchmarks.
The results show no single combination of encoding/decoding//u functions evaluated was best for all benchmarks but also demonstrate BEL improves accuracy across range of regression problems.
5 CONCLUSION
This work proposes binary-encoded labels (BEL) to pose regression as binary classification. We propose taxonomy identifying the key design aspects for regression by binary classification and demonstrate the impact of classification error and encoding/decoding functions on the expected label error. Different encoding, decoding, and loss functions are explored to evaluate our approach using four complex regression tasks. BEL results in an average 9.9%, 15.5%, and 7.2% lower error than direct regression, multiclass classification, and task-specific regression approaches, respectively. BEL improves accuracy over state-of-the-ar approaches for head pose estimation (BIWI, AFLW20000, facial landmark detection (COFW), age estimation (AFAD), and end-to-end autonomous driving (PilotNet). Our analysis and empirical evaluation in this work demonstrate the potential of the vast design space of BEL for regression problems and the importance of finding suitable design parameters for given task. The best performing encoding/decoddng function pair may be task, dataset, and network specific. possibility this suggests, which we leave to future work, is that may be beneficial to develop automated approaches for optimizing these functions.

---

Published as a conference paper at ICLR 2022
6 ACKNOWLEDGEMENTS
This research has been funded in part by the National Sciences and Engineering Research Council of Canada (NSERC) Strategic Project Grant. Tor M. Aamodt serves as a consultant for Huawei Technologies Canada Co. Ltd. and Intel Corp. Deval Shah is partly funded by the Four Year Doctoral Fellowship (4YF) provided by the University of British Columbia.
Reproducibility: We have provided detailed discussion about training hyperparameters. experi- mental setup, and modifications made in publicly available network architectures in Appendix [D.] D.4 for all benchmarks. Code is available at atttps:: //qithull thub. com/ www.uuucaaaa -aaoddt-rroup//aoo BBLL We have provided the training and inference code with trained models.
regression on
Code of Ethics Some of the major applications of regression problems are artificial intelligence and autonomous machines, and regression improvement can accelerate the development of autonomous systems. However, depending upon the use, autonomous systems can have some negative societal impacts, such as job loss in some sectors and ethical concerns.
REFERENCES
Fsa-net:
Learning fine-grained structure aggregation for head pose estimation from single im- age. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 10-Juue::087-10900 2019.
Erin L.. Allwein, Robert Schapire, and Yoram Singer. Reducing multiclass to binary: unifying approach for margin classifiers. J. Mach. Learn. Res., 1:113-141, September 2001. doi: 10. 10.1162/
15324430152733133.
Ardhendu Behera, Zachary Wharton, Pradeep Hewage, and Swagat Kumar. Rotation axis focused attention network (rafa-net) for estimating head pose. In Computer Vision ACCV 2020, 2021.
Axel Berg, Magnus Oskarsson, and Mark O'Connor. Deep ordinal regression with label diversity. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 2740-2747. IEEE, 2021.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to End Learning for Self-Driving Cars. arXiv: 07.16, 2016.
Mariusz Bojarski, Philip Yeres, Anna Choromanaska, Krzysztof Choromanski, Bernhard Fimner, Lawrence Jackel, and Urs Muller. Explaining how deep neural network trained with end-to-end learning steers a car. arXiv: Toxkv:/700..000 170.00991, 2017.
R.C. Bose and S.S. Shrikhande. A note on result in the theory of code construction. Information and Control, 2(2):183-194, 1959. doi: https://doi.org//
http///oooor///11.0 (rg//..1uu (16500000099998899997777777
Adrian Bulat and Georgios Tzimiropoulos. Human Pose Estimation via Convolutional Part Heatmap Regression. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 717-732, 2016.
Xavier Burgos-Artizzu, Pietro Perona, and Piotr Dollár. Robust Face Landmark Estimation under Occlusion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1513- 1520, 12 2013. doi: 10. 10.1109/CCC22213..1.99
Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. Rank consistent ordinal regression for neural networks with application age estimation. Pattern Recognition Letters, 400:255331,, 2020. 2020. doi:
http:///doo.r1110.11001 http:://rrrg..100oooarrre0000000 p0trec.2020.111
Jaime S. Cardoso and Joaquim F. Pinto da Costa. Learning to Classify Ordinal Data: The Data Replication Method. Journal of Machine Learning Research, 8:1393-1429, 2007.
Sully Chen. Dttiniaeae..tttoe////g/////////n
10

---

Published as a conference paper at ICLR 2022
Wei Chu and S. Sathiya Keerthi. New approaches support vector ordinal regression. In Proceedings of the 22nd International Conference on Machine Learning, ICML '05, pp. 145-152, 2005.
M. Cissé, T. Artières, and Patrick Gallinari. Learning Compact Class Codes for Fast Inference in Large Multi Class Classification. In Peter A. Flach, Tijl De Bie, and Nello Cristianini (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 506-520. Springer Berlin Heidelberg, 2012.
Koby Crammer and Yoram Singer. Pranking with Ranking, In Proceedings of the 14th International Conference on Neural Information Processing Systems: Natural and Synthetic, NIPS'O1 01, pp. 641-647.20 MIT Press, 2001.
T. G. Dietterich and G. Bakiri. Solving Multiclass Learning Problems via Error-Correcting Output Codes. Journal of Artificial Intelligence Research, 2:263-286. 1995. doi: 10.1613/jair, 105.
Xuanyi Dong, Yan Yan, Wanli Ouyang, and Yi Yang. Style Aggregated Network for Facial Landmark Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 379-388, 2018
Raúl Díaz and Amit Marathe. Soft labels for ordinal regression. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi: 10.1 10.1100/22..2 9,04977.
Itay Evron, Edward Moroshko, and Koby Crammer. Efficient for Extreme Classification. Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 7233-7244, 2018.
Decoding Graphs
Gabriele Fanelli, Matthias Dantone, Juergen Gall, Andrea Fossati, and Luc Gool. Random forests for real time 3d face analysis. International Journal of Computer Vision, 101(3):437-458 February 2013.
Huan Fu, Mingming Gong, Chaohui Wang. Kayhan Batmanghelich, and Dacheng Tao. Deep Ordinal Regression Network for Monocular Depth Estimation. Proceedings the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2002-2011, 2018.
Bin-Bin Gao, Hong- Yu Zhou, Jianxin Wu, and Xin Geng. Age estimation using expectation of label distribution learning. In Proceedings of the Twenty-Sevnnth International Joint Conference on Artificial Intelligence, IJCAI-18, pp. 712-718, 2018. doi: 10.24963/icca.20118999
Frank Gray. Pulse code communication, 1953.
Heng- Wei Hsu, Tung Yu Wu, Sheng Wan, Wing Hung Wong, and Chen- Yi Lee. Quatnet: Quaternion- based head pose estimation with multiregression loss, IEEE Transactions on Multimedia, 21(4): 1045-1046, 2019. doi: 10.1 1009000...000.2000020
Marek Kowalski, Jacek Naruniec, and Tomasz Trzcinski. Deep alignment network: A convolutional neural network for robust face alignment. In IEEE Conference Computer Vision and Pattern Recognition Workshops, pp.
2017. doi: 10. 10.1099922..2017225444
Abhinav Kumar, Tim K. Marks, Wenxuan Mou, Ye Wang. Michael Jones, Anoop Cherian, Toshiaki Koike-Akino, Xiaoming Liu, and Chen Feng. LUVLi face alignment: Estimating Landmarks' loca- tion, uncertainty, and visibility likelihood. Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2020. doi: 10. 00.11099CCRR4442000000000000000
M. Köstinger, P. Wohlhart, P. M. Roth, and H. Bischof. Annotated Facial Landmarks the Wild: A large-scale, real-world database for facial landmark localization. In 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 2144-2151, 2011.
Ling and Hsuan-Tien Lin. Ordinal regression by extended binary classification. In Proceedings of the 19th International Conference Neural Information Processing Systems, pp. 865-872, 2006.
William Libaw and Leonard Craig. A photoelectric decimal-coder shaft digitizer. Electronic Computers, Transactions of the I.R.E. Professional Group on, EC-2:1 4, 10 1953.
11

---

Published as a conference paper at ICLR 2022
J. Lv. X. Shao, J. Xing. C. Cheng, and X. Zhou. A deep regression architecture with two-stage re-initialization for high performance facial landmark detection. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3691-3700, 2017. doi: 10.1109/CVPR.
2017.393.
X. Miao, X. Zhen, X. Liu, C. Deng, V. Athitsos, and H. Huang. Direct shape regression networks for end-to-end face alignment. In 2018 IEFF/CVF Conference on Computer Vision and Puttern Recognition, pp. 5040-5049 2018.
S. S. Mukherjee and N. M. Robertson. Deep head pose: Caazedireection estimation in multimodal video. IEEE Transactions Multimedia, 17(1 (7(11):209) (:0:)::22422120 2107, 2015.
Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. Ordinal regression with multiple output CNN for age estimation. Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2016.
Hongyu Pan, Hu Han, Shiguang Shan, and Xilin Chen. Mcan-variance loss for deep age estimation from a face. In 2018 JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5285-5299 2018. doi: 10.1 1099///..2008..0065..
Sebastian Raschka. MLxtend: Providing machine learning and data science utilitics and extensions to Python's scientific computing stack. Journal of Open Source Software, 3(24):638, April 2018. doi: 10.21 105/joss.006388
K. Ricanek and T. Tesafaye. Morph: longitudinal image database of normal adult age-progression. In 7th International Conference on Automatic Face and Gesture Recognition (FGR06), pp. 341-345, 2006. doi: 10.1 1,.1109//RR.006.777
Nataniel Ruiz, Eunji Chong, and James M. Rehg. Fine-grained head pose estimation without key- points. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic. 300 faces in-the-wile challenge: The first facial landmark localization challenge. 2013 IEEE International Conference on Computer Vision Workshops, pp. 397-403, 2013.
Shizhan Zhu, Cheng Li, C. C. Loy, and X. Tang. Face alignment by coarse-to-fine shape searching. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 4998 50065 2015.
Yang Song, Qiyu Kang, and Wee Peng Tay. Error-Correcting Output Codes with Ensemble Diversity for Robust Learning in Neural Networks. AAAI, 2021.
Y. Sun, X. Wang. and X. Tang. Deep convolutional network cascade for facial point detection. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3476-3483, 2013. doi: 10.1 10.1109/CVPR.2013.44
G. Tzimiropoulos. Project-out cascaded regression with an application to face alignment. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp- 3659- 3667, 2015.
Gunjan Verma and Ananthram Swami. Error correcting output codes improve probability estimation and adversarial robustness of deep neural networks. Advances in Neural Information Processing Systems, 32NuuulPs 2019.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recognition. IEEE transactions pattern analysis and machine intelligence, PP, April 2020.
Xinyao Wang, Liefeng Bo. and Fuxin Li. Adaptive wing loss for robust face alignment via heatmap regression. In 2019 JEEE International Conference on Computer Vision (ICCV), pp. 6970-6980, 2019. doi: 10.1 109/ICCV.001..00707
12

---

Published as a conference paper at ICLR 2022
Chai Wah Wu. Designing communication systems via iterative improvement: correction coding with bayes decoder and codebook optimized for source symbol error. ArXiv:1805.07423. 2018.
W. Wu and S. Yang. Leveraging intra and inter-dataset variations for robust face alignment. In 2017 IEEE Conference Computer Vision and Puttern Recognition Workshops (CVPRW), pp. 2096-2105, 2017.
W. Wu, Chen Qian, S. Yang, Q. Wang, Y. Cai, and Qiang Zhou. Look at boundary: A boundary- aware face alignment algorithm. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2129-2138, 2018.
Ley Xie, Huifang Chen, Peiliang Qiu, and Ming Zhang. The modified hamming bound for unequal error protection codes. In IEEE 2002 International Conference on Communications, Circuits and Systems and West Sino Expositions, 2002. doi: 10.1 100//CCCCAS
X. Xiong and F. De la Torre. Supervised descent method and its applications face alignment. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 532-539, 2013.
Zixuan Xu, Banghuai Li, Miao Geng, Ye Yuan, and Gang Yu. Anchorface: An anchor-based facial landmark detector across large poses. ArXiv:2007.0322 2020.
Tsun-Y Yi Yang, Yi-Hsuan Huang, Yen-V Yu Lin, Pi-Cheng Hsiu, and Yung- Yu Chuang. Ssr-net: A compact soft stagewise regression network for age estimation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCA' pp. 1078-1084, AAAI Press, 2018. ISBN 9780999241127.
Jie Zhang, Shiguang Shan, Meina Kan, and Xilin Chen. Coarse-to-fine auto auto-encoder networks (cfan) for real-time face alignment. In Computer Vision - ECCV 2014, pp. 1-16, 2014.
S. Zhu, C. Li, C. C. Loy, and X. Tang. Unconstrained face alignment via cascaded compositional learning. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2016. doi: 10.1109/CVPR.2016.377..
Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and Stan Li. Face alignment across large poses: A 3d solution. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 146-155, 06 2016
13

---

Published as a conference paper at ICLR 2022
A ABLATION STUDY
Impact of combination of encoding, decoding, and loss functions: We propose multiple combinations of encoding, decoding, and loss functions that can be used with BEL. In Tables 3 13 we show the effect of each combination of encoding, decoding, and loss function on the error of the model. Although general trends exist and some combinations perform consistently well across datasets, the optimal combination varies based on the dataset.
Table 3: Comparison of BEL design parameters on MAE for head pose estimation with BIWI dataset and ResNet50 feature extractor (HIPI)).
Encoding function
Decoding lunction
Loss function
BIJDI
B2ID
IEXI
HAD
BEI JIBBIIU
BCF
3,38
3.65
GEN EX
BCB
3.37
3.64
5.11
8.02
4.76
7.53
GEN
BCB
3.63
3.16
8.16
4.99
7.73
GEN-EX
3.88
4.08
5.50
GEN
CE
4.25
3.62
3.93
4.06
4.39
5.48
GEN
1.2
3.56
3.93
3.66
3.59
5.99
4.21
Table 4: Comparison of BEL design parameters MAE for head pose estimation with 300LP/AFLW2000 datasets and ResNe150 feature extractor (HPE2).
Encoding function
Decoding function
Loss function
B211
HEXI
HAD
BEL-J/BLLUU
BCE
4.78
4.84
GEN-EX EX
BCR
4.77
4.84
5.43
5.09
4.94
7.84
GEN
BCB
4.78
4.87
5.11
5.05
5.15
8.54
GEN
CE
4.93
5.04
5.04
4.97
4.79
5.64
5.13
GEN-EX
L2
5.08
5.18
5.19
5.09
5.17
5,07
Table 5: Comparison of BEL design parameters on head pose estimation with BIWI dataset and RAFA-Net feature extractor (HPE3).
Encoding function
Decoding function
Loss function
BIJDJ
B2JDJ
HEXU
HAD
BEL-I/BEL-U
BCE
3.47
3.16
GEN-EX
BCE
3.40
3.12
3.30
3.35
3,80
5.75
GEN
RCE
3.49
3.14
3.62
3.78
4.44
5.83
GEN-EX
CE
3.82
3.91
3.52
3.49
3.98
3.88
GEN
CE
3.92
4.09
3.62
3.65
4.35
4.28
GEN-EX
3.60
4.29
14

---

Published as a conference paper at ICLR 2022
Table 6: Comparison of BEL design parameters on MAE for head pose cstimation with 300LP/AFLW datasets and RAFA-Net feature extractor (IIPE4).
Encoding function
Decoding function
Loss Function
U
BTDD
B5DD
TIEX]
IAD
REI -IBELLU
RCE
3.94
4,00
GEN-EX
BCE
3.90
3.93
4.19
4.12
4.39
9.17
GEN
BCE
3.93
3.94
4.21
4.25
4,53
9.21
GEN-EX
4.62
4.34
5.12
GEN
CE
4.68
4.75
4.46
4.61
4.63
5.29
GEN-EX
4.45
5.87
5.11
9.34
10.43
17.89
Table Comparison of BEL design parameters NME for facial landmark detection with COFW dataset and HRNetV2-W 2- feature extractor (FLDI).
Encoding function BIJDJ
Decoding
function
B2IDJ
HAD
BEL-V/BEL-U
BCE
3.45
GEN-EX
BCR
3.45
3.43
3.42
3.41
3.47
4.28
GEN
BCB
3.46
3.45
3,43
3.47
3.66
4,43
GEN
CF
3.37
3.37
3.38
3.41
3.34
3.69
GEN
3.44
3.49
3.69
GEN-EX
LI
3.44
3.41
3.45
3.47
3.41
4.52
Table Comparison BEL design parameters on NME for facial landmark detection with 300W dataset and HRNetV2-W 18 feature extractor (FLD2).
Encoding function
Deunding function
Loss function
BTJDS
H211
HEXI
HAD
BEI-V/BLLU
BCB
3.5
3.49
GEN EX
BCB
3.48
3.46
3.43
3.42
3.38
4.71
GEN
BCB
3.45
3,45
4.78
GEN-EX
CE
3.40
3.36
3.37
3.41
3.37
3,62
GEN
CE
3.50
3:45
3:45
3.31
3.39
3.65
GEN-EX
3.67
Table 9: Comparison of BEL design parameters on NME for facial landmark detection with WFLW dataset and IRRNetV2-W 2-- feature extractor (FLD3).
Encoding function
Decoding function
Loss function
U
BIDD
B2IDJ
HEXI
HAD
BEL-J/BL-U
BCB
4.54
GEN-EX
BCB
4.6
4.51
4.43
4.38
4.37
7.18
GEN
BCB
4.62
4.33
4.44
4.42
4.3
7.14
GEN-EX
5.15
GEN
CE
4,46
4.44
4.47
4.47
4.56
4.83
GEN-EX
LI
4.39
4.42
4.47
4.47
4.45
4.74
Table 10: Comparison of BEL design parameters NME for facial landmark detection with AFLW dataset and HRNetV2-W feature extractor (FLD4).
Encoding function BTJDJ
Decoding
function
B2IDI
BEL.MBELLU
BCB
1.51
1.52
1.47
GEN
BCB
1.51
1.52
1.50
1.49
1.54
1.55
GEN-EX
CE
1.48
1.47
1.47
1.47
1.47
1.47
1.51
GEN-EX
1.47
1.47
1.48
1.48
1.48
1.59
15

---

Published as a conference paper at ICLR 2022
Table 11: Comparison of BEL design parameters on MAE for age estimation with MORPH-II dataset and ResNet50 feature extractor (AEI).
Encoding function
Decoding function
Loss Tunction
BIJDJ
BJDDJ
HEXI
HAD
BEL-I/BEL-U
BCE
2.27
GEN-EX
BCE
2.30
2.29
2.35
2:49
2.45
2.99
GEN
BCE
2.28
2.28
2.34
2,51
2.54
3.07
2.75
GEN
CE
2.60
2.58
2.61
2.66
2.61
3.10
GEN-EX
L1
2.30
2.30
2.32
2.30
2.32
2.29
Table Comparison of design parameters on MAE for age estimation with AFAD dataset and ResNet50 feature extractor (AE2).
Encoding function
Deanding function
Loss function
BTJDS
H211
HEXI
HAD
BEI-V/BLLU
BCB
3.13
3.15
GEN
BCB
3.14
3.16
3.32
3.35
3,28
3.34
GEN
BCB
3.41
3.44
3.52
GEN-EX
CE
3.26
3.29
3.38
3.44
3,40
3.30
GEN
CE
3.36
3.34
3.42
3.47
3:40
3.45
GEN-EX
3.11
3.12
3.13
Table 13: Comparison of BEL design parameters MAE for end-to-end autonomous driving with PilotNet dataset and feature extractor (PN).
Encoding function
Decoding function
function
BIDDJ
B2JDJ
HEXI
HAD
BEL-I/BEL-U
BCE
4.34
3.91
GEN-EX
BCE
4.57
4.20
4.83
4.96
3.29
10.12
GEN
BCE
4.37
3.95
3.51
3.61
4.01
10.00
GEN-EX
CE
4.30
4.16
499
3.87
5.39
87.17
GEN-EX
4.10
4.11
4.34
4.34
4.11
5.09
16

---

Published as a conference paper at ICLR 2022
Impact of quantization and decoding functions: As discussed in Section real-valued label is quantized to a discrete value in [1.,...,,N) before applying the encoding function. In Table 14 show the effect of increasing the number of quantization levels (N) on the error for correlation-based decoding (DGEN, which returns a quantized prediction) and expected correlation- based decoding (DGEN-EX, which returns continuous prediction). As shown in the table, there exists a tradcoff between reducing quantization error and using fewer classifiers. The error is lower for 128 quantization levels than it is for 256 the improvement resulting from fewer binary classifiers is higher than the increase in quantization error. Moreover, the use of proposed decoding function DGNNEN for regression consistently results lower error compared DGFN
Table 14; Impact of the quantization and decoding functions on NME for facial landmark detection.
COFW
300W
Quanlization levels
128
256
64
128
256
GBEL-1
DOEN
3,66
3.51
3,46
3.79
3.59
3.46
g DUEVER
3,46
3,41
3,44
3.54
3,47
3.44
[ DDGN
3.65
3.49
3.43
3,76
3,58
3.46
G DIGNEX
3,45
3.40
3,42
3.52
3.45
3.43
Impact of the number of training samples on BEL: of different encoding functions varies depending on the availability of sufficient training data. In Table [5, we analyze the effect of the number of available training samples for both simple and measure of the complexity of classifier.
As discussed in Section 4 the performance
complex encodings. We the number of bit transitions As the number of training samples decreases, simpler encodings (U and J) perform better than more complex encodings (BIJDJ, B2JDJ, and HEXJ). Using more complex encoding reduces the number of classifiers; however, it increases each classifier's complexity (i.e. the number of bit transitions) and thus performs poorly with less training data.
Table 15:
Effect of training dataset size on optimal encoding function for facial landmark detection. BCE loss function and GEN-EX decoding function are used for the training and evaluation.
Recluction number samples
Encoding
#Classifers/lbel
#hit
0%
20%
40%
60%
80%
90%
95%
transitionséllassffeer
COPW (FLDI)
U
256
3.45
3.48
3.55
3.72
334
4.52
6.29
128
3,43
3.48
3.51
3.61
3,88
4.32
5.39
BUDI
3.42
3.44
3.52
3.60
4.11
4.50
5.68
B2IDJ
3,45
3.48
3.80
3.94
HEXI
17
32
3:47
3.69
3.78
4.03
4.61
5.48
6.69
U
256
3,48
3.35
3.38
3.64
3.89
4.26
5.66
128
3.46
3.56
3.52
3.58
3.79
4.04
4.58
BIJDI
3.43
3.48
3.53
3.61
3.89
6.10
B2IDI
34
3.42
3.47
3.51
3.54
3.88
4.50
5.80
HEXI
17
32
3,38
3.64
3.73
3.97
4.41
5.38
6.60
WFLW (FLD3)
U
256
4.60
4.67
4.83
3.00
3.37
6.04
7.46
4.60
4.65
4.84
5.23
5.64
6.39
BIJDI
65
4.43
4.44
4.52
4.66
5.08
5.90
8.39
B2JDJ
34
4.38
4.46
4.49
4,61
5.02
5.95
8.78
HEXI
32
4.37
4.50
4.72
4.96
5.72
6.86
8.09
AFLW (FL.4)
U
256
1.50
1.53
1.53
1.56
1.61
1.68
1.83
128
1.30
1.11
1.52
1.54
1.60
1.68
1.79
BUDI
65
1.47
1.50
1.52
1.54
1.60
1.67
1.78
B2IDI
34
8
1.47
1.50
1.50
1.52
1.57
1.64
1.73
1.49
1.54
1.54
1.55
1.59
Impact of reflected binary conversion: As mentioned in Section 3.2
we reflected binary to increase the distance between distant labels based on the design properties of suitable regression encodings we proposed. Table 16 shows the impact of using reflected binary conversion error for
17

---

Published as a conference paper at ICLR 2022
facial landmark detection benchmarks. As shown in the table, the use of reflected binary significantly reduces the error.
Table 16: Effect of reflected binary conversion for BJJDJ encoding on facial landmark detection, Here, BCE
loss and GEN-EX decoding functions are used.
COFW
300W
WFLW
AFLW
BIJDJ BIJDJ- binary
3.43
3,46
4.43
Use of binary heamaps: Facial landmark detection approaches typically use heatmap regression. We also evaluate BEL-H-x, in which the real-valued heatmaps are converted to binary heatmaps with 8 quantization levels. Table [ the impact of using binary heatmaps error for facial landmark detection benchmarks. For unary code, a 64 x 64 real-valued heatmap of one facial landmark is converted eight 64 64 binary heatmaps, resulting in 32, 768 (8 64 64) binary classifiers compared to 512 for BEL-U. We believe that training high number of binary classifiers results in high error for BEL-H-x.
Table 17: Comparison of BEL with heatmaps for facial landmark detection. Here, BCE loss and GEN-ER EXX-XX decoding functions are used.
FLD1
FLD2
FLD3
FLD4
BEL-1
3.45
3.46
4,60
1.50
BEL-J
3.48
3.46
4.51
1.50
I.U
4.43
5,70
BEL-H-
10.17
33.02
22.50
2.99
Hyperparameter 0: As shown in Figure 4b we introduce a feature vector of size & before the output layer. Figure @ compares the decrease in the error for different encodings and values. We observe that more complex encodings benefit more from an increase in the value of 0, while lower value of 0 can be used for simpler encodings.
Absolute improvement in NME
0.125
0.100
B1DD
0.075
0.050
0.025
0.000
50
Length of the feature vector
Figure Effect 0 on error for different encodings on FLDI.
Impact of increasing the number of fully connected layers: For BEL, we propose to add a fully connected bottleneck layer the regressor reduce the feature vector size to 0 and thus decrease the number of parameters in the regressor. We perform an ablation study to study the impact of this added fully connected layer on relative performance of direct regression, multiclass classification, and binary encoded labels. Table 18 provides the error (MAE or NME) for direct regression and multiclass classification with one or two fully connected layers after the feature extractor. Further, we evaluate BEL, direct regression, and multiclass classification for higher number of fully connected layers as shown in Table 19 We observe that increasing the number fully connected layers in direct regression and multiclass classification does not improve the accuracy for most benchmarks (possibly due to overparameterizztion)) BEL with two fully connected layers outperforms direct regression and multiclass classification in both cases. Furthermore, for higher number of fully connected
18

---

Published as a conference paper at ICLR 2022
Table 18: Impact increasing number of fully connected layers in direct regression and multiclass classification on the error (MAE or NME).
Direct regression
Multiclass classification layer
BEL
Benehmark
layer
layers
layers
FC layers
HPE1
4.76
4.49
4.82
HPEZ
5.65
HPES
3,40
3.54
4,45
4.54
3.12
HPE4
4.14
4.22
5.14
5.45
3,90
FLD1
3.56
FLD2
3.54
3.58
3.51
3.62
3.36
FLD3
4.64
4.63
4.50
4.64
4.33
FL.44
1.56
AEI
2.44
2.35
2.75
2,81
2.27
AF2
3.21
3.14
3,38
3.40
3.11
PN
4.24
4.33
4.56
5.74
3.11
layers in BEL, the suitability of encoding function varies with the dataset, demonstrating the importance of BEL design space.
Table 19: Impact increasing number of fully connected layers in direct regression, multiclass classification, and BEL. GEN-EX decoding function and BCE loss function are used BEL..
FC layers (size of FC layers)
Direct gression
Multiclass classification
Benchmark
BIDD
B2JDJ
HEXI
(1024-x)
3.6
3.58
FLDI
(1024-30-x)
3,56
3,42
3.41
3.47
3(0024 30224-30110 10-1
3.63
3.94
3.55
3.47
3.82
4.02
3.62
(1024-x)
3.54
3.31
FLD2
(1024-100x)
3.58
3.62
3.48
3.46
3,43
3.42
3.38
3(1024 10xx)
3.35
3.78
3.42
3.46
3.61
3.52
Training-var valliiinivaliattiinnnnn set based evaluation: Ideally, validation set should be used for model selection. Hence we have reevaluated the benchmarks with a validation set to select the best design parameters and the best model (i.e., which model is the best over multiple epochs). Since datasets used in benchmarks do not provide separate validation datasets, we use 20% of the training data as validation set. Since earlier works use 100% training data for the reported results and use test error for model selection, we have re-run specialized approaches (if possible), direct regression, and multiclass classification. was not possible for us to re-run experiments for all specialized approaches due to resource constraints, and the comparison is conservative for many benchmarks.
Table 20 compares different regression approaches for this evaluation setup. Note that the additional results do not diminish the effectiveness of BEL and BEL outperforms direct regression and multiclass classification for all benchmarks and specialized approaches for several benchmarks.
B EXPECTED ERROR DERIVATION
This section explains the expected error equations used to compare BEL-U and BEL-J Sec- tion 3.1. We first explain the encoding and decoding function used for BEL-U and derive the relation between the expected regression error and classification error for BEL-U. Then, we explain the encoding/decoding functions and expected error relation for BEL-J.
B.1 PRELIMINARIES
Given a sample i drawn from dataset with minimum label a and maximum label b, let E Ja, represent the target label for that sample. Assuming uniform quantization, the range of target labels can be quantized using q: [a, - 21, ..., N through Equation 7
N
((v) (y a)
19

---

Published as a conference paper at ICLR 2022
Table 20: Comparison of BEL with different regression approaches. Specialized approaches for each benchmark are described Table
Error (MAE or NME) Model size
Approach training set)
HPFI
Specialized approach (100%)
3.40 69.8M
4.14/ 69.8M
Specialized upproach (80%)
4.08-10.11
4.69:10.022/
Direct regression (804)
6.12 23.5M
5.974:0.09 23.5M
4.08 1:0.1 11 69.8M
4.6744.70/ 6M
Multiclass clossification (80%)
5,38=0.03 24.2M
5.60200 137 24.8M
5.58
72.0M
3.86=0.100 107 72.0M
BEL (803)
3,91=0.08 23.6M
4.91::0:10 10/ 23.6M
3.50::008
69.8
3.99 69.8M 8M
BEL C/D/L functions
URGN-EX//22
U/GEN-EX/BEE
BIDIIGENEXTBCE
UIGN-EEXBCCE
Approach (9 (mmining sel)
FLDI
FLD2
FLD3
FLD4
Specialized approach (100%)
3.32/
4.32/
9.6M
Direct regression (80%)
3.70
10.2M
3.69:10.006
10.2M
4,71 10002
10.2M
1.51 00001 10.2M
Multiclass clossification (80%)
0.64=00007 23.4M
0.60::0.02
45.2M
4.77:0.02
6613M
1.56 +001 / 2M
BEL (80%)
3.35=
10.6
3.40:0033/ 11.2
4,37:0001 ILLM
148
10.8M
BEL C/D/E functions
HEXIFGEN-EXCCE
U/GEN-E/ICE
BUDDIGN-EXICE
BUDDIGEN-EX/CE
Approach training
AEI
AE2
Specialized approach (100%)
21
4.24/18M
Direct regression (80%)
2.45 1:0.01 / 23.1M
3.34 1:002/ / 23.1M
4,56 100457 L&M
Multiclass clossification (80%)
2.85 20.03/23.1M
3.47 +0.0 / 23.1M
6.37 =0007 1.9M
BEL (80%)
2.36 2:0.01 /23.1M
3.20 /0.00/2..1111
3,49 7.8M
BEL E/D/L functions
M/BL-J/BCE
BUDDRGNN-EXII
IIGENCC
We define the encoding function : (1, 1 - {0. 1.Mr M to convert target quantized level Q: E [1, .... N }} } to a binary code B, E {0, 1) We further define the decoding function D: (0. + [a, b] to convert the predicted binary code B, to the predicted label y;-
Although the decoding functions used in this analysis predict the quantized label and introduce quantization error, do not include quantization error in the expected absolute error for analysis as it is constant for both BEL-U and BEL-J. The expected value of absolute error between the target and predicted labels ý is used for the analysis as typically mean absolute error is used as the evaluation metric in regression problems.
Let denote the error probability of binary classifier C used to predict bit ls in binary code B, = E(n) as (n), where n is the target quantized label Q. Then,
(x(n) E( 1/())
(8)
(r = F) for target label Q; = and target binary code B; = En)
where 8 = indicates a correct binary classification by classifier C (8) == b/r for sample and
j F indicates an incorrect binary classification by classifier C* (6) * b/:) for sample i.
B.2 EXPECTED ERROR FOR BEL-U
Encoding and decoding funcciion:
The encoding and decoding functions for BEL-U are defined
as:
{1,
k< Q;
BBELU =b], b,,,,,,, NN-2 where 69
(9)
0,
Otherwise
N 2 8
BELLUU(B! (6..6?....,
+
(10)
20

---

Published as a conference paper at ICLR 2022
Expected error: For target quantized label Qi (n tion eτrоr, the expected error between target y/4 and predicted label ii can derived as:
1)). NN
ignoring the quantiza-
2- Σ
N-2
B( (( vil) E
(1
(0) +1)
Σ
(0) + 1)
N
Σ (0) b5)l)
N-2 2
Σ 10 0/1)
E
(II)
N-2
TE 11O b
N-2
(n) (using Equation
For uniform distribution of target labels in the range [1, N 1], the expected error can derived as:
N 2
E E(
DEL.
vl)
en(n)
(12)
N
B.3 EXPECTED ERROR FOR BEL-J
Encoding and decoding functions: For target quantized label Q: E (1, 1}. BEL-J
encoding requires N bits/binary classifiers. The encoding for BEL-J can be defined as:
2
1, NN --Q.<KEN Qi
(BELJ((Q)) by ,,.,, 0. ,whre by
(13)
0, Otherwise
Tf + Te
0
1
1
+2
3
1
1
4
+3
+4
3
5
0
+4
+4
5
0
7
Decoding function D
Decoded output
B,
Figure 7: Encnding and Decoding functions' output for BEI-J approach and label E [1, N where N = 8.
Decoding function's 's output calculated using y' = Tl + Tf + Tc, where -max (1..... 44 kš),
Tf = ke 5) ( &: 1) 8 and Te
21

---

Published as a conference paper at ICLR 2022
Case 2: Bi
147-2% =
Case
18/2/o41-k
-...
.....
Bit position
Figure 8: Effeett of classifier error on Tf, Tfi for label Q; Case 1 and case represent erroneous outputs. 0/1 1 highlighted in red color represents in the classifier's output. "-" represents error/no in both cases.
Similarly, the decoding functions for BEL-J can defined as:
DBEL (B.) = T((BB) + ((B)) + Te
where, TI(B:)
max kb,
AE (I..... (..
(14)
Tf(B,)
max (E(1... ( - k
85, Tc 1
N
In Equation| 14 TI( finds the location of the last occurrence of "1" in the predicted binary code B,- Similarly, f finds the location of the first occurrence of "1" the binary code B,. Figure 7 gives examples of binary codes for label Q; E {1. 2. .... 7} and the corresponding values of the different terms in Equation| [4] For example, for label Q: 3, the binary code is "0111". Here, the last occurrence of "1" is at position 4, and T' -4. Similarly, the first occurrence of "I" is position 2,
and f = (4 + 1) - =3
Expected error: For BEL-J code, binary classifiers (C²,C²2 C²,..., C) are used. For given input sample i, emror in any of the binary classifiers' outputs (6) ",,,,,,.,.b ...,%)) will result in error between T TfBB))/((B) and T' (B:)) in Equation 14 We refer ( and T(())) as T', and Tl, (predicted binary code), and (B, and (B,),) as Tf, and T4, (target binary code) for brevity. Expected value of the absolute error can be further expanded as:
E( y1l) E(Ify, + T4, Tc (T + TI, Tc)))
= E([(T', - Tf) + (T4 -4)1)
(15)
< -TVff,-ffI-- - T44)
E(ITS, - TTA) + E(177 -T1--TT4]]
Thus, the of the expected error of T)) and TI() is the upper bound of the label's expected emror. Further, we derive the relation between binary classifiers' error probabilities and E([TJ, f; Tfll and E([TI, - 77,))-
We consider Q: = n, where < & for our derivation. such a case, Tf. = n and TV; - N it can be shown that the derived equation for However, as the code symmetric around Q; = 7 [0. 34 can be used for 1 & Q; N
Derivation of [T',, - Tfik: As shown in Equation TS( finds the location & of first occurrence of "1" in the binary sequence. In the case of an erroneous binary sequence, the position of the first occurrence of "1" might shift, which results in an error between T'f; and T ft- Figure 8 shows
examples of the correct and erroneous outputs of classifiers for label Q; = n.. For label Q = n,
By for 21,
n} and by = 1 for & E (
+ .... 2
For case 1, error in a classifier C+ (1,2, classifier C* will result in erroneous Ti, only if all proceeding classifiers are correct, since if any the proceeding classifier is incorrect, i.c. 8: = 1, then the location of the first occurrence of "1" will be shifted z, and any error in the following classifiers
2 N n} is considered, where by = 0 and 8 1.
For & 21,
n),
22

---

Published as a conference paper at ICLR 2022
will not affect the value of Tf. Such case (6)
= b.... T
F. F. i
T/F,..,,8 8, = F)F considers total of 2 combinations out of for ls E 11, N - n). Assuming that the binary classifiers are mutually independent, the error value and the probability of this combination can be shown to be:
N 11+1-k,
IT'; Tfl
(16)
2
Pr(O) = bT T, ..... 8,-11 T. = F) = Pr(B' = Pr = Pr)... PrP = F)
II (1 (1))(n))))) (x(
(17)
The above term considers combinations (b = T,b = T, bk... = T,0 = F1b T/E,.., 8 =T/F) for & E {1,2, N. combinations out of 2
2 - -n}. which constitutes to a total of Σk=1 2 -k
For case 2, error in a classifier C* h E ( n 1, N 7 - n +2 and 8 = 0. We consider combination (b) T, TT. F, by T, i T/F...,, = For this case, the position of the first occurrence of "1" will be moved to k, which will result in erroneous Tf Such a case would cover 2 -k combinations out of 2 for E \ n + 1, N combination can be shown to be:
N is considered, where by: = 1 7 T, b, --- F, F...,. 8/-1
2 n + 2, ..., 2 J. The enror value and the probability of this
.N. F,... 80-1= = T') =
IT', Tfl
k (
+1)
(18)
Pr(O) = T,
T,
6 -- = =T TTb
-nt1
(19)
II (1 (n()) Ø=1
II
ey(m))=
(1-es(n))
1= n+1
The above term considers combinations (6) T, F, B = T, 6,+ T/F,..., 6. TT/F), ). which constitutes to a total of Σ
=T =
=T N -n+1
= F,... 8
2 -k combina-
kmm N nn+1
tions out of 2 for &
N n + 1, n + 22....., ).
Combining Equation 16 to Equation
19 the expected value of (Tf: - fil can be derived as:
A-1
E(TÍ. ((Tff:-Tfaa)) =
n + 1 -k)
(IIa-e ex(m)
2
k=1
2
k
N
2 - n +1)
II (1 - (()()
k-1
II
(es(a)) (1 (1- (l-e)(())
k= -n+1
j=1
-
h-1
+ 1 -k) cx(n)
II (1- ((n))) + j=1
II en())
2
k=
f= (20)
The first term in Equation 20 covers Σ 2
combinations and the second term considers
Σ - 2 combinations. Adding one combination where all the classifiers are correct,
Equation 20 considers all of the possible combinations 2 to find expected value of ITS, Tfal-
23

---

Published as a conference paper at ICLR 2022
Case 2:
IN/21-41
Case
12,
B,
position
Figure Effect classifier enror 7/, Ti, for label Q. n. Case case represent erroneous outputs. 0/1 highlighted in red color represents an error in the classifier's 's output. represents error/no error in both cases.
2. Derivation of [B [T7, T4,: As shown in Equation 14 TI() finds the location & of the last occurrence "1" in the binary sequence. In the case of an erroneous binary sequence, the position of the last occurrence of "1" might shift, which results in an erroneous value of TV; Figure shows examples of correct and erroneous outputs of classifiers for label Qi n.
+ 2, .... F, FF...,," B F, 6 T, 1T, TYF...., 6 = F/F). For this case, position of the last occurrence of "1" will be moved to which will result in erroneous T4. Such a case would cover 2 combinations out 2 The error value and the probability of this combination be shown to be:
For case 1, an eπror in classifier C* k E 1 n + 1,
N is considered, where 2
6f and by O. We consider combination (6, F.
N 2
[T, -7,-441 = - k
(21)
Pr(B N = =F =F, F,..., B.ttt = F. = T)
II
en(m) - (1 ex(())
(22)
j=k+1
The above term considers combinations (6,²
F, b,² -1 F. ..., B F, T, 87.8
T/ 8 = T/F) for l E 1 + 1, n + 2, ..., }. which constitutes to a total of
Σ n+1 2 combinations out of _
For case 2, error in classifier C* , E 21, N n) is considered, where by = 0 and 8 = 1. We consider a combination (6. = = F..... 6, -n+1 6, F, n T, ..., 8 8 F. 0 T/F...,, 6 = F/F). For this case, position of the last occurrence of "1" will be moved to k. which will result in erroneous T4, Such case would cover 2 combinations out of _ The error value and the probability of this combination can be shown to be:
N 2
TT. -1:1
k)
(23)
N pr(6, F,
6 n+1
=F, n
T
Be+1
T, T, F)
n
(24)
ej(n))
II (1 (1- (ex(n)) j=k+1
j= -
The above term considers combinations (6 F, F,... 8 x
F, -"
T, gk+1
T. 8 = F. 1-- = T/F.... by = T/F) for à E {1, .... 2 n), which constitutes to total of Σ Σ -- 2 combinations out of Y
N
Combining Equation
21
Equation
24 the expected value of 174, Tl. can be derived
24

---

Published as a conference paper at ICLR 2022
N k)
E(IT4: T41))
II j=k+1
ej(n)) (1 (1- (x(n))
k= -++1
n
$ --
k
sj(n))
II
(1 - (n(])) (ex(n)) (25)
2
k=1
j= v
4
[II (n)
II
es(n))
II (1- e(())) j=k
k=
-n+1
j=
j= 4=
The first term in Equation 25 covers
n++ 2 combinations and the second term
considers 2 combinations. Adding one combination where all the classifiers are correct,
Equation 25 considers all of the possible combinations 2 to find expected value of 171, - Th/-
Combining Equation 15 Equation 20 and Equation 25 the expected value of error for Q: n in terms of classifiers' error probabilities can be derived as:
N
E(U
REL.J
1/1
- (n).
II (1 (1))()))))= +
II
e(())
2
1=1 1
==1
k= -
-++1
m
N -m
([e(())
II es(n))
(1 ey(n))
(26)
k=
n+1
f=k
1= +
k=1
j=k
As the binary code is symmetric around 2 as shown in Figure 7 the expected errors for label Y: [1, l can be mirrored to find expected enrors for label y; E . 1]. For uniform distribution target labels in the range [1, N - 1], the expected error can be derived as:
N
N
N
N
R-1
A
E(GbELJJ - y) <
m+11k)
(n)
(())) j=l
Σ
II en(n)
N 1
2
n=1
kml
k= ++1
n++1
-n
(I
II ey(n))
II (1 (1-c
(27)
-+
j=k
1= -1+1
We also verify the equation by comparing the expected value of error based on Equation 26 for Qi E (1, - 1} with the expected error calculated by 100, 000 random samples of binary sequences for the same error probabilities ex(m). Figure [0 compares the expected error from Equation 26) and measured from statistical samples, and validates error upper bounds calculated using Equation and
and Equation 27
25

---

Published as a conference paper at ICLR 2022
2.50
Equation (26)
2.25
Statistical Model
2.00
Average Error
1.75
1.50
1.25
1.00
0.75
0.50
25
50
75 Label
100
125
150
Figure 10: Comparison of expected value of eror from Equation probabilities of the classifiers.
26 and random samples for given emoor
C ERROR PROBABILITY CLASSIFIERS
It is known that the emror/misclassffcation probability (n) of classifier tends to increase the target label value n is closer to the classifier's s decision boundaries (Cardoso & Pinto da Costa 2007). We approximate en (v) for classifier C with t bit transitions Gaussian distributions. Here, each Gaussian term centered around a bit transition. Figure TT shows the empirically observed error probability distributions for different classifiers trained for different combinations of network and dataset. We also show the approximate error probability distribution using a linear combination of Gaussian distributions. Here scalar multiplied with probability density of gaussian distribution and a is the standard deviation (Equation 3 and Equation
linear combination of
26

---

Published as a conference paper at ICLR 2022
Target output the dassifier
Target output of the classifier
0.8
0.7
Empirical
0.6
probabilty
Gaussian aporoximation
(r-6.6, d-2.2)
0.5
0.5
0.4
0.3
00.3
0.2
0.2
0.1
0.0
0.0
50
75
100
125
150
50
100
150
200
250
Target Label
Target
(n) ResNer50 (BTWT head pose estimation)
(b) HRNatV2-W18 (AFLW facial landmark detection)
Target output the classiñier
Target output of the classifier
0.6
0.6
Brror probabilty
0.5
(r=3.2, (=2)
0.5
[r-3.9, d=2.1)
0.3
0.3
0.2
0.2
0.1
0.1
0.0
200
250
50
100
150
200
250
Target Label
Target
(c) HRNNVV2-WWW W18 (300W facial landmark detection)
(d) HRNetV2- (300W facial landmark detection)
Target output the classifier
Target output of the classifier
(r-5.5,
Ertor probabilty
0.6
0.6
[r=4, 2.244)
0.5
0.5
0.3
0.3
0.22
0.2
0.1
0.1
0.0
50
100
Target Label
Target Label
HRNeIV2-W18 (WFLW facial landmark dotection)
(1 HRNN (WFLW facial landmark detection)
Figure 11: Classification error probability versus target label for different classifiers, The top horizontal har represent target output of the classifier. Blue color represents output 1.
D EXPERIMENTAL METHODOLOGY
All experiments are conducted on Linux machine with an Intel 19-9900X processor and an Nvidia RTX 2080 Ti GPU with 1GBB of memory. Our code is implemented using Python 3.8.3 with Pytorch 1.5.1 using CUDA 10.2. Our evaluation averaged over training runs with separate seeds.
D. HEAD POSE ESTIMATION
Head pose estimation aims to find a human head's pose in terms of three angles: yaw, pitch, and roll. In this work, consider landmark-free 2D head pose estimation.
27

---

Published as a conference paper at ICLR 2022
Datasets: We follow the evaluation setting of Hopenet (Ruiz al. 2018 and FSA-Net (fsa 2019)
and use two evaluation protocols with three widely used datasets: 300W-LP (Zhu et al. 2016). BIWI (Fanelli al. 2013). and AFLW2000 (Zhu et al. 2016).
Protocol 1: BIWI dataset is used for training and evaluation in this protocol. BIWI dataset consists of 24 videos of 20 subjects with total 15, 128 frames. Three random splits of 70%-30% images are used for training and evaluation. For the BIWI dataset, the yaw angle is in the range ,-75", 75%). }. the pitch in the range [-65", 885). and the roll angle in the range [-55%, , 445)..
Protocol 2: In this setting, the synthetic 300W-LP dataset used for training, consisting of 122, 450 samples. The trained network is tested a real-world AFLW2000 dataset. Yaw, pitch, and roll angles are in the range [-99", [99 for both datasets.
Evaluation metrics: Mean Absolute Error (MAE) between the target and predicted values used as the evaluation metric for this benchmark. MAE for regression task is defined as:
N MAE
(28)
N
j=1
Here, N is the number of test samples, and P is the dimension of the regression task output. For head pose estimation, the dimension of regression output is three (i.e., yaw, pitch, and roll). y is the target, and ÿ is the predicted label.
Network architecture and training parameters: We evaluate our approach on two models: ResNet-50 and RAFA-Net. With ResNet-50, two runs with different random seeds for each combi- nation of learning rate {0.001, 0.0001, 0.00001] and batch size {8, 16] are used for hyperparameter tuning. For data augmentation, images are loosely cropped around the center in the training and testing datasets with random flipping. With RAFA-Net, we use the training parameters and data augmentation used in Behera et al. (202T).
We refer to Protocol 1 evaluated with ResNet-50 as HPE1, Protocol evaluated with RAFA-Net as HPE3, Protocol evaluated with ResNet-50 as HPE2, and Protocol 2 evaluated with RAFA-NO as HPE4. Table 21 provides a summary of the training parameters used with protocol 1. Table 22 provides a summary of the training parameters used with protocol 2.
Table 21: Training parameters for head pose estimation with protocol 1.
Approach
Label rangelQuantization levels
Optimizer
Epochs
Batch size
Leaming rate
Leaminng rate schecuuee
Training time (GPU hours)
Yaw: 1-75", /150,
Adamn, weight
HPEI
Pitch: 65" 85") //500
decay=(),
50
0.0001
1/10 after 30 Epochs
Roll: -55""45") |
momentum 0
RMSProp. momentum=O. rho
HPE3
180%|30000
100
16
0.001
6
Table 22: Training parameters for head pose estimation with protocol 2.
Approach
Label runee/Qunnizatiinn levels
Optimizer
Epochs
Batch size
Leamingg rate
Leamingg schedule
Training time (GPU hours)
Adam, weight decay=0, momentum 0
HPE2
[-99",99") //000
20
16
0.00001
1/10 after 10 Epochs
RMSProp. momentum=O, rho = 0.9
HPF4
[-179%, 1-1779" //00
100
0.001
Related work Existing approaches for head pose estimation include stage-wise soft regres-
sion Yang et al. 2018 [sa 2019). a combination classification and regression (Mukherjed
& Robertson 2015 Ruiz et al. 2018), and ordinal regression (Hsu et al. 2019). SSR-Net (Yang et al.
28

---

Published as a conference paper at ICLR 2022
2018) proposes the use of stage- wise soft regression to the softmax values of classification output to refine the label. FSA-Net (fsa 2019) proposes extending stage-wise estimation to head pose estimation using feature aggregation. HopeNet (Ruiz et al., 2018) uses combination of classification and regression loss to train a model for head pose estimation. Whereas, QuatNet (Hsu et al. (2099) proposes combination of 1.2 loss and custom ordinal regression loss. RAFA-Net (Behera et al.) 2021 uses an attention based approach for feature extraction with direct regression.
We compare BEL with the performance of related work in Table 23 and Table 24 95% confidence intervals are given.
Table 23: Landmark-free 2D Head poses estimation evaluation for protocol (HPEI and HPE3).
Feature
#Params (M)
Approach
Yow
Pitch
Roll
MAE
SSR Net-MD (Yang al. 2018
SSR-Net
1.1
4.24
4.35
4.19
4.26
(Soft regression)
PSA--Cas- Caps-Fusion Fusion [sa 2019
(Soft regression)
PSA-Net
5.1
2.89
4.29
3.60
3.60
ResNet50
Direct regression loss)
23.5
4.62
5.24
4.43
4.76
0.35
(HPE1)
ReaNet50 (IIPE))
BEL U/GEN EXLL
23.6
3.32
3.80
3.53
3.56
0.91
RAFA-Net (Behera et 2021 (Direct Regression)
RAFA-Net
69.8
3.07
4.30
2.82
3.40
(IPES)
BEL BUDIIEEEEEEEEEIIEE
RAFA-Net (HPE3)
69.8
3.21
3.34
3.43
3,30 + 0.04
Table 24: Landmark-free 2D Head poses estimation evaluation for protocol 2 (HPE2 and HPE4).
Approach
Feature Extractor
#Params (M)
Yaw
Pitch
Roll
MAE
SSR-Net-Mii Yann al. 2018)
SSR-Net
1.1
5.14
7.09
5.89
6.01
(Soft regression)
FSA- BLAA-Caps-Fun Pusion Usa 2019
FSA-Net
5.1
4.50
6.08
4.64
5.07
(Soft regression)
HopeNet* (a 2) Ruiz et al.
2018 (classification
ResNe150
23.9
6.47
6.56
5,44
6.16
regression loss)
ResNo130
Direct regression (L2 loss)
23.5
5.85
6.34
4,80
5.65 + 0.13
(HPE2)
ResNe150 (HPE2)
BELLUUGEN-EX/CCC
23.6
4.54
5.76
3.96
4.77 + 0.05
RAFA-Na Net (Behera et al. 2021
RAFA-Net (IPP44)
69.8
3.60
4.92
3.88
4.13
(Direct Regression)
RAFA.Net (HPE4)
FELL-UGEN..N EX/BCF
3.55
0.03
D.2 FACIAL LANDMARK DETECTION
Facial landmark detection a problem of detecting the (a, y) coordinates of keypoints in given face image.
Datasets We use the COFW (Burgos-Artizzu et al. 2013). 300W (Sagonas et al. 2013), WFLW (Wu
et al. 2018), and AFLW (Köstinger et al. 2011) datasets with data augmentation and evaluation protocols described (Wang et al. 2020). ). Data augmentation is performed by random flipping. 0.75 1.25 scaling, and +30 degrees in in-plane rotation for all the datasets. We use 256 quantization levels for binary-encoder labels.
COFW: The COFW dataset (Burgos- Artizzu al. 2013) consists of 1, 345 training and 507 testing
images. Each image is annotated with 29 facial landmarks.
300W: This dataset is combination of HELEN, LFPW, AFW, XM2VTS, and IBUG datasets. Each image is annotated with 68 facial landmarks. The training dataset consists of 3, 148 images. We evaluate the trained model on four test sets: full test set with 689 images, common subset with 554
29

---

Published as a conference paper at ICLR 2022
images from HELEN and LFPW, challenging subset with 135 images from IBUG, and the official test set with 300 indoor and 300 outdoor images.
WFLW: WFLW dataset consists of 7, 500 training images where each image is annotated with 98 facial landmarks. Full test dataset consists 2, 500 images. We use test subsets: large pose (326 images), expression (314 images), illumination (698 images), make-up (206 images), occlusion (736 images), and blur (773 images).
AFLW: Each image has 19 annotated facial key points in this dataset. AFLW dataset consists of 20, 000 training images where each image is annotated with 19 facial landmarks. The full test dataset consists of 4, 836 images, and the frontal test set consists of 1, 314 images.
Evaluation metries: Mean Normalized Error (NME) between the target and predicted values used as the evaluation metric for this benchmark. NME for regression task is defined as:
P Σ j=1
1 NME
yi,
D1.sk
(29)
N P
Here, . is the number of test samples, and the dimension of the regression task output, i.e., the number of landmarks for facial landmark detection, is the target, and ý is the predicted label. L is normalization factor. Inter-ocular distance normalization is used COFW, 300W, and WFLW datasets, and bounding box-based normalization is used for AFLW dataset.
We also report failure rate (f@ 10%) for some datasets. The failure rate (f@ 10%) defined as the fraction of test samples with normalized errors higher than 0.1.
Network architecture and training parameters: We evaluate BEL by applying it HRNetV2- W18. HRNe feature extractor's output is 240 channels of size 64 x 64. For heatmap regression, convolution used get / heatmaps of size 64 64, where / the number of landmarks. Since BEL-X predicts (x, y) coordinates directly we modify the architecture of
HRNetV2-W18 support direct prediction of landmarks. Figure 12 shows the modified architecture of HRNetV2-WIE for BEL-x.
The state-of-the-as approaches for facial landmark detection uses heatmap regression, which mini- mizes the pixel-level loss between the predicted and target heatmaps. We evaluate the applicability of BEL on heatmap regression Appendix contrast, BEL-x predicts (r, y) coordinates directly with 256 quantization levels.
label
32/32/40
32/32 32/32/ilabels
whits
64/64/240
1024 latels
1024
label2
Feature Extactor
Upsample
#bbit
1024
Conv Butchnorm ReLU
MaxPool (3x3, stride 2)
labeIN
Concutenate
256/256/3
#biis
1024
HRNetv2-W15 Feature extractor
Feature extraction extension (our addition)
Figure 12: HRNetV2-W1 -111 feature extractor combined with BEL regressor for (x.y) coordinates
We use two runs with different random seeds to decide the learning rate. We consider learning rates (0.0003, 0.0005, 0.0007) and E (10, 30).
Table 25 provides a summary of all the training parameters. We refer to HRNetV2- W11 evaluated on COFW FLDI, on 300W FL.D2, on WFLW FLD33, and AFLW FLD44
30

---

Published as a conference paper at ICLR 2022
Table 25: Training parameters for facial landmark detection for HRNetV2-W18 feature extractor.
Learning rate (BEL/Direct regres- Mult/Maliicssss classlication)
Dalaset
Optimizer
Egochs
Batch size
Leurning rale schedule
Triining
hours)
Adam, weight decay=0,
0.0005/8.0003/
1/10 after 30 and 50 Epochs
momentum =
0.0003
Adam, weight decay=0,
0.000770.0003/
1/10
300W
momentum = 0
60
0.0003
Epochs
WFLW
Adam, weight decay=0. momentum
60
8
0.000330.
1/10 after 30 and 50 Fpochs
5
0.0003
Adam, weight decay==0,
0.000511.00015/
1/10 alter 30 and 50
AFLW
60
Fppochs
Related work Facial landmark detection is extensively studied problem used for facial anal- ysis and modeling. Common regression approaches for this tasks includes regression using MSE loss (Xiong & De la Torre, 2013 Ly et al., (017), cascaded regression (Miao et al.. 2018 Tz. imiropoulos 2015; Zhu et al. 201 6 Sun et al. (2013), and coarse-to-fing toneett--eeeeeee regression (Sun et al. 2013 Shizhan Zhu et al. 2015 Zhang al. 2014). State-of-the
methods for this task learn heatmaps by regression to find facial landmarks. SAN (Dong et al. (2118) augments training data GAN-generated
using temporal information and GAN generated faces. DVLN (Wu & Yang. 2017). CFSS (Shizhan
Zhu et al. 2015), LAB (Wu et al. 2018). DSRN (Miao et al. 2018) take advantage of correlations between facial landmarks. DAN (Kowalski et al.) 2017) introduces a progressive refinement ap- proach using predicted landmark heatmaps. LAB (Wu et al. 2018)) also exploits extra boundary information to improve the accuracy. LUVLi (Kumar et al.) (2020) proposes a landmark's 's location, uncertainty, and visibility likelihood-based loss. Bulat & Tzimiropoulos (2016) proposes the of binary heatmaps with pixel-wise binary cross-entropy loss. AWing (Wang et al. 2019) proposes adapted wing loss improve the accuracy of heatmap regression. AnchorFace (Xu et al. 2020) demonstrates that anchoring facial landmarks on templates improves regression performance for large poses. HRNet (Wang et al., 2020) proposes CNN architecture maintain high-resolution representations across the network, and uses heatmap regression. The target heatmap is generated by assuming a Gaussian distribution around the landmark location.
We compare BEL with related work in Table 26 95% confidence intervals are provided.
Table 26: Facial landmark detection results on COFW dataset (FLDI). The failure rate is measured at the threshold 0.1. 0 = 30 is used for BEL.
#Params/ GPiops
Approach
Feature Extractor
Test NME
FRo.
LAB (w B) [Wu al.
2018)
Hourglas
25.1/19.1
3.92
0.39
AWing ( (Wang et
2019
Hourglass
25.1/19.
4.94
HRNcIV2-W18 (Wang
2020
HRNaVV2-W18
9.6/4.6
3:45
0.19
(Heatmap regression)
Direct regression loss)
HRNcVV2-W18
10.24.7
3.96 + 0.02
0.29
Direct regression (I.l loss)
HRN
10.2/4.7
3.60
4 0.02
0.29
BEL EXCC
HRNeTV2-W18
10.64.6
3.34 + 0.02
0.40
UUses different augmentation the training
31

---

Published as a conference paper at ICLR 2022
Table Facial landmark detection results on 300W dataset (FLD2).
10 is used BEL.
#Parama/ Gilpps
Appronch
Feature Extractor
Test
Common
Challenging
Full
DAN (Kowal et
2017
3.19
5.24
3.59
Hourglass
Anchorrace (Xu et
20200
ShufteeNtt-2
3.12
6.19
3.72
AWing Wang et al.
2019
Hourglass
25.1/11.1
2.72
4.52
3.07
LUVLI [Kumar ot
2020
CU-Net
2.76
5.16
HRNetV2-Wi1 Wang
HRNcIV2-W18
9.6446
2.87
5.15
3.32
2020 (Heatmap regression)
Direct regression (1.2 loss)
W1S
10.274.7
4.40
3.25
5.65
3.71 = 0.05
Direct regression (L. loss)
VR WI8
10.2/4.7
4,26
3.10
5.42
3.54 J. 0.03
BEI MM--UGEN-FXCC
HRNe1V2 WI8
11.2/4.6
4.09
2.91
5.50
3.40 d: 0.02
Usee different data augmentation for the training
Table 28: Facial landmark detection results (NME) on WFLW test (FLD3) and subsets: pose, expression (expr.), illumination (illu.), make-up up (mu.), occlusion (occu.) and blur. 10 is used for BEL.
#Params/ GFiops
Appreach
Feature Extractor
Test
Expr.
Illu.
Blur
LAB (w B)
Wu et al.
Hourglass
25.1/19.1
5.27
10.24
5.51
5.23
5.15 6.79
6.32
AnchorFacee Xu al.
20200
IIRNeV2--11
7.51
Wang
2019
Hourglass
2/11
4.36
7.38
4.58
4.32
4.27
5.19
4.96
LUNT
(Kmar et al.
CU-Neet
4.37
20200
NRN (Wang
2020
(Heatmap
HRNaWV2W11
9.6/4.6
7.94
4.29
5.42
regression
Direct regression (1.2 luss)
IIRNeIV2 WIS
10.2/4.7
5,56 1. 0.05
10.17
6.13
5.49
5.29
6.83
6.52
Direct regression loes)
HRN
10.2447
4
8.13
BEL-B BIDDGEENE-EECC
HRNGVV2-11
11774.6
4.36 = 0.02
7,33
4.64
4.28
4.19
5.19
5.05
different augmentation the training
Table 29: Facial landmark detection results on AFLW datasct (FL.D4). 0 = 30 used for BEI
PParams/ GFlops
Approach
Full
Frontal
LAB (wlo B) Wu
2018
Hourglass ShuffeNNe-V2
25.1/19.1
1.85
1.62
AnchorFace Xu et
2020
1.56
LUVLi (Kumar
1.39
1.19
HRNetV2-WN W18 (Wang el
2020
IRRNeVV W18
9.644.6
1.57
1.46
(Heatmap regression)
Direct regression (1.2
HRNeVV2
10.2/4.7
2.10
0.02
1.71
Direct regression (L. luss)
IIRNaVV2 WI8
10.24.7
1.51
0.01
1.34
BEL- BEL-BUDDGEN-ENCC
HRNeIV2-W18
10.84466
1.47
0.00
1.30
uUses different data augmentation for the training
32

---

Published as a conference paper at ICLR 2022
D.3 AGE ESTIMATION
Age estimation aims to predict the age given an image of a human head.
Datasets We use the MORPH-II (Ricanek Tesafaye 2006) and AFAD (Niu al. 2016) datasets for our evaluation. Cumulative Score (CS) and MAE are used as evaluation metrics, We preprocess the MORPH-II dataset by aligning images first along the average eye position (Raschka 2018), then by re-aligning so that the tip of the nose is in the center of each image. We do not preprocess the AFAD dataset as faces are already centered. Afterwards, face images are resized to 256 x 256 x and randomly cropped to 224 224 x 3 for training. For testing, a center crop of 224 x 224 x 3 is taken.
This dataset consists of 55,608 face images with age labels between 16 and 70. The dataset is randomly divided into 39,617 training, 4,398 validation, and 11,001 testing images.
MORPH-II:
AFAD: This dataset consists of 164,432 Asian facial images and age labels between 15 and 40. The
dataset is randomly divided into 118,492 training. 13,166 validation, and 32,763 testing images.
Evaluation metries: MAE (Equation 28 is used the evaluation metric. We report Cumulative Score (CS#) for some datasets. CS# is defined as the fraction of test images with absolute error less than years.
Network architecture and training parameters: We evaluate our approach on ResNet- 50. We perform two runs with different random seeds to determine the learning rate between [0.00001, 0.0001, 0.001] and use a batch size of 64 for all experiments. We use ImageNet pretrained weights to initialize the network. Full training parameters are described in Table 30 We refer to our evaluation on MORPH-II as AE1 and AFAD AE2.
Table 30: Training parameters for age estimation using MORPH-II and AFAD dataset
Learning rate schedule
Optimizer
Epochs
Batch size
Learning rate
Adam, weight decay=D,
0.0001
momentum=0
Related work Existing approaches for age estimation include ordinal regression (Niu al. 2016
Cao et al., 2020), soft regression (Yang et al. [2018), and expected value ordinal regression (Pan
ct al.) 2018 Gao et al. 2018). OR-CNN (Niu et al., 2016) proposed the use of ordinal regression
via binary classification to predict the label. CORAL-CNI (Cao et al. 2020) refined this approach
by enforcing the ordinality of the model output. SSR-Net (Yang et al. 2018) proposed the use of
stage-wise soft regression using the softmax of the classification output to refine the predicted label. MV-Loss (Pan et al. 2018) extended the soft regression approach by penalizing the output of the model based on the variance of the age distribution, while DLDL (Gao al.) 2018) proposed to use the KL-divergence between the softmax output and a generated label distribution to train a model.
We compare BEL with related work in Table 31 and Table 32 95% confidence intervals are provided.
D.4 END-TO-END SELF DRIVING
We evaluate our approach on the NVIDIA PilotNet dataset and PilotNet model for end-to-end autonomous driving (Bojarski et al. 2016). In this task, the steering wheel's next angle predicted from an image of the road. We refer to these experiments as PN. MAE (Equation 28 is used as the evaluation metric.
Dataset We use a driving dataset consisting of 45,500 images taken around Rancho Palos Verdes and San Pedro, California (Chen). We crop images to 256 x 70 x 3 then resize them to 200 x 66 x 3. We randomly vary the brightness of the image between |0.2x, 1.5x], randomly flip images, and make random minor perturbations on the steering direction. We use = 10 with 670 quantization levels for BEI..
33

---

Published as a conference paper at ICLR 2022
Table 31: Age estimation results MORPH-II dataset (AE1). 0 10 is used for BEL.
MORPH-II
Approach
Feature extractor
#Parameters (M)
MORPH-II (MAB)
(CS8
ORN
(Ordinal
2.58
0.71
regression by binary
dassiniationn
MV Loss (an el
2018
(Direct
VGG- 16
138.4
2.41
0.889
DLD (Gao et al
2018) (Ordinal
ThinAgeNet
1.96
rogression with mulii-llass classification)
CORAT-CNN (Can al.
2020
(Ordinal
regression by binary classincatioo)
ResNef34
21..3
Diroct Regression (1. Loss)
ResNet50
23.1
2.44
0.01
0,903 + 0.002
BEL.J/EEL.J/RCC
ResNe150
23.1
2.27 1: 0.01
0.928 1: 0.001
augmentation training
Table 32: Age estimation results on AFAD dataset (AE2). 0 10 is used for BEL.
Approach
Feature
#Paramoters (M)
AFAD (MAE)
AFAD (CSP 5)
OR-CNN
(Ordinal
1.0
3.51
0.74
regression by binary classincation
CORAI-CNN (Cn 2020 (Ondinal
ResNe134
21.3
3,47
regression binary clssiictioo
Direct Regression (L2 Loss)
ReaNe150
23.1
3.21
4 0.02
0810 1 0.02
BEL-BIIDVGEN-ML
ResNo150
3.11
0,23 0.001
Training parameters We perform two runs with different random seeds to determine the learning rate between (0.00001, 0.0001, 0.001] and use batch size of 64 for all experiments. Full training parameters are described Table 33
Table 33: Training parameters end-to-end autonomous driving using PilotNet.
Optimizer
Egochs
Batch size
Learning rale
Learning rate schedule
SGD with weight decay=le-5, momentum=0
1/10 at 10, 30 epochs
50
64
0.1
Related work End-to-end autonomous driving is novel task that has become increasingly relevant due to the rise of self-driving vehicles. The autonomous driving model's task is to predict the future driving angle based forward-face facrnd--ainnggg image from the perspective of the vehicle. PilotNet (Bojarski et al. 2017) used a small, application-specific network to provide good accuracy within the time constraints of autonomous driving.
We compare BEL with the baseline PilotNet architecture in Table 34 95% confidence intervals are provided.
34

---

Published as a conference paper at ICLR 2022
Table 34: End-to-end autonomous driving results on PilotNet dataset (PN) and architecture 2016).
(Bojarsii et
2017
Approach
Feature extracter
#Parameters (M)
MAE
PilotNer Bojarski al. 2017
PilotNer
1.8
4,24 0.45
BEL BEGIGEE
PilotNet
1.8
3.11 2: 0.01
35


Reviews:
[{"review_id": "uZz2CHOQ7ec", "reviewer": "Reviewer_HgS5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n- It is interesting to consider such encoding/decoding approaches for regression problems in an application-agnostic setting.\n- The end-to-end training using L1/L2 loss with the proposed D^{GEN-EX} seems reasonable, though experimentally \"CE + GEN-EX\" works better in some cases (D2 vs D4 in Fig.5).\n- The error probability of a binary classifier is modeled well with the mixture of Gaussians, and Fig.3(c) gives a nice explanation for the comparison between BEL-U and BEL-J.\n- The experiments are conducted with a wide range of regression applications.\n\n## Cons\n- I wonder how the best BEL results are selected in Table 2. If they are chosen to minimize the test error, it is not fair to compare them with the results of other approaches. A validation set should be used for model selection.\n- For example, HEXJ/GEN-EX/BCE is chosen for PN and its MAE is 3.11 in Table 2, however, the same combination (D5+HEXJ) results in about 5.2 MAE in Fig.5. Why do such discrepancies occur?\n\n## Minor comments\n- I would like to see more detailed analysis about the dicision boundaries (red lines in Fig.2) and its relationship to the continuous prediction. It is denoted in p.6 that \"Although the hamming distance between 1 and 7 is small, since the differing bits (C1 and C4) are far from decision boundaries and less likely to be mispredicted, a 1 is less likely to be mispredicted as a 7\". I can understand the statement, however, it is not clear how such findings affect the prediction.\n- In Fig.2, what do NRB mean?\n- In Fig.2, I could not understand how B1JDJ and B2JDJ differ in predicting continuous targets.\n- In p.7, facial landmark detection seems better abbrebiated by FLD (instead of FDL).\n- In p.8, it is denoted that \"For HPE4, FDL1, and AE1, Johnson does better than unary.\" However, I can see no such trend in Fig.5.\n- In my opinion, scientific manuscripts should not refer to wikipedia.\n", "summary_of_the_paper": "This paper analyzes regression problems in which continuous targets are predicted using a set of binary classifiers. There is a large design space including quantization, encoding, decoding, and loss functions. An expected value-based function (D^{GEN-EX}) is proposed as a new decoding function.\n\nThe authors apply various combinations of these functions and compare them theoretically and experimentally. It is suggested that no single combination outperforms the others, but the proposed BEL approach has a potential to improve the prediction accuracy of continuous regression problems.\n", "summary_of_the_review": "Overall, I vote for weak rejection. I like the motivation of the work and the authors provide a nice analysis of the problem from both theoretical and empirical perspectives. My major concern is about the clarity of the experiments in which the proposed BEL approach is compared with the existing regression models, as mentioned in the main review above.\n\n=====POST-REBUTTAL COMMENTS========\n\nThanks for the authors' response. The newly added experimental results and manuscript modifications address all of my concerns. I have raised my score to recommend this paper to be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ta_wlQIhVLD", "reviewer": "Reviewer_qL8F", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- as far as i understood the regression network and the decoding function are trained end-to-end. It is unclear how to perform end-to-end training. what is the total loss?\n- is the proposed method applicable to the prediction models other than regression networks? (although end-to-end training may not be available)\n- it would be interesting to see the effect of skewness, multimodality, and outliers of the real-valued label on the effectiveness of the proposed method. they needs to be investigated to demonstrate the robustness of the method.\n- also, how does the scale of the original label and any normalization of the label on the proposed method? \n- for each dataset how did you choose the hyperparameters for label encoding? e.g. which encoding/decoding functions to choos, quantization level and theta. How they can be determined for a new dataset? Although it seems they were empirically determined in the experiments, it would be great if a guideline is suggested.\n- for readers' convinence, please describe what each notation is in the main body of the manuscript. (e.g. theta in Table 1 is only described in appendix)", "summary_of_the_paper": "This study presents a label encoding method (binary encoded labels) to transform a regression problem into multiple binary classifiation problems. The authors designed encoding and decoding functions to encode a real-value into multiple classes and decode the classes into the real value. The proposed demonstrated improved performance on benchmark datasets.\n", "summary_of_the_review": "The idea is interesting. The method is simple yet effective. I think the some issues needs to be clarified.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qO7FTHDyoKS", "reviewer": "Reviewer_t2zZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is detailed with a lot of information useful for the reproduction of the experiments.\n- The theory of the binary-encoded label is well described also with clarifying examples.\n- The analysis of Encoding/Decoding functions has been conducted for a simple Unary coding (BEL-U) and the Johnson coding (BEL-U) (in the Appendix).\n- The authors introduced an estimation of the error probability for BEL-J and BEL-U and an interesting relation between the two methods (in terms of the percentage difference between errors).\n- Even though the usage of classification for regression problem is not original, the work try to shed lights on some design choices.\n\nWeaknesses:\n- The paper is generally well written but I suggest to revise the organization because some parts are introduced after they are effectively used and this can make confusion. \nFor example: \n  - The Unary code, Johnson Code and other Encodings are described on 3.2 but Unary and Johnson are used already in 3.1. I suggest to anticipate the description of the encoding functions or at least for Unary and Johnson ones.\n  - The name of BEL-J decoding is not introduced neither in 3.1 nor in 3.3 but in the appendix and it is used in the caption of Figure 5.  \n- For the two detailed codes (Unary and Johnson), I suggest to describe clearly the Encoding and Decoding functions and then the computation of the error and other aspects present in sections 3.1, 3.2 and 3.3. For this reason I think that the Sections 3.1, 3.2, 3.2 should be revised in order to improve readability. \n- According to Figure 4, the authors seems to have used only 1 Fully connected layer to connect feature vector to the output. In the Figure 13, there is a \"feature extraction extension\" but, generally speaking, it seems that only a shallow network is present between the feature extractor and the output layer. In this situation the using of binary-encoded classifier seems to present some advantages. \nUnfortunately, no tests have been performed to understand if increasing the number of Fully connected layers impact on the results.\n- The BEL approaches outperform the other tested approaches but the best design parameters depend on task and dataset. As authors said this is left for the future works but the doubt is that with more layers the situation could change.\n \n\nOther comments:\n- Figure 1(a) the caption should be \"(a) Training (top) and inference (bottom) flow\"\n- Instead of use \"MAE/NME\" I suggest to use \"Error (MAE or NME)\" as done in Table 2 since the currently used name could be misleading.\n- Why in Table 2 some approaches (all specialized approach and some direct regression) don't have the indication of standard deviation? \n- Caption of Figure 5 should be improved.\n\nSome typos:\nresepct --> respect\ndeocding --> decoding", "summary_of_the_paper": "The paper describes a general framework to solve a regression problem as a binary classification.\nThe labels (targets) are binary encoded and hence the regression problem is solved using a set of binary classifiers.\nSince the process is dependent by the type of Encoding and Decoding used, the authors compare different type of configurations showing, empirically, the effect of different design parameters on the final accuracy.", "summary_of_the_review": "The authors propose a general framework to approach regression problems using binary encoding and a set of classifiers. The paper is detailed, some interesting theoretical aspects are discussed and experiments seem to demonstrate the high accuracy of proposed approach even though only using shallow network after the feature extractor.\n\n=====POST-REBUTTAL COMMENTS========\n\nThe added experimental results and manuscript modifications address all of my concerns. \nI have raised my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "m87CobrDl0W", "reviewer": "Reviewer_zpsJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# A. Strengths\n\n1. **Novelty.**  \n  Error-correcting codes are used in ML for classification tasks since at least 1995. To the best of *my* knowledge, the authors are the first to propose using such codes in regression tasks. This is interesting.\n\n1. **Reported empirical performance seems much better than other methods.**  \nThe results speak for themselves (especially in Table 2). \nI should say that I am not an expert in deep regression tasks, and cannot say whether the methods the authors compare to (e.g. Ruiz et al. 2018 or Behera et al. 2021) are sufficient.\nMoreover, I have some reservations regarding the comparison to the simplest baseline (the direct regression baseline) which I address below.\n\n# B. Weaknesses\n\n1. **Empirical advantage over other methods should be more carefully demonstrated.**  \nSince the reported results are *remarkable* and are claimed to be SOTA, I feel like \"extraordinary claims require extraordinary evidence\".\nImportantly, no code is provided, thus I feel even more obligated to verify that the comparisons are fair.\nEspecially, I am bothered by the comparison to the \"direct regression\" baseline. \nIt seems like the feature extractors' parameters were *frozen* when comparing direct regression to the proposed BEL methods.\nIn this case, I suspect the added linear layer (reduction to $\\theta$ features) significantly helps the model, and that its absence from the direction regression models makes the comparison *unfair*.  \nI would even say that I would be slightly surprised if when the pretrained feature extractors will be also learned (rather than frozen), the direction regression approach will not be competitive with the proposed BEL method.  \nI therefore decrease my general score, until more evidence and details are provided.\n\n2. **Readability issues.**  \nThroughout my reading, several points were not very clear, and it made the reading slower and more difficult than actually needed.\nFor instance,\n* The experimental part is especially hard to understand. It was unclear what method exactly is the \"Multiclass classification\" one. Appendices D-G are also hard to interpret. The reader is expected to cross between many tables that use different names for the same methods. For instance, see Table 11. How can one distinguish between results for HPE2 and results for HPE4?\n* Usage of the \"decision boundaries\" term is untraditional and can be confusing.\n* Description of the HEXJ encoding is hard to understand.\n* Paper says \"We evaluate the Hadamard code to show that...\" but the results are not actually shown anywhere.\n* Figure 1b is hard to understand, especially on B&W hard copies.\n\n\n3. **Analytic results do not contribute much.**   \nSection 3.1 offers some basic analysis of two encoding-decoding combinations (using the Unary and Johnson codebooks).\nI am not sure what is the contribution of this analysis to the rest of the paper or to the readers' understanding.\n\n# C. Other comments / thoughts\n\n1. I allow myself to assume the Hadamard codebook fails because each column has many \"bit flips\" (i.e. decision boundaries in the authors' terminology). This implies that traditional classification codebooks (e.g. Hadamard and random ones) are not suitable for regression tasks. \n\n1. Using a fine quantization, codebooks like the unary or Johnson codebooks will largely \"inflate\" the output space. The authors' HEXJ codebook hints that the output space does not have to grow *linearly* with the number of quantization \"buckets\". This is related to the use of ECOC for extreme classification (see *Learning compact class codes for fast inference in large multi class classification* [Cisse et al. 2012] and *Efficient Loss-Based Decoding on Graphs for Extreme Classification* [Evron et al. 2018]). A brief discussion in that direction can be interesting.\n\n1. In the second paragraph of Section 3.2, the statement \"we note that a Hamming distance does not...\" should be justified.\n\n1. Just before equation (1), should it be \"for $k>Q_i$ instead of $k<Q_i$\"?\n\n1. The decoding function in equation (1) is not really intuitive and it feels unmotivated given the structure of the unary code.\n\n1. In Table 2, the best FDL2 method is the specialized one, not BEL.", "summary_of_the_paper": "The paper tackles regression problems using an error-correcting code (ECOC) approach, i.e., reducing a *regression* problem into multiple binary *classification* problems.\nECOC have been so far mainly used for classification tasks, and their application to regression problems in this paper is elegant and seems novel.\nThe paper examines several combinations of (encoding method $\\times$ decoding method $\\times$ loss function) on several (deep) regression tasks and reports significant improvements over existing methods.\nSome brief analytic discussion is made.", "summary_of_the_review": "Idea is novel and interesting.\nResults and findings are mainly empirical and look remarkable, achieving SOTA performance.\nI believe these findings require more careful description of the exact empirical process, and that as a reviewer I should make sure the comparison to other baselines is fair. A provided code would also be helpful and more assuring.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focused on deep regression problems and proposed a label encoding technique which can be thought as a sibling of the famous error-correcting output codes but designed for regression problems. The main idea is well illustrated in Figure 1 at the top of page 3, where the encoder and decoder are the main objects of the proposal (and a quantizer is also needed for using the encoder/decoder which is a uniform quantizer in the paper). The idea/proposal is supported by solid theoretical arguments and convincing empirical evidences (not only the paper but also the rebuttal). While there were some concerns in the beginning, the authors have successfully clarified all the concerns and then the average score has been increased from 5.5 to 7.5. As a result, the paper is clearly above the bar of acceptance. What is more, an advantage is that the proposed method is task-agnostic and can be combined with different task-specific feature extractors borrowed from very complex regression problems (e.g., head pose estimation, facial landmark detection, age estimation, and end-to-end autonomous driving), making its significance and potential impact high. Given these facts, I think the paper might be selected as a spotlight presentation at the conference.
Decision: Accept (Spotlight)