Year: 2021

Abstract:
Deep neural networks are used for a wide range of regression problems and are typically trained by minimizing the squared or absolute error of output labels. Solving a regression problem with a set of binary classifiers can improve the accuracy and also utilizes well-studied binary classification algorithms. However, a limited design space of regression by binary classification has been studied by prior works. This work introduces a novel approach using binary-encoded labels (BEL) to generalize the application of binary classification to regression, and a taxonomy identifying the key design aspects of such formulation. Of these design aspects, we primarily explore and identify the desirable properties of suitable encoding and decoding functions used for the conversion between real-valued and binary-encoded labels based on theoretical and empirical study. BEL can be combined with off-the-shelf task-specific feature extractors and trained end-toend. We further propose a series of sample encoding, decoding, and training loss functions applicable in this domain and demonstrate that our generic regression approach results in lower error than direct regression and specialized approaches while being suitable for a diverse set of regression problems, network architectures, and evaluation metrics.

text: Deep neural networks are used for a wide range of regression problems and are typically trained by minimizing the squared or absolute error of output labels. Solving a regression problem with a set of binary classifiers can improve the accuracy and also utilizes well-studied binary classification algorithms. However, a limited design space of regression by binary classification has been studied by prior works. This work introduces a novel approach using binary-encoded labels (BEL) to generalize the application of binary classification to regression, and a taxonomy identifying the key design aspects of such formulation. Of these design aspects, we primarily explore and identify the desirable properties of suitable encoding and decoding functions used for the conversion between real-valued and binary-encoded labels based on theoretical and empirical study. BEL can be combined with off-the-shelf task-specific feature extractors and trained end-toend. We further propose a series of sample encoding, decoding, and training loss functions applicable in this domain and demonstrate that our generic regression approach results in lower error than direct regression and specialized approaches while being suitable for a diverse set of regression problems, network architectures, and evaluation metrics.

heading: 1 INTRODUCTION
text: Deep regression networks, in which a continuous output is predicted for a given input, are traditionally trained by minimizing squared/absolute error of a logit, which we refer to as direct regression. More recently, several works have proposed specialized regression approaches for regression problems such as head pose estimation, age estimation, and facial landmark estimation, and shown significant accuracy improvements over direct regression. However, developing task-agnostic generic approaches to improve accuracy for regression is desirable.
A regression problem can be posed as a set of binary classification sub-problems. A similar approach has been applied to other domains such as ordinal regression (Li & Lin, 2006) and multiclass classification (Dietterich & Bakiri, 1995). Such formulation allows the use of well-studied binary classification approaches. Further, new generalization bounds for ordinal regression or multiclass classification can be derived from the known generalization bounds of binary classification. This reduces the efforts for design, implementation, and theoretical analysis significantly (Li & Lin, 2006). Dietterich & Bakiri (1995) demonstrated that posing multiclass classification as a set of binary classification problems can increase error tolerance and improve accuracy. However, the proposed approaches for multiclass classification do not apply to regression due to the differences in task objective and properties of the error probability distribution of classifiers (Section 2). On the other hand, prior works on ordinal regression have explored the application of binary classifiers in a more restricted way which limits its application to a wide range of complex regression problems (Section 2). There exists a lack of a generic framework that unifies possible formulations for using binary classification to solve regression.
In this work, we propose binary-encoded labels (BEL) which improves regression by generalizing the application of binary classification. In BEL, a target label is quantized and converted to a binary code of length M , and M binary classifiers are then used to learn binary-encoded labels. An encoding function is introduced to convert the target label to a binary code, and a decoding function is introduced to decode the output of binary classifiers to the predicted label. BEL allows using an adjustable number of binary classifiers depending upon the quantization, encoding, and decoding
functions. BEL opens possible avenues to improve the accuracy of regression problems with a large design space of the quantization function, encoding/decoding functions, and optimization algorithm.
We focus on the encoding and decoding functions and theoretically study the relations between the absolute error of label and binary classifiers’ errors for sample encoding and decoding functions. This analysis demonstrates the impact of binary classifiers’ error distribution over the numeric range of target labels on the suitability of different encoding and decoding functions. Based on our analysis and empirically observed binary classifiers’ error distribution, we propose properties of suitable encoding functions for regression and explore various encoding functions on a wide range of tasks. We also propose an expected correlation-based decoding function for regression that can effectively reduce the quantization error introduced by the encoding.
A deep regression network consists of a feature extractor and a regressor and is trained end-toend. Our proposed regression approach (BEL) can be combined with off-the-shelf task-specific feature extractors by increasing the regressor’s output logits. We find that the correlation between multiple binary classifiers’ outputs can be exploited to reduce the size of the feature vector and consequently reduce the number of parameters in the regressor. We explore the use of decoding functions for training loss formulation and evaluate the use of the binary cross-entropy, cross-entropy, and squared/absolute error loss functions for BEL. We evaluate our approach on four complex regression problems: head pose estimation, facial landmark detection, age estimation, and end-to-end autonomous driving. We make the following contributions in this work:
• We propose binary-encoded labels for regression and introduce a general framework and a taxonomy for the design aspects of regression by binary classification. We propose desirable properties of encoding and decoding functions suitable for regression problems.
• We present a series of suitable encoding, decoding, and loss functions for regression with BEL. We present an end-to-end learning approach and regression layer architecture for BEL. We combine BEL with task-specific feature extractors for four tasks and evaluate multiple encoding, decoding, and loss functions. BEL outperforms direct regression for all the problems and specialized approaches for several tasks.
• We theoretically and empirically demonstrate the effect of different design parameters on the accuracy, how it varies across different tasks, datasets, and network architectures, and provide preliminary insights and motivation for further study.

heading: 2 RELATED WORK
text: Binary classification for regression: Prior works have proposed binary classification-based approaches for ordinal regression (Crammer & Singer, 2001; Chu & Keerthi, 2005; Li & Lin, 2006). Ordinal regression is a class of supervised learning problems, where the samples are labeled by a rank that belongs to an ordinal scale. Ordinal regression approaches can be applied to regression by discretizing the numeric range of the real-valued labels (Fu et al., 2018; Berg et al.; Niu et al., 2016). In the existing works on ordinal regression by binary classification, N − 1 binary classifiers are used for target labels ∈ {1, 2, ..., N}, where classifier-k predicts if the label is greater than k or not for a given input. Li & Lin (2006) provided a reduction framework and generalization bound for the same. However, the proposed binary classification formulation is restricted. It requires several binary classifiers if the numeric range of output is extensive, whereas reducing the number of classifiers by using fewer quantization levels increases quantization error. Thus, a more generalized approach for using binary classification for regression is desirable to allow flexibility in the design of classifiers. Binary classification for multiclass classification: Dietterich & Bakiri (1995) proposed the use of error-correcting output codes (ECOC) to convert a multiclass classification to a set of binary classification sub-problems. This improves the accuracy of the network as it introduces tolerance to binary classifiers’ errors depending upon the hamming distance (i.e., number of bits changed between two binary strings) between two codes. Allwein et al. (2001) provided a unifying framework and multiclass loss bounds in terms of binary classification loss. More recent works have also used Hadamard code, a widely used error-correcting code (Song et al., 2021; Verma & Swami, 2019). Other works have focused on the problem of extreme classification and the use of ECOC and compact codes (sublinear increase in the length of codes with the number of classes) for a classification problem with a large number of classes (Cissé et al., 2012; Evron et al., 2018). However, the proposed encoding and decoding approaches do not consider the ordinality of labels for regression. Further,
the binary classifiers possess distinct error probability distribution properties for regression problems as observed empirically (Section 3.1), which can be exploited to design codes suitable for regression.
Multiclass classification and ordinal regression by binary classification can be viewed as special cases falling under the BEL framework. As shown in Section 4, other BEL designs yield improvements in accuracy over these approaches. Task-specific regression techniques are well explored as summarized below (see also Appendix D. While effective, task-specific approaches lack generality by design.
Head pose estimation: SSR-Net (Rothe et al., 2015) and FSA-Net (Yang et al., 2019) used softmax to refine labels. HopeNet (Ruiz et al., 2018) used a combination of classification and regression loss. QuatNet (Hsu et al., 2019) used a combination of regression and ordinal regression loss. Facial landmark detection: Wang et al. (2020) minimize L2 loss between predicted and target 2D heatmaps with the latter formed using small variance Gaussians centered on ground truth landmarks. AWing (Wang et al., 2019) modified loss for different pixels in the heatmap. LUVLi (Kumar et al., 2020) proposed a landmark’s location, uncertainty, and visibility likelihood-based loss. Bulat & Tzimiropoulos (2016) used binary heatmaps with pixel-wise binary cross-entropy loss. Age estimation: OR-CNN (Niu et al., 2016) and CORAL-CNN (Cao et al., 2020) used ordinal regression via binary classification. MV-Loss (Pan et al., 2018) proposed to penalize the model output based on the age distribution’s variance, while Gao et al. (2018) proposed to use the KL-divergence between the softmax output and a generated label distribution for training.

heading: 3 BINARY-ENCODED LABELS FOR REGRESSION (BEL)
text: We consider regression problems where the goal is to minimize the error between real-valued target labels yi and predicted labels ŷi, over a set of training samples i. We transform this problem to a set of binary classification sub-problems by converting a real-valued label to a binary code.
Figure 1 shows the training and inference flow for BEL. The red-colored blocks highlight functions that vary under BEL. A real-valued label yi ∈ R is quantized to level Qi ∈ {1, 2, ..., N}. The quantized label is then converted to a binary sequence Bi ∈ {0, 1}M using an encoding function E . The binary-encoded labels Bi are used to train M classifiers. In the prediction phase, the classifiers predict a binary code B̂i ∈ {0, 1}M for an input xi. The predicted binary code (B̂i) or the real-valued confidence vector of classifiers’ prediction (Ẑi) is then decoded to a predicted label ŷi ∈ R using a decoding function D. We explore decoding functions that yield either quantized or continuous predicted outputs. For the latter, we propose an expected correlation-based decoding function that decodes classifiers’ output to a continuous value, reducing the quantization error (Section 3.3).
There are five key design parameters when applying BEL: quantization, encoding, decoding, regressor network architecture, and training loss formulation yielding a large design space. In this work we consider only uniform quantization while leaving nonuniform quantization (Fu et al., 2018) to future work. Section 3.2 and 3.3 explore the characteristics of suitable encoding, decoding, and training loss functions. Section 3.4 explore the impact of regression network architecture. We find varying any of these aspects can improve accuracy. While BEL provides a framework and some codes appear generally better than others, the most suitable BEL code to employ appears to vary across task, dataset, and network architecture, as we show both theoretically (Section 3.1) and empirically (Section 4).

heading: 3.1 ANALYSIS OF ENCODING/DECODING FUNCTIONS
text: This section analytically analyzes the potential impact of encoding/decoding functions on regression error based upon empirically observed error distributions for the underlying classifiers. With this
analytical study, we aim to understand the properties of classifiers for ordinal labels and their effect on regression error using simple {E ,D} functions and draw insights on desirable properties of these functions for regression and empirical results. We use unary (Figure 2b) and Johnson code (Figure 2c) for the analysis. The design of these codes and intuition for trying these codes are discussed in Section 3.2. We divide our analysis into three parts. First, the expected error of predicted labels is derived in terms of classifiers’ errors for two {E ,D} functions. Next, we propose an approximate classifier’s error probability distribution over the numeric range of target labels for regression based on empirical study. Lastly, we compare the expected error of sample {E ,D} functions based on our analysis. We use labels yi ∈ [1, N−1], with quantization levelsQi ∈ {1, 2, ..., N−1}. Quantization error is not included as it is not affected by {E ,D} functions.
Expected absolute error bounds in terms of classification error: First, we analyze the expected absolute error given the classification error for a unary code, BEL-U. The encoding function EBEL-U converts Qi to Bi = b1i , b 2 i , ..., b N−2 i , where b k i = 1 for k < Qi, else 0. A single-bit error changes the predicted value by one quantization level for a decoding function that counts the number in the binary code. Hence the decoding function is defined as:
DBEL-U(b̂1i , b̂2i , ..., b̂N−2i ) = N−2∑ k=1 b̂ki + 1 (1)
Let ek(n) be the error probability of classifier k for target quantized label Qi = n. For a uniform distribution of yi in the range [1, N − 1], the expected error for BEL-U is:
E(|ŷ − y|) 6 1 N − 1 N−1∑ n=1 (N−2∑ k=1 ek(n) )
(2)
A similar analysis is applied for the Johnson code (referred to as BEL-J) in which Qi is encoded to a binary code Bi = b1i , b 2 i , ..., b N/2 i , where, b k i = 1 for N 2 −Qi < k 6 N −Qi, else 0 (Appendix B).
Error probability of classifiers: A classifier’s target output is 0 or 1 for quantized target labels on an ordinal scale, and consists of one or more bit transitions. The bit transition is a 0→ 1 or 1→ 0 transition in the classifier’s target output over the numeric range of target labels. For example, for the unary code (Figure 2b), the target output of classifier 2 (b2) changes from 1 to 0 at a bit transition between Qi = 2 and Qi = 3. This classifier should learn a decision boundary between targets in [1, 2.5] and [2.5, 8]. Thus, ideally, a classifier learns a decision boundary for all bit transitions, and the complexity of the decision boundary increases with the number of bit transitions. It is known that the error/misclassification probability of a classifier tends to increase as the target label is closer to the classifier’s decision boundaries (Cardoso & Pinto da Costa, 2007). Thus, we approximate ek(y) for a classifier k with t bit transitions as a linear combination of t Gaussian distributions. Here, each Gaussian term is centered around a bit transition. Let fN (µ,σ2)(y) denote the probability density of a normal distribution with mean µ and variance σ2. Each classifier for BEL-U encoding has one bit transition, whereas, each classifier for BEL-J encoding has two bit transitions (except the first and last classifiers). ek(y) of a classifier k for BEL-U and BEL-J encoding is approximated as:
eBEL−Uk (y) = rfN (µk,σ2)(y),where, µk = k + 0.5 (3)
eBEL−Jk (y) = rfN (µ1k,σ2)(y) + rfN (µ2k,σ2)(y),where, µ1k = N
2 − k + 0.5, µ2k = N − k + 0.5 (4)
Here, r is a scaling factor for the error probabilities. Figure 3a and Figure 3b show the empirically observed and proposed approximate error probability distributions for two classifiers trained using HRNetV2-W18 (Wang et al., 2020) and COFW facial landmark detection dataset (Burgos-Artizzu et al., 2013) for labels y ∈ [1, 256].
Comparison of expected absolute error for BEL-U and BEL-J: Based on equation 2- equation 4, we compare expected absolute errors of BEL-U and BEL-J encoding/decoding functions. Figure 3c represents the percentage increase in absolute error for BEL-U compared to BEL-J for valid values of variance σ2 (y−axis) and scaling factor r (x−axis) from equation 3 and equation 4. Here, BEL-J has a lower error in the red-colored region (%increase> 0), whereas BEL-U has a lower error in the blue-colored region (%increase < 0). The figure shows that BEL-J or BEL-U has lower error depending upon the values σ and r used to approximate the classifiers’ error probability distribution. We leave the derivation of these parameters from the first principles to future work. Further theoretical study of the properties of classifiers for ordinal labels can provide bounds on these values, which will also help with encoding/decoding design. Thus, the choice of {E ,D} functions heavily depends upon the classifiers’ error probability distribution, which depends upon many factors, including the task, dataset, label distribution, network architecture, and optimization approach. We provide the derivation of expected error for BEL-U and BEL-J and classifiers’ empirical error probability distributions for different architectures, datasets, and encodings in Appendix B-C.

heading: 3.2 DESIGN OF ENCODING FUNCTIONS
text: The distance between two codes (e.g., Hamming distance) gives a measure of error correction capability. The objective of regression is to minimize the absolute/squared error between predicted and target values. The encoding function for BEL can be designed to take advantage of this fact. Thus, a desirable property for a BEL encoding function is that: the distance between two quantized values should be proportional to the distance between their codes.
Further, the error/misclassification probability of classifiers tends to increase as the target gets closer to a bit transition (Figure 3a-3b). Since hamming distance (i.e., number of bits changed between two codes) weighs all the bits equally, hamming distance-based code design provides equal error protection capability to all the bits (Wu, 2018; Xie et al., 2002) and does not account for which classifiers are more likely to mispredict for given target value. This gives the second property of a suitable encoding: the distance metric must consider how likely each classifier is to mispredict for some target value to account for classifiers’ nonuniform error probability distribution for a regression problem. This design property covers the impact of bit transitions in codes on continuous predictions.
We evaluate several encodings that to greater or lesser extent exhibit these desirable properties. We focus on encodings that reduce the number of classifiers (bits) while balancing the separation between the bit transitions per classifier. Below we empirically evaluate the impact of these different encodings on previously proposed deep regression networks. We leave the development of procedures for automatically discovering optimized encodings to future work. Figure 2 shows example codes for encodings explored. We explain all encodings in detail below.
Unary code (U): The unary code (Figure 2b) is a simple code with one bit transition per classifier and requires N − 1 bits to encode N values. Prior works on ordinal regression by binary classification (Li & Lin, 2006; Niu et al., 2016) uses a similar formulation of classifiers as the unary code.
Johnson code (J): The Johnson code sequence (Figure 2c) is based on Libaw-Craig code (Libaw & Craig, 1953). We select this code as it has well-separated bit transitions and requires N2 bits compared to N required for unary code. This code exemplifies the difference between hamming distance and a distance that considers classifiers’ error probability distribution. Although the hamming distance between 1 and 7 is small, since the differing bits (C1 and C4) are far from bit transitions and less likely to be mispredicted, a 1 is less likely to be mispredicted as a 7, reducing the regression error.
Base+displacement based code (B1JDJ/B2JDJ): We further reduce the number of bits using a base+displacement-based representation. In this representation, a value is represented in base-k using a radix-term r and displacement d via r * k + d. r and d are represented using Johnson codes. Further, to improve the distance between two remote codes, we adapt reflected binary codes for term d (Gray, 1953). We evaluate base-2 (B1JDJ - Figure 2e) and base-4 codes (B2JDJ - Figure 2f).
Binary coded hex - Johnson code (HEXJ): In HEXJ, each digit (0-F) of the hexadecimal representation of a number is converted to an 8-bit binary code using Johnson code. For example, for the decimal number 47 (i.e., 2F in hex), HEXJ(47) = Concetanate(Johnson(2), Johnson(F)). A 16-bit HEXJ code can represent numbers in the range of 00 to FF (a total of 256). The number of bits increases sublinearly with the number of quantization levels for HEXJ, making it suitable for regression problems with many quantization levels.
Hadamard code (HAD): Hadamard codes (Bose & Shrikhande, 1959) are widely used as errorcorrecting codes due to their fault-tolerant properties and also have been used to encode classes in multiclass classification (Dietterich & Bakiri, 1995; Verma & Swami, 2019). It requires N bits to encode N values. However, in the Hadamard code, each column has many bit transitions, increasing the complexity of the decision boundary to be learned by each binary classifier. Further, the Hadamard code does not follow the desirable properties proposed in this work for regression tasks. We evaluate the Hadamard code to show that it is not suitable for regression (Section A).

heading: 3.3 DESIGN OF DECODING FUNCTIONS
text: We explore three decoding functions: custom decoding, correlation-based decoding, and expected correlation-based decoding. Custom decoding functions are specific to the encoding function, and are only evaluated for unary and Johnson codes. In contrast, correlation-based decoding, first explored in prior work studying ECOC for multiclass classification (Allwein et al., 2001), can be applied to all codes. For labels in {1, 2, ..., N}, we define a code matrix C of size N ×M , where M is the number of bits/classifiers used for the binary-encoded label. Each row Ck,: in this matrix represents the binary code for label Qi = k. For example, Figure 2b can be considered a code matrix, where the first row represents code for label Qi = 1. Let Ẑi ∈ RM denote the output logit values of the classifiers. For decoding, the row with the highest correlation with the output Ẑi is selected as the decoded label. Output logit value vector Ẑi is used instead of output binary code B̂i to find the correlation as it uses the confidence of a classifier to make a more accurate prediction. For target quantized labels Qi ∈ {1, 2, ..., N}, the decoding function is defined as:
DGEN(Ẑi) = argmax k∈{1,2,...,N}
( Ẑi ·Ck,: ) (5)
However, DGEN outputs a quantized prediction, introducing quantization error. To remedy this concern and demonstrate the potential of more sophisticated decoding rules, we propose and evaluate an expected correlation-based decoding function, which allows prediction of real-valued label ŷi. For target labels yi ∈ [1, N ], the decoding function is defined as:
DGEN-EX(Ẑi) = N∑ k=1 kσk,where σk = eẐi·Ck,:∑N k=1 e Ẑi·Ck,: (6)
Training loss functions: A deep neural network with multiple output binary classifiers can be trained using the binary cross-entropy (BCE) loss LBCE ( Ẑi, E(Qi) ) . However, this loss minimizes
the mismatch between predicted and target code but does not directly minimize the error between the target and predicted values. Decoding functions DGEN and DGEN-EX can be used to calculate the loss and minimize the mismatch between decoded predictions and target values directly. Decoding function DGEN finds the correlation between each row of the code matrix (Ck,:) and the output Ẑi. CẐi gives the correlation vector, and the index with the highest correlation is used as the predicted label. In this case, cross-entropy loss LCE ( CẐi, Qi ) can be used to train the network. Similarly, for decoding
functionDGEN-EX, which predicts a continuous value, L1 or L2 loss LL1/L2 ( DGEN-EX(Ẑi), yi ) can also be used for training. We evaluate multiple combinations of decoding and loss functions in Section 4.

heading: 3.4 REGRESSION NETWORK ARCHITECTURE FOR BEL
text: A deep regression network typically consists of a feature extractor and regressor. The regressor consists of a fully connected layer between the feature extractor’ output (i.e., feature vector) and output logits. Figure 4a represents a network architecture for direct regression. In BEL, the number of output logits is increased to the number of classifiers (bits) used. The size of the fully connected regression layer increases with the number of output logits. However, empirically, we find small feature vectors suffice as the output logits are highly correlated for the encoding functions we explore. Adding a fully connected layer to reduce feature vector size to θ reduces the number of parameters and provide a trade-off between the model size and accuracy. Figure 4b shows the modified neural network architecture for BEL.

heading: 4 EVALUATION
text: Table 1 summarizes the tasks, datasets, and network architectures used for the evaluation of BEL. These tasks are commonly used for evaluation of regression approaches by prior works due to the complexity of problem and network architectures (Díaz & Marathe, 2019). Landmark-free 2D head pose estimation (HPE) aims to find a human head’s pose in terms of three angles: yaw, pitch, and roll from a 2D image without landmarks. Facial landmark detection (FLD) is a problem of detecting the (x, y) coordinates of keypoints in a given face image. Age estimation aims to predict the age of a person from an image. In end-to-end autonomous driving, the steering wheel’s next angle is predicted from an image of the road. Normalized Mean Error (NME) and Mean Absolute Error (MAE) with respect to raw real-valued labels are used as the evaluation metric for FLD and the rest, respectively.
The feature extractor, data augmentation, evaluation protocols, and the number of training iterations are kept uniform across different methods for each benchmark. We also evaluate direct regression and multiclass classification as baseline regression approaches. For direct regression, L1 or L2 loss functions are used. Label values are scaled to reduce the range of labels. The loss function and the scaling factors are set using hyperparameter tuning. In the multiclass classification-based regression, the target values are quantized and converted to a class. The network is trained using cross-entropy loss in this case. In our evaluation, the entire network (i.e., feature extractor and regressor) are trained end-to-end for direct regression, multiclass classification, and BEL. We report average of five training runs and error margin of 95% confidence interval. Details on datasets, training parameters, related work for specific tasks, and other evaluation metrics are provided in Appendix C.
BEL introduces several design parameters for regression by binary classification. We evaluate different encoding (E), decoding (D), and training loss (L) functions for BEL across all the benchmarks and study the extent and nature of the impact of these design parameters on accuracy.
Encoding function (E): Figure 5 plots error (MAE or NME) using different encodings. We do not show results for Hadamard codes in the figure as it results in significantly higher error than other encodings (Appendix A). On average, Hadamard codes result in 60% higher error than J encoding, which shows that these codes are unsuitable for regression. The results show the encoding function has a significant impact on the accuracy of several benchmarks and the best-performing encoding function varies across tasks, datasets, and network architectures (e.g., HPE1 and HPE3 are trained on the same dataset and different architecture). In Section 3.1 we observed that which encoding/decoding functions result in lower error depends upon the classifiers’ error distribution. For HPE3, FLD1, and AE1, J does better than U for D1 (decoding functions used for analytical comparison in Section 3.1); we attribute this to misclassification errors occurring more frequently near bit transitions in these tasks based on the analytical study.
The encoding function impacts the number of classifiers and the complexity of the function to be learned by a classifier. We observe a trade-off between these two parameters. For some benchmarks, the availability of sufficient training data and network capacity facilitates the learning of complex classifiers such as B2JDJ. In such a case, a reduced number of classifiers compared to U, J, or B1JDJ codes results in a lower error. We provide empirical results for the same in Appendix A.
Decoding (D) and training loss (L) functions: We explore three decoding functions (Section 3.3). The training loss functions can be selected from BCE, CE, and L1/L2 loss. However, not all the combinations of decoding and loss functions perform well. For example, CE, L1, or L2 losses do not use BEL-J or BEL-U decodings. Therefore, optimizing the network for these losses does not directly minimize the absolute error between the target and decoded predictions. In contrast, GENEX decoding is used for L1/L2 loss function calculation; in such a case, the training loss directly
minimizes the error between target values and decoded values. Out of such nine combinations, we present results for five combinations. Figure 5 compares error (MAE or NME) achieved by different D/L combinations and highlights the range of error variations across D/L. DGEN-EX results in the lowest error for the majority of the benchmarks as it reduces quantization error and also takes advantage of the output logit confidence values. DGEN-EX consistently perform better than DGEN function that has been used for multiclass classification by prior works (Allwein et al., 2001). The use of CE or L1/L2 loss results in a lower error with DGEN-EX for most benchmarks as the training loss function minimizes the error between decoded labels and target labels.
Comparison of BEL with regression approaches: Table 2 compares BEL with other approaches for different benchmarks (Table 1). We explore and evaluate multiple combinations of encoding (E), decoding (D), and loss (L) functions for BEL in this work. The 20% of the training set is used as validation set and the validation error is used to choose the best BEL approach. An ablation study for using more fully connected layers for direct regression and multiclass classification in Appendix A. BEL results in lower error than direct regression and multiclass classification and even outperforms task-specific regression approaches for several benchmarks.
The results show no single combination of encoding/decoding/loss functions evaluated was best for all benchmarks but also demonstrate BEL improves accuracy across a range of regression problems.

heading: 5 CONCLUSION
text: This work proposes binary-encoded labels (BEL) to pose regression as binary classification. We propose a taxonomy identifying the key design aspects for regression by binary classification and demonstrate the impact of classification error and encoding/decoding functions on the expected label error. Different encoding, decoding, and loss functions are explored to evaluate our approach using four complex regression tasks. BEL results in an average 9.9%, 15.5%, and 7.2% lower error than direct regression, multiclass classification, and task-specific regression approaches, respectively. BEL improves accuracy over state-of-the-art approaches for head pose estimation (BIWI, AFLW2000), facial landmark detection (COFW), age estimation (AFAD), and end-to-end autonomous driving (PilotNet). Our analysis and empirical evaluation in this work demonstrate the potential of the vast design space of BEL for regression problems and the importance of finding suitable design parameters for a given task. The best performing encoding/decoding function pair may be task, dataset, and network specific. A possibility this suggests, which we leave to future work, is that it may be beneficial to develop automated approaches to optimizing these functions,
Reproducibility: We have provided a detailed discussion about training hyperparameters, experimental setup, and modifications made in publicly available network architectures in Appendix D.1-D.4 for all benchmarks. We have added a comment directed to the reviewers and area chairs to provide a link to our training and inference code with trained models. We will publicly release our code if accepted.
Code of Ethics Some of the major applications of regression problems are artificial intelligence and autonomous machines, and regression improvement can accelerate the development of autonomous systems. However, depending upon the use, autonomous systems can have some negative societal impacts, such as job loss in some sectors and ethical concerns.

heading: A ABLATION STUDY
text: Impact of BEL encoding, decoding, and loss functions: We propose multiple combinations of encoding, decoding, and loss functions that can be used with BEL. In Tables 3- 13, we show the effect of each combination of encoding, decoding, and loss function on the error of the model. Although general trends exist and some combinations perform consistently well across dataset, the optimal combination varies based on the dataset.
Impact of quantization and decoding functions: As discussed in Section 3, a real-valued label is quantized to a discrete value in {1, 2, ..., N}. In Table 14, we show the effect of increasing the number of quantization levels (N ) on the error for correlation-based decoding (DGEN, which returns a quantized prediction) and expected correlation-based decoding (DGEN-EX, which returns a continuous prediction). We vary the number of quantization levels from 64 to 256 and observe that there exists a tradeoff between reducing quantization error and using fewer classifiers. The error is lower for 128 quantization levels than it is for 256 as the improvement resulting from fewer binary classifiers is higher than the increase in quantization error. Moreover, the use of DGEN-EX consistently results in lower error compared to DGEN.
Effect of the number of training samples on BEL: As discussed in Section 4, the performance of different encoding functions varies depending on the availability of sufficient training data. In Table 15, we analyze the effect of the number of available training samples for both simple and complex encodings. As the number of training samples decreases, simpler encodings (U and J) perform better than more complex encodings (B1JDJ, B2JDJ, and Hex). Using a more complex encoding reduces the number of classifiers; however, it increases each classifier’s complexity (i.e. the number of bit transitions) and thus performs poorly with less training data.

heading: COFW

heading: WFLW

heading: AFLW
text: Effect of reflected binary conversion: As mentioned in Section 3.2, we use reflected binary to increase the distance between distant labels based on the design properties of suitable regression encodings we proposed. Table 16 shows the impact of using reflected binary conversion on error for facial landmark detection benchmarks. As shown in the table, the use of reflected binary significantly reduces the error.
Use of binary heamaps: Facial landmark detection approaches typically use heatmap regression. We also evaluate BEL-H-x, in which the real-valued heatmaps are converted to binary heatmaps with 8 quantization levels. Table 17 shows the impact of using binary heatmaps on error for facial landmark detection benchmarks. In BEL-H-x, each real-valued pixel of the target heatmap is converted to a binary code using eight quantization levels. For unary code, a 64× 64 real-valued heatmap of one facial landmark is converted to eight 64x64 binary heatmaps, resulting in 32, 768 (8×64×64) binary classifiers compared to 512 for BEL-U. We believe that training a high number of binary classifiers results in high error for BEL-H-x.
Hyperparameter θ: As shown in Figure 4b, we introduce a feature vector of size θ before the output layer. Figure 6 compares the decrease in the error for different encodings and θ values. We observe that more complex encodings that show lower correlation between the classifiers’ outputs benefit more from an increase in the value of θ, while a lower value of θ can be used for encodings with a high correlation between the classifiers’ outputs.
Effect of increasing the number of fully connected layers: For BEL, we propose to add a fully connected layer in the regressor to reduce the feature vector size to θ and thus decrease the number of parameters in the regressor. We perform an ablation study to study the impact of these added fully connected layers on direct regression, multiclass classification, and binary encoded labels. Table 18 provides the error (MAE or NME) for direct regression and multiclass classification with one or two fully connected layers after the feature extractor. Table 19 shows the impact of increasing the number of fully connected layers up to three on the error (MAE or NME) for different regression approaches. We observe that increasing the number of fully connected layers in direct regression and multiclass classification does not improve the accuracy for most benchmarks (possibly due to overparameterization). BEL with two fully connected layers outperforms direct regression and multiclass classification in both cases. Furthermore, even for a higher number of fully connected layers in BEL, different encoding functions result in the lowest error.
Training-validation set based evaluation Ideally, a validation set should be used for model selection. Hence we have reevaluated the benchmarks with 20% of the training data being used as a validation set to select the best model (i.e., which model is the best over multiple epochs).
Datasets used in benchmarks do not provide separate validation datasets. We use 80% of the training data for training and use validation error to select the best design parameters for BEL and the best model for all approaches. Since earlier works use 100% training data for the reported results and use test error for model selection, we have re-run specialized approaches (if possible), direct regression, and multiclass classification. It was not possible for us to re-run experiments for all specialized approaches due to resource constraints, and the comparison is conservative for many benchmarks since we use less data while selecting the best BEL approach than prior works used for model selection.
Table 20 compares different regression approaches for this evaluation setup. Note that the additional results do not diminish the effectiveness of BEL and BEL outperforms direct regression and multiclass classification for all benchmarks and specialized approaches for several benchmarks. Here, r is a scaling factor for the error probabilities.

heading: B EXPECTED ERROR DERIVATION
text: This section explains the expected error equations used to compare BEL-U and BEL-J in Section 3.1. We first explain the decoding function used for BEL-U and derive the relation between the expected error of labels and binary classifiers for BEL-U. Then, we explain the encoding/decoding functions and expected error relation for BEL-J.

heading: B.1 PRELIMINARIES
text: We compose binary-encoded labels as a quantization function q, an encoding function E , and a decoding function D. Given a sample i drawn from a dataset with minimum label a and maximum label b, let yi ∈ [a, b] represent the target label for that sample. Assuming uniform quantization, the range of target labels can be quantized using q : [a, b]→ {1, 2, ..., N − 1} through equation 7.
q(yi) = (yi − a) ∗ N − 1 b− a
(7)
We define the encoding function E : {1, 2, ..., N − 1} → {0, 1}M to convert a target quantized level Qi ∈ {1, 2, ..., N − 1} to a binary code Bi ∈ {0, 1}M . We further define the decoding function D : {0, 1}M → [a, b] to convert the predicted binary code B̂i to the predicted label ŷi. Although the decoding functions used in this analysis predict the quantized label and introduce quantization error, we do not include quantization error in the expected label error for our analysis and it is constant for both BEL-U and BEL-J. The expected value of absolute error between the target y and predicted labels ŷ is used for the analysis as typically mean absolute error is used as the evaluation metric in regression problems.
Let us denote the error probability of a binary classifier Ck used to predict bit k in a binary code Bi = E(n) as ek(n), where n is the target quantized label Qi. Then,
ek(n) = E(|b̂ki − bki |) = Pr(b̂ki = F ) for target label Qi = n and target binary code Bi = E(Qi)
(8)
where b̂ki = T indicate a correct binary classification by classifier C k (b̂ki == b k i ) for sample i and b̂ki = F indicat an incorrect binary classification by classifier C k (b̂ki 6= bki ) for sample i.

heading: B.2 EXPECTED ERROR FOR BEL-U
text: Encoding and decoding functions The encoding and decoding functions for BEL-U are defined as:
EBEL-U(Qi) = b1i , b2i , .., bN−2i , where b k i = { 1 k < Qi 0 Otherwise
(9)
DBEL-U(b̂1i , b̂2i , ..., b̂N−2i ) = N−2∑ k=1 b̂ki + 1 (10)
Expected error For target quantized labelQi = n (n ∈ {1, 2, ..., N−1}), ignoring the quantization error, the expected error between target yi and predicted label ŷi can be derived as:
E(|ŷi − yi|) = E (∣∣∣(N−2∑
k=1 b̂ki + 1 ) − (N−2∑ k=1 bki + 1 )∣∣∣)
= E (∣∣∣N−2∑
k=1
(b̂ki − bki ) ∣∣∣)
6 E (N−2∑ k=1 |b̂ki − bki | )
= N−2∑ k=1 E |b̂ki − bki |
= N−2∑ k=1 ek(n) (using equation 8)
(11)
For a uniform distribution of target labels in the range [1, N − 1], the expected error can be derived as:
E(|ŷ − y|) 6 1 N − 1 N−1∑ n=1 N−2∑ k=1 ek(n) (12)

heading: B.3 EXPECTED ERROR FOR BEL-J
text: Encoding and decoding functions For target quantized label Qi ∈ {1, 2, ..., N − 1}, BEL-J encoding requires N2 bits/binary classifiers. The encoding for BEL-J can be defined as:
EBEL-J(Qi) = b1i , b2i , .., b N 2 i ,where b k i = { 1 N2 −Qi < k 6 N −Qi 0 Otherwise
(13)
1
)
b̂ki , and Tc = N2
Similarly, the decoding functions for BEL-J can be defined as:
DBEL-J(B̂i) = T l(B̂i) + Tf(B̂i) + Tc where, T l(B̂i) = − max
k∈{1...N2 } kb̂ki
Tf(B̂i) = max k∈{1...N2 }
(N 2 − k + 1 ) b̂ki , T c = N 2
(14)
In equation 14, T l() finds the location of the last occurrence of “1” in the predicted binary code B̂i. Similarly, Tf() finds the location of the first occurrence of “1” in the binary code B̂i. Figure 7 gives examples of binary codes for label Qi ∈ {1, 2, ..., 7} and the corresponding values of the different terms in equation 14. For example, for label Qi = 3, the binary code is “0111”. Here, the last occurrence of “1” is at position 4, and T l = −4. Similarly, the first occurrence of “1” is at position 2, and Tf = (4 + 1)− 2 = 3.
Expected error For BEL-J code, binary classifiers (C1, C2, ..., C N2 ) are used. For a given input sample i, an error in any of the binary classifiers’ outputs (b̂1i , b̂ 2 i , ..., b̂ N 2 ) will result in an error between Tf(B̂i)/T l(B̂i) and Tf(Bi)/T l(Bi) in equation 14. We refer to Tf(B̂i) and T l(B̂i) as T̂ f i and T̂ li (predicted binary code), and Tf(Bi) and T l(Bi) as Tfi and T li (target binary code) for brevity. Expected value of the absolute error can be further expanded as:
E(|ŷi − yi|) = E(|T̂ f i + T̂ li + Tc− (Tfi + T li + Tc)|) = E(|(T̂ f i − Tfi) + (T̂ li − T li)|) 6 E(|T̂ f i − Tfi|+ |T̂ li − T li|) = E(|T̂ f i − Tfi|) + E(|T̂ li − T li|)
(15)
Thus, the sum of the expected error of Tf() and T l() is the upper bound of the label’s expected error. Further, we derive the relation between binary classifiers’ error probabilities and E(|T̂ f i − Tfi|) and E(|T̂ li − T li|).
We consider Qi = n,where1 6 n 6 N2 for our derivation. In such a case, Tfi = n and T li = − N 2 . However, as the code is symmetric around Qi = N2 , it can be shown that the derived equation for E |ŷi − yi| can be used for 1 6 Qi 6 N − 1.
1. Derivation of E |T̂ f i − Tfi|: As shown in equation 14, Tf() finds the location k of the first occurrence of “1” in the binary sequence. In the case of an erroneous binary sequence, the position of the first occurrence of “1” might shift, which results in an error between T̂ f i and Tfi. Figure 8 shows examples of the correct and erroneous outputs of classifiers for label Qi = n. For label Qi = n, bki = 0 for k ∈ {1, 2, ..., N2 − n} and b k i = 1 for k ∈ {N2 − n+ 1, N 2 − n+ 2, ..., N 2 }.
For case 1, error in a classifier Ck, k ∈ {1, 2, ..., N2 − n} is considered, where b k i = 0 and b̂ k i = 1. For k ∈ {1, 2, ..., N2 −n}, an error at classifier C k will result in erroneous T̂ f i only if all proceeding classifiers are correct, since if any of the proceeding classifier z is incorrect, i.e. b̂zi = 1, then the location of the first occurrence of “1” will be shifted to z, and any error in the following classifiers
will not affect the value of T̂ f . Such a case (b̂1i = T, b̂ 2 i = T, ..., b̂ k−1 i = T, b̂ k i = F, b̂ k+1 i = T/F, ..., b̂ N 2 i = T/F ) considers a total of 2 N 2 −k combinations out of 2 N 2 for k ∈ {1, 2, ..., N2 − n}. Assuming that the binary classifiers are mutually independent, the error value and the probability of this combination can be shown to be:
|T̂ f i − Tfi| = (N 2 − n+ 1− k ) (16)
Pr(b̂1i = T, b̂ 2 i = T, ..., b̂ k−1 i = T, b̂ k i = F ) = Pr(b̂ 1 = T )Pr(b̂2i = T )...P r(b̂ k−1 i = T )Pr(b̂ k = F )
= ( k−1∏ j=1 (1− ej(n)) ) · ek(n)
(17)
The above term considers combinations (b′1 = T, b′2 = T, ..., b′k−1 = T, b′k = F, b′k+1 = T/F, ..., b′
N 2 =T/F ) for k ∈ {1, 2, ..., N2 − n}, which constitutes to a total of ∑N 2 −n k=1 2 N 2 −k
combinations out of 2 N 2 .
For case 2, error in a classifier Ck, k ∈ {N2 − n+ 1, N 2 − n+ 2, ..., N 2 } is considered, where b k i = 1
and b̂ki = 0. We consider a combination (b̂ 1 i = T, b̂ 2 i = T, ..., b̂ N 2 −n i = T, b̂ N 2 −n+1 i = F, ..., b̂ k−1 i = F, b̂ki = T, b̂ k+1 i = T/F, ..., b̂ N 2 =T/F i ). For this case, the position of the first occurrence of “1” will be moved to k, which will result in erroneous T̂ f i. Such a case would cover 2 N 2 −k combinations out of 2 N 2 for k ∈ {N2 − n + 1, N 2 − n + 2, ..., N 2 }. The error value and the probability of this combination can be shown to be:
|T̂ f i − Tfi| = ( k − (N
2 − n+ 1)
) (18)
Pr(b̂1i = T, b̂ 2 i = T, ..., b̂ N 2 −n i = T, b̂ N 2 −n+1 i = F, ..., b̂ k−1 i = F, b̂
k i = T ) =( N2 −n∏
j=1 (1− ej(n)) ) · ( k−1∏ j=N2 −n+1 ej(n) ) · ( 1− ek(n) ) (19) The above term considers combinations (b̂1i = T, b̂ 2 i = T, ..., b̂ N 2 −n i = T, b̂ N 2 −n+1 i = F, ..., b̂ k−1 i = F, b̂ki = T, b̂ k+1 i = T/F, ..., b̂ N 2 =T/F i ), which constitutes to a total of ∑N 2
k=N2 −n+1 2
N 2 −k combina-
tions out of 2 N 2 for k ∈ {N2 − n+ 1, N 2 − n+ 2, ..., N 2 }.
Combining equation 16-equation 19, the expected value of |T̂ f i − Tfi| can be derived as: E(|T̂ f i − Tfi|) = N 2 −n∑ k=1 (N 2 − n+ 1− k ) · ( k−1∏ j=1 (1− ej(n)) ) · ek(n)
+
N 2∑
k=N2 −n+1
( k − (N
2 − n+ 1) ) · ( N2 −n∏ j=1 (1− ej(n)) ) · ( k−1∏ j=N2 −n+1 (ej(n)) ) · ( 1− ek(n) )
= N 2 −n∑ k=1 (N 2 − n+ 1− k ) · ek(n) · ( k−1∏ j=1 (1− ej(n)) ) +
N 2∑
k=N2 −n+1 ( k∏ j=N2 −n+1 ej(n) )
(20) The first term in equation 20 covers ∑N 2 −n k=1 2
N 2 −k combinations and the second term considers∑N
2
k=N2 −n+1 2
N 2 −k combinations. Adding one combination where all the classifiers are correct,
equation 20 considers all of the possible combinations 2 N 2 to find expected value of |T̂ f i − Tfi|.
2. Derivation of E |T̂ li − T li|: As shown in equation 14, T l() finds the location k of the last occurrence of “1” in the binary sequence. In the case of an erroneous binary sequence, the position of the last occurrence of “1” might shift, which results in an erroneous value of T̂ li. Figure 9 shows examples of correct and erroneous outputs of classifiers for label Qi = n.
For case 1, an error in a classifier Ck, k ∈ {N2 − n + 1, N 2 − n + 2, ..., N 2 } is considered, where
bki = 1 and b̂ k i = 0. We consider a combination (b̂ N 2 i = F, b̂ N 2 −1 i = F, ..., b̂ k+1 i = F, b̂ k i = T, b̂ k−1 i = T/F, ..., b̂1i = T/F ). For this case, position of the last occurrence of “1” will be moved to k, which will result in erroneous T̂ li. Such a case would cover 2k−1 combinations out of 2 N 2 . The error value and the probability of this combination can be shown to be:
|T̂ li − T li| = (N 2 − k )
(21)
Pr(b̂ N 2 i = F, b̂ N 2 −1 i = F, ..., b̂ k+1 i = F, b̂ k i = T ) = ( N2∏ j=k+1 ej(n) ) · (1− ek(n)) (22)
The above term considers combinations (b̂ N 2 i = F, b̂ N 2 −1 i = F, ..., b̂ k+1 i = F, b̂ k i = T, b̂ k−1 i = T/F, ..., b̂1i = T/F ) for k ∈ {N2 − n + 1, N 2 − n + 2, ...,
N 2 }, which constitutes to a total of∑N
2 k=N2 −n+1 2k−1 combinations out of 2 N 2 .
For case 2, an error in a classifier Ck, k ∈ {1, 2, ..., N2 − n} is considered, where b k i = 0 and b̂ k i = 1. We consider a combination (b̂ N 2 i = F, ..., b̂ N 2 −n+1 i = F, b̂ N 2 −n i = T, ..., b̂ k+1 i = T, b̂ k i = F, b̂ k−1 i = T/F, ..., b̂1i = T/F ). For this case, position of the last occurrence of “1” will be moved to k, which will result in erroneous T̂ li. Such a case would cover 2k−1 combinations out of 2 N 2 . The error value and the probability of this combination can be shown to be:
|T̂ li − T li| = (N 2 − k )
(23)
Pr(b̂ N 2 i = F, ..., b̂ N 2 −n+1 i = F, b̂ N 2 −n i = T, ..., b̂ k+1 i = T, b̂
k i = F ) =( N2∏
j=N2 −n+1
ej(n) ) · ( N2 −n∏ j=k+1 (1− ej(n)) ) · (ek(n))
(24)
The above term considers combinations (b̂ N 2 i = F, ..., b̂ N 2 −n+1 i = F, b̂ N 2 −n i = T, ..., b̂ k+1 i = T, b̂ki = F, b̂ k−1 i = T/F, ..., b̂
1 i = T/F ) for k ∈ {1, 2, ..., N2 − n}, which constitutes to a total of∑N
2 −n k=1 2 k−1 combinations out of 2 N 2 .
Combining equation 21-equation 24, the expected value of |T̂ li − T li| can be derived as:
E(|T̂ li − T li|) = N 2∑
k=N2 −n+1
(N 2 − k ) · ( N2∏ j=k+1 ej(n) ) · (1− ek(n))
+ N 2 −n∑ k=1 (N 2 − k ) · ( N2∏ j=N2 −n+1 ej(n) ) · ( N2 −n∏ j=k+1 (1− ej(n)) ) · (ek(n))
=
N 2∑
k=N2 −n+1
( N2∏ j=k ej(n) ) + ( N2∏ j=N2 −n+1 ej(n) ) · N 2 −n∑ k=1 ( N2 −n∏ j=k (1− ej(n)) )
(25)
The first term in equation 25 covers ∑N 2
k=N2 −n+1 2k−1 combinations and the second term considers ∑N
2 −n k=1 2 k−1 combinations. Adding one combination where all the classifiers are correct, equation 25 considers all of the possible combinations 2 N 2 to find expected value of |T̂ li − T li|.
Combining equation 15, equation 20, and equation 25, the expected value of average error in terms of classifiers’ error probabilities can de derived as:
E(ŷi−yi) 6 N 2 −n∑ k=1 (N 2 −n+1−k ) ·ek(n) · ( k−1∏ j=1 (1−ej(n)) ) + N 2∑
k=N2 −n+1 ( k∏ j=N2 −n+1 ej(n) )
+
N 2∑
k=N2 −n+1
( N2∏ j=k ej(n) ) + ( N2∏ j=N2 −n+1 ej(n) ) · N 2 −n∑ k=1 ( N2 −n∏ j=k (1− ej(n)) ) (26)
As the binary code is symmetric around N2 as shown in Figure 7, the expected errors for label yi ∈ [1, N2 ] can be mirrored to find expected errors for label yi ∈ [ N 2 , N − 1]. For a uniform distribution of target labels in the range [1, N − 1], the expected error can be derived as:
E(ŷ−y) 6 1 N − 1 N−1∑ n=1 [ N2 −n∑ k=1 (N 2 −n+1−k ) ·ek(n)· ( k−1∏ j=1 (1−ej(n)) ) + N 2∑
k=N2 −n+1 ( k∏ j=N2 −n+1 ej(n) )
+
N 2∑
k=N2 −n+1
( N2∏ j=k ej(n) ) + ( N2∏ j=N2 −n+1 ej(n) ) · N 2 −n∑ k=1 ( N2 −n∏ j=k (1− ej(n)) )] (27)
We also verify the equation by comparing the expected value of error based on equation 26 for Qi ∈ 1, 2, ..., N − 1 with the expected error calculated by 100, 000 random samples of binary sequences for the same error probabilities ek(n). Figure 10 compares the expected error from equation 26 and measured from statistical samples, and validates error upper bounds calculated using equation 26 and equation 27.

heading: C ERROR PROBABILITY OF CLASSIFIERS
text: It is known that the error/misclassification probability ek(n) of a classifier tends to increase as the target label value n is closer to the classifier’s decision boundaries Cardoso & Pinto da Costa (2007). We approximate ek(y) for a classifier Ck with t bit transitions as a linear combination of t Gaussian distributions. Here, each Gaussian term is centered around a bit transition. Figure 11 shows the empirically observed error probability distributions for different classifiers trained for different combinations of network and dataset. We also show the approximate error probability distribution using a linear combination of Gaussian distributions. Here r is a scalar multiplied with probability density of gaussian distribution and σ2 is the variance (equation 3 and equation 4).

heading: D EXPERIMENTAL METHODOLOGY
text: All experiments are conducted on a Linux machine with an Intel i9-9900X processor and a Nvidia RTX 2080 Ti GPU with 11GB of memory. Our code is implemented using Python 3.8.3 with Pytorch 1.5.1 using CUDA 10.2. Our evaluation is averaged over 5 training runs with separate seeds.

heading: D.1 HEAD POSE ESTIMATION
text: Head pose estimation aims to find a human head’s pose in terms of three angles: yaw, pitch, and roll. In this work, we consider landmark-free 2D head pose estimation.
Datasets: We follow the evaluation setting of Hopenet (Ruiz et al., 2018) and FSA-Net (Yang et al., 2019) and use two evaluation protocols with three widely used datasets: 300W-LP (Zhu et al., 2016), BIWI (Fanelli et al., 2013), and AFLW2000 (Zhu et al., 2016). We use two evaluation protocols.
Protocol 1: In this setting, the synthetic 300W-LP dataset is used for training, consisting of 122, 450 samples. The trained network is tested on a real-world AFLW2000 dataset. Yaw, pitch, and roll angles are in the range [−99, 99] for both datasets. Angles in range [−180◦,180◦] with 360 quantization levels are considered for BEL.
Protocol 2: BIWI dataset is used for training and evaluation in this protocol. BIWI dataset consists of 24 videos of 20 subjects with total 15, 128 frames. Three random splits of 70%-30% images are used for training and evaluation. For the BIWI dataset, the yaw angle is in the range [−75, 75], the pitch is in the range [−65, 85], and the roll angle is in the range [−55, 45]. 150 quantization levels are used for yaw and pitch, and 100 quantization levels are used for roll.
Network Architecture and training parameters: We evaluate our approach on two models: ResNet-50 and RAFA-Net. With ResNet-50 (Direct regression and BEL), two runs with different random seeds for each combination of learning rate {0.001, 0.0001, 0.00001} and batch size {8, 16} are used for hyperparameter tuning. For data augmentation, images are loosely cropped around the center in the training and testing datasets with random flipping. With RAFA-Net, we use the training parameters used in Behera et al. (2021). For data augmentation, we use the approach described in RAFA-Net Behera et al. (2021).
We refer to Protocol 1 evaluated with ResNet-50 as HPE1, Protocol 1 evaluated with RAFA-Net as HPE3, Protocol 2 evaluated with ResNet-50 as HPE2, and Protocol 2 evaluated with RAFA-Net as HPE4. Table 21 provides a summary of the training parameters used with protocol 1. Table 22 provides a summary of the training parameters used with protocol 2.
Related Work Existing approaches for head pose estimation include stage-wise soft regression (Rothe et al., 2015; Yang et al., 2019), a combination of classification and regression (Mukherjee & Robertson, 2015; Ruiz et al., 2018), and ordinal regression (Hsu et al., 2019). SSR-Net (Rothe et al., 2015) proposes the use of stage-wise soft regression to use the softmax values of classification output to refine the label. FSA-Net (Yang et al., 2019) proposes extending stage-wise estimation to head pose estimation using feature aggregation. HopeNet (Ruiz et al., 2018) uses a combination of classification and regression loss to train a model for head pose estimation whereas QuatNet (Hsu et al., 2019) proposes a combination of L2 loss and a custom ordinal regression loss. RAFA-Net (Behera et al., 2021) uses an attention based approach for feature extraction with direct regression.
We compare BEL with the performance of related work in Table 23 and Table 24. 95% confidence intervals are given.

heading: D.2 FACIAL LANDMARK DETECTION
text: Facial landmark detection is a problem of detecting the (x, y) coordinates of keypoints in a given face image.
Datasets We use the COFW (Burgos-Artizzu et al., 2013), 300W (Sagonas et al., 2013), WFLW (Wu et al., 2018), and AFLW (Köstinger et al., 2011) datasets with data augmentation and evaluation protocols described in (Wang et al., 2020). Normalized mean error (NME) and failure rate (f@10%) are used as evaluation metrics. . Inter-ocular distance normalization is used for the COFW, 300W, and WFLW datasets and bounding box-based normalization is used for the AFLW dataset. Data augmentation is performed by random flipping, 0.75− 1.25 scaling, and ±30 degrees in-plane rotation for all the datasets. We use 256 quantization levels for binary-encoded labels.
COFW The COFW dataset (Burgos-Artizzu et al., 2013) consists of 1, 345 training and 507 testing images. Each image is annotated with 29 facial landmarks.
300W This dataset is a combination of HELEN, LFPW, AFW, XM2VTS, and IBUG datasets. Each image is annotated with 68 facial landmarks. The training dataset consists of 3, 148 images. We evaluate the trained model on four test sets: full test set with 689 images, common subset with 554 images from HELEN and LFPW, challenging subset with 135 images from IBUG, and the official test set with 300 indoor and 300 outdoor images.
WFLW WFLW dataset consists of 7, 500 training images where each image is annotated with 98 facial landmarks. Full test dataset consists of 2, 500 images. We use test subsets: large pose (326 images), expression (314 images), illumination (698 images), make-up (206 images), occlusion (736 images), and blur (773 images).
AFLW Each image has 19 annotated facial key points in this dataset. AFLW dataset consists of 20, 000 training images where each image is annotated with 19 facial landmarks. The full test dataset consists of 4, 836 images, and the frontal test set consists of 1, 314 images.
Network Architecture and training parameters: We evaluate BEL by applying it on HRNetV2W18. HRNetV2-W18 feature extractor’s output is 240 channels of size 64 × 64. For heatmap regression, a 1× 1 convolution is used to get P heatmaps of size 64× 64, where P is the number of landmarks. Since BEL-x predicts (x, y) coordinates directly we modify the architecture of HRNetV2-W18 to support direct prediction of landmarks. Figure 12 shows the modified architecture of HRNetV2-W18 for BEL-x.
The state-of-the-art approach with HRNetV2-W18 (Wang et al., 2020) uses heatmap regression, which minimizes the pixel-level loss between the predicted and target heatmaps. We evaluate the applicability of BELon heatmap regression in Appendix A. In contrast, BEL-x predicts (x, y) coordinates directly with 256 quantization levels.
We use two runs with different random seeds to decide the learning rate. We consider learning rates {0.0003, 0.0005, 0.0007} and θ ∈ 10, 30. The value of θ is selected for NME, but Table 25 provides a summary of all the training parameters. We refer to HRNetV2-W18 evaluated on COFW as FLD1, on 300W as FLD2, on WFLW as FLD3 and on AFLW as FLD4.
Related Work Facial landmark detection is an extensively studied problem used for facial analysis and modeling. Common regression approaches for this tasks includes regression using MSE loss (Xiong & De la Torre, 2013; Lv et al., 2017), cascaded regression (Miao et al., 2018; Tzimiropoulos, 2015; Zhu et al., 2016; Sun et al., 2013b), and coarse-to-fine regression (Sun et al., 2013a; Shizhan Zhu et al., 2015; Zhang et al., 2014). State-of-the-art methods for this task learn heatmaps by regression to find facial landmarks. SAN (Dong et al., 2018) augments training data using temporal information and GAN-generated faces. DVLN (Wu & Yang, 2017), CFSS (Shizhan Zhu et al., 2015), LAB (Wu et al., 2018), DSRN (Miao et al., 2018) take advantage of correlations between facial landmarks. DAN (Kowalski et al., 2017) introduces a progressive refinement approach using predicted landmark heatmaps. LAB (Wu et al., 2018) also exploits extra boundary information to improve the accuracy. LUVLi (Kumar et al., 2020) proposes a landmark’s location, uncertainty, and
visibility likelihood-based loss. Bulat & Tzimiropoulos (2016) proposes the use of binary heatmaps with pixel-wise binary cross-entropy loss. AWing (Wang et al., 2019) proposes adapted wing loss to improve the accuracy of heatmap regression. AnchorFace (Xu et al., 2020) demonstrates that anchoring facial landmarks on templates improves regression performance for large poses. HRNet (Wang et al., 2020) proposes a CNN architecture to maintain high-resolution representations across the network, and uses heatmap regression. The target heatmap is generated by assuming a Gaussian distribution around the landmark location.
We compare BEL with related work in Table 26- 28. 95% confidence intervals are provided.

heading: D.3 AGE ESTIMATION
text: Age estimation aims to predict the age given an image of a human head.
Datasets We use the MORPH-II (Ricanek & Tesafaye, 2006) and AFAD (Niu et al., 2016) datasets for our evaluation. Cumulative Score (CS) and MAE are used as evaluation metrics. We preprocess the MORPH-II dataset by aligning images first along the average eye position (Raschka, 2018), then by re-aligning so that the tip of the nose is in the center of each image. We do not preprocess the AFAD dataset as faces are already centered. Afterwards, face images are resized to 256× 256× 3 and randomly cropped to 224× 224× 3 for training. For testing, a center crop of 224× 224× 3 is taken.
MORPH-II: This dataset consists of 55,608 face images with age labels between 16 and 70. The dataset is randomly divided into 39,617 training, 4,398 validation, and 11,001 testing images.
AFAD: This dataset consists of 164,432 Asian facial images and age labels between 15 and 40. The dataset is randomly divided into 118,492 training, 13,166 validation, and 32,763 testing images.
Network architecture and training parameters: We evaluate our approach on ResNet50. We perform two runs with different random seeds to determine the learning rate between [0.00001, 0.0001, 0.001] and use a batch size of 64 for all experiments. We use ImageNet pretrained weights to initialize the network. Full training parameters are described in Table 30. We refer to our evaluation on MORPH-II as AE1 and AFAD as AE2.
Related Work Existing approaches for age estimation include ordinal regression (Niu et al., 2016; Cao et al., 2020), soft regression (Rothe et al., 2015), and expected value ordinal regression (Pan et al., 2018; Gao et al., 2018). OR-CNN (Niu et al., 2016) proposed the use of ordinal regression via binary classification to predict the label. CORAL-CNN (Cao et al., 2020) refined this approach by enforcing the ordinality of the model output. SSR-Net (Rothe et al., 2015) proposed the use of stage-wise soft regression using the softmax of the classification output to refine the predicted label. MV-Loss (Pan et al., 2018) extended the soft regression approach by penalizing the output of the model based on the variance of the age distribution, while DLDL (Gao et al., 2018) proposed to use the KL-divergence between the softmax output and a generated label distribution to train a model.
We compare BEL with related work in Table 31 and Table 32. 95% confidence intervals are provided.
Uses custom data augmentation for the training

heading: D.4 END-TO-END SELF DRIVING
text: We evaluate our approach on the NVIDIA PilotNet dataset and PilotNet model for end-to-end autonomous driving (Bojarski et al., 2016). In this task, the steering wheel’s next angle is predicted from an image of the road. We refer to these experiments as PN.
Dataset We use a driving dataset consisting of 45,500 images taken around Rancho Palos Verdes and San Pedro, California (Chen). We crop images to 256× 70× 3 then resize them to 200× 66× 3. We randomly vary the brightness of the image between [0.2×, 1.5×], randomly flip images, and make random minor perturbations on the steering direction. We use θ = 10 with 670 quantization levels for BEL.
Training parameters We perform two runs with different random seeds to determine the learning rate between [0.00001, 0.0001, 0.001] and use a batch size of 64 for all experiments. Full training parameters are described in Table 33.
Related Work End-to-end autonomous driving is a novel task that has become increasingly relevant due to the rise of self-driving vehicles. The autonomous driving model’s task is to predict the future driving angle based on a forward-facing image from the perspective of the vehicle. PilotNet (Bojarski et al., 2017) used a small, application-specific network to succeed within the time constraints of autonomous driving.
We compare BEL with the baseline PilotNet architecture in Table 34. 95% confidence intervals are provided

Review summaries:
The paper tackles regression problems using an error-correcting code (ECOC) approach, i.e., reducing a *regression* problem into multiple binary *classification* problems. ECOC have been so far mainly used for classification tasks, and their application to regression problems in this paper is elegant and seems novel. The paper examines several combinations of (encoding method $\times$ decoding method $\times$ loss function) on several (deep) regression tasks and reports significant improvements over existing methods. Some brief analytic discussion is made.

The paper describes a general framework to solve a regression problem as a binary classification. The labels (targets) are binary encoded and hence the regression problem is solved using a set of binary classifiers. Since the process is dependent by the type of Encoding and Decoding used, the authors compare different type of configurations showing, empirically, the effect of different design parameters on the final accuracy.

This paper analyzes regression problems in which continuous targets are predicted using a set of binary classifiers. There is a large design space including quantization, encoding, decoding, and loss functions. An expected value-based function (D^{GEN-EX}) is proposed as a new decoding function.  The authors apply various combinations of these functions and compare them theoretically and experimentally. It is suggested that no single combination outperforms the others, but the proposed BEL approach has a potential to improve the prediction accuracy of continuous regression problems. 

This study presents a label encoding method (binary encoded labels) to transform a regression problem into multiple binary classifiation problems. The authors designed encoding and decoding functions to encode a real-value into multiple classes and decode the classes into the real value. The proposed demonstrated improved performance on benchmark datasets.