Label Encoding for Regression Networks

Deep neural networks are used for a wide range of regression problems and are typically trained by minimizing the squared or absolute error of output labels . Solving a regression problem with a set of binary classifiers can improve the accuracy and also utilizes well-studied binary classification algorithms . However , a limited design space of regression by binary classification has been studied by prior works . This work introduces a novel approach using binary-encoded labels ( BEL ) to generalize the application of binary classification to regression , and a taxonomy identifying the key design aspects of such formulation . Of these design aspects , we primarily explore and identify the desirable properties of suitable encoding and decoding functions used for the conversion between real-valued and binary-encoded labels based on theoretical and empirical study . BEL can be combined with off-the-shelf task-specific feature extractors and trained end-toend . We further propose a series of sample encoding , decoding , and training loss functions applicable in this domain and demonstrate that our generic regression approach results in lower error than direct regression and specialized approaches while being suitable for a diverse set of regression problems , network architectures , and evaluation metrics . 1 INTRODUCTION . Deep regression networks , in which a continuous output is predicted for a given input , are traditionally trained by minimizing squared/absolute error of a logit , which we refer to as direct regression . More recently , several works have proposed specialized regression approaches for regression problems such as head pose estimation , age estimation , and facial landmark estimation , and shown significant accuracy improvements over direct regression . However , developing task-agnostic generic approaches to improve accuracy for regression is desirable . A regression problem can be posed as a set of binary classification sub-problems . A similar approach has been applied to other domains such as ordinal regression ( Li & Lin , 2006 ) and multiclass classification ( Dietterich & Bakiri , 1995 ) . Such formulation allows the use of well-studied binary classification approaches . Further , new generalization bounds for ordinal regression or multiclass classification can be derived from the known generalization bounds of binary classification . This reduces the efforts for design , implementation , and theoretical analysis significantly ( Li & Lin , 2006 ) . Dietterich & Bakiri ( 1995 ) demonstrated that posing multiclass classification as a set of binary classification problems can increase error tolerance and improve accuracy . However , the proposed approaches for multiclass classification do not apply to regression due to the differences in task objective and properties of the error probability distribution of classifiers ( Section 2 ) . On the other hand , prior works on ordinal regression have explored the application of binary classifiers in a more restricted way which limits its application to a wide range of complex regression problems ( Section 2 ) . There exists a lack of a generic framework that unifies possible formulations for using binary classification to solve regression . In this work , we propose binary-encoded labels ( BEL ) which improves regression by generalizing the application of binary classification . In BEL , a target label is quantized and converted to a binary code of length M , and M binary classifiers are then used to learn binary-encoded labels . An encoding function is introduced to convert the target label to a binary code , and a decoding function is introduced to decode the output of binary classifiers to the predicted label . BEL allows using an adjustable number of binary classifiers depending upon the quantization , encoding , and decoding functions . BEL opens possible avenues to improve the accuracy of regression problems with a large design space of the quantization function , encoding/decoding functions , and optimization algorithm . We focus on the encoding and decoding functions and theoretically study the relations between the absolute error of label and binary classifiers ’ errors for sample encoding and decoding functions . This analysis demonstrates the impact of binary classifiers ’ error distribution over the numeric range of target labels on the suitability of different encoding and decoding functions . Based on our analysis and empirically observed binary classifiers ’ error distribution , we propose properties of suitable encoding functions for regression and explore various encoding functions on a wide range of tasks . We also propose an expected correlation-based decoding function for regression that can effectively reduce the quantization error introduced by the encoding . A deep regression network consists of a feature extractor and a regressor and is trained end-toend . Our proposed regression approach ( BEL ) can be combined with off-the-shelf task-specific feature extractors by increasing the regressor ’ s output logits . We find that the correlation between multiple binary classifiers ’ outputs can be exploited to reduce the size of the feature vector and consequently reduce the number of parameters in the regressor . We explore the use of decoding functions for training loss formulation and evaluate the use of the binary cross-entropy , cross-entropy , and squared/absolute error loss functions for BEL . We evaluate our approach on four complex regression problems : head pose estimation , facial landmark detection , age estimation , and end-to-end autonomous driving . We make the following contributions in this work : • We propose binary-encoded labels for regression and introduce a general framework and a taxonomy for the design aspects of regression by binary classification . We propose desirable properties of encoding and decoding functions suitable for regression problems . • We present a series of suitable encoding , decoding , and loss functions for regression with BEL . We present an end-to-end learning approach and regression layer architecture for BEL . We combine BEL with task-specific feature extractors for four tasks and evaluate multiple encoding , decoding , and loss functions . BEL outperforms direct regression for all the problems and specialized approaches for several tasks . • We theoretically and empirically demonstrate the effect of different design parameters on the accuracy , how it varies across different tasks , datasets , and network architectures , and provide preliminary insights and motivation for further study . 2 RELATED WORK . Binary classification for regression : Prior works have proposed binary classification-based approaches for ordinal regression ( Crammer & Singer , 2001 ; Chu & Keerthi , 2005 ; Li & Lin , 2006 ) . Ordinal regression is a class of supervised learning problems , where the samples are labeled by a rank that belongs to an ordinal scale . Ordinal regression approaches can be applied to regression by discretizing the numeric range of the real-valued labels ( Fu et al. , 2018 ; Berg et al . ; Niu et al. , 2016 ) . In the existing works on ordinal regression by binary classification , N − 1 binary classifiers are used for target labels ∈ { 1 , 2 , ... , N } , where classifier-k predicts if the label is greater than k or not for a given input . Li & Lin ( 2006 ) provided a reduction framework and generalization bound for the same . However , the proposed binary classification formulation is restricted . It requires several binary classifiers if the numeric range of output is extensive , whereas reducing the number of classifiers by using fewer quantization levels increases quantization error . Thus , a more generalized approach for using binary classification for regression is desirable to allow flexibility in the design of classifiers . Binary classification for multiclass classification : Dietterich & Bakiri ( 1995 ) proposed the use of error-correcting output codes ( ECOC ) to convert a multiclass classification to a set of binary classification sub-problems . This improves the accuracy of the network as it introduces tolerance to binary classifiers ’ errors depending upon the hamming distance ( i.e. , number of bits changed between two binary strings ) between two codes . Allwein et al . ( 2001 ) provided a unifying framework and multiclass loss bounds in terms of binary classification loss . More recent works have also used Hadamard code , a widely used error-correcting code ( Song et al. , 2021 ; Verma & Swami , 2019 ) . Other works have focused on the problem of extreme classification and the use of ECOC and compact codes ( sublinear increase in the length of codes with the number of classes ) for a classification problem with a large number of classes ( Cissé et al. , 2012 ; Evron et al. , 2018 ) . However , the proposed encoding and decoding approaches do not consider the ordinality of labels for regression . Further , the binary classifiers possess distinct error probability distribution properties for regression problems as observed empirically ( Section 3.1 ) , which can be exploited to design codes suitable for regression . Multiclass classification and ordinal regression by binary classification can be viewed as special cases falling under the BEL framework . As shown in Section 4 , other BEL designs yield improvements in accuracy over these approaches . Task-specific regression techniques are well explored as summarized below ( see also Appendix D. While effective , task-specific approaches lack generality by design . Head pose estimation : SSR-Net ( Rothe et al. , 2015 ) and FSA-Net ( Yang et al. , 2019 ) used softmax to refine labels . HopeNet ( Ruiz et al. , 2018 ) used a combination of classification and regression loss . QuatNet ( Hsu et al. , 2019 ) used a combination of regression and ordinal regression loss . Facial landmark detection : Wang et al . ( 2020 ) minimize L2 loss between predicted and target 2D heatmaps with the latter formed using small variance Gaussians centered on ground truth landmarks . AWing ( Wang et al. , 2019 ) modified loss for different pixels in the heatmap . LUVLi ( Kumar et al. , 2020 ) proposed a landmark ’ s location , uncertainty , and visibility likelihood-based loss . Bulat & Tzimiropoulos ( 2016 ) used binary heatmaps with pixel-wise binary cross-entropy loss . Age estimation : OR-CNN ( Niu et al. , 2016 ) and CORAL-CNN ( Cao et al. , 2020 ) used ordinal regression via binary classification . MV-Loss ( Pan et al. , 2018 ) proposed to penalize the model output based on the age distribution ’ s variance , while Gao et al . ( 2018 ) proposed to use the KL-divergence between the softmax output and a generated label distribution for training . 3 BINARY-ENCODED LABELS FOR REGRESSION ( BEL ) . We consider regression problems where the goal is to minimize the error between real-valued target labels yi and predicted labels ŷi , over a set of training samples i . We transform this problem to a set of binary classification sub-problems by converting a real-valued label to a binary code . Figure 1 shows the training and inference flow for BEL . The red-colored blocks highlight functions that vary under BEL . A real-valued label yi ∈ R is quantized to level Qi ∈ { 1 , 2 , ... , N } . The quantized label is then converted to a binary sequence Bi ∈ { 0 , 1 } M using an encoding function E . The binary-encoded labels Bi are used to train M classifiers . In the prediction phase , the classifiers predict a binary code B̂i ∈ { 0 , 1 } M for an input xi . The predicted binary code ( B̂i ) or the real-valued confidence vector of classifiers ’ prediction ( Ẑi ) is then decoded to a predicted label ŷi ∈ R using a decoding function D. We explore decoding functions that yield either quantized or continuous predicted outputs . For the latter , we propose an expected correlation-based decoding function that decodes classifiers ’ output to a continuous value , reducing the quantization error ( Section 3.3 ) . There are five key design parameters when applying BEL : quantization , encoding , decoding , regressor network architecture , and training loss formulation yielding a large design space . In this work we consider only uniform quantization while leaving nonuniform quantization ( Fu et al. , 2018 ) to future work . Section 3.2 and 3.3 explore the characteristics of suitable encoding , decoding , and training loss functions . Section 3.4 explore the impact of regression network architecture . We find varying any of these aspects can improve accuracy . While BEL provides a framework and some codes appear generally better than others , the most suitable BEL code to employ appears to vary across task , dataset , and network architecture , as we show both theoretically ( Section 3.1 ) and empirically ( Section 4 ) .

Summary: This paper analyzes regression problems in which continuous targets are predicted using a set of binary classifiers. There is a large design space including quantization, encoding, decoding, and loss functions. An expected value-based function (D^{GEN-EX}) is proposed as a new decoding function.  The authors apply various combinations of these functions and compare them theoretically and experimentally. It is suggested that no single combination outperforms the others, but the proposed BEL approach has a potential to improve the prediction accuracy of continuous regression problems.