Label Encoding for Regression Networks

Deep neural networks are used for a wide range of regression problems and are typically trained by minimizing the squared or absolute error of output labels . Solving a regression problem with a set of binary classifiers can improve the accuracy and also utilizes well-studied binary classification algorithms . However , a limited design space of regression by binary classification has been studied by prior works . This work introduces a novel approach using binary-encoded labels ( BEL ) to generalize the application of binary classification to regression , and a taxonomy identifying the key design aspects of such formulation . Of these design aspects , we primarily explore and identify the desirable properties of suitable encoding and decoding functions used for the conversion between real-valued and binary-encoded labels based on theoretical and empirical study . BEL can be combined with off-the-shelf task-specific feature extractors and trained end-toend . We further propose a series of sample encoding , decoding , and training loss functions applicable in this domain and demonstrate that our generic regression approach results in lower error than direct regression and specialized approaches while being suitable for a diverse set of regression problems , network architectures , and evaluation metrics . 1 INTRODUCTION . Deep regression networks , in which a continuous output is predicted for a given input , are traditionally trained by minimizing squared/absolute error of a logit , which we refer to as direct regression . More recently , several works have proposed specialized regression approaches for regression problems such as head pose estimation , age estimation , and facial landmark estimation , and shown significant accuracy improvements over direct regression . However , developing task-agnostic generic approaches to improve accuracy for regression is desirable . A regression problem can be posed as a set of binary classification sub-problems . A similar approach has been applied to other domains such as ordinal regression ( Li & Lin , 2006 ) and multiclass classification ( Dietterich & Bakiri , 1995 ) . Such formulation allows the use of well-studied binary classification approaches . Further , new generalization bounds for ordinal regression or multiclass classification can be derived from the known generalization bounds of binary classification . This reduces the efforts for design , implementation , and theoretical analysis significantly ( Li & Lin , 2006 ) . Dietterich & Bakiri ( 1995 ) demonstrated that posing multiclass classification as a set of binary classification problems can increase error tolerance and improve accuracy . However , the proposed approaches for multiclass classification do not apply to regression due to the differences in task objective and properties of the error probability distribution of classifiers ( Section 2 ) . On the other hand , prior works on ordinal regression have explored the application of binary classifiers in a more restricted way which limits its application to a wide range of complex regression problems ( Section 2 ) . There exists a lack of a generic framework that unifies possible formulations for using binary classification to solve regression . In this work , we propose binary-encoded labels ( BEL ) which improves regression by generalizing the application of binary classification . In BEL , a target label is quantized and converted to a binary code of length M , and M binary classifiers are then used to learn binary-encoded labels . An encoding function is introduced to convert the target label to a binary code , and a decoding function is introduced to decode the output of binary classifiers to the predicted label . BEL allows using an adjustable number of binary classifiers depending upon the quantization , encoding , and decoding functions . BEL opens possible avenues to improve the accuracy of regression problems with a large design space of the quantization function , encoding/decoding functions , and optimization algorithm . We focus on the encoding and decoding functions and theoretically study the relations between the absolute error of label and binary classifiers ’ errors for sample encoding and decoding functions . This analysis demonstrates the impact of binary classifiers ’ error distribution over the numeric range of target labels on the suitability of different encoding and decoding functions . Based on our analysis and empirically observed binary classifiers ’ error distribution , we propose properties of suitable encoding functions for regression and explore various encoding functions on a wide range of tasks . We also propose an expected correlation-based decoding function for regression that can effectively reduce the quantization error introduced by the encoding . A deep regression network consists of a feature extractor and a regressor and is trained end-toend . Our proposed regression approach ( BEL ) can be combined with off-the-shelf task-specific feature extractors by increasing the regressor ’ s output logits . We find that the correlation between multiple binary classifiers ’ outputs can be exploited to reduce the size of the feature vector and consequently reduce the number of parameters in the regressor . We explore the use of decoding functions for training loss formulation and evaluate the use of the binary cross-entropy , cross-entropy , and squared/absolute error loss functions for BEL . We evaluate our approach on four complex regression problems : head pose estimation , facial landmark detection , age estimation , and end-to-end autonomous driving . We make the following contributions in this work : • We propose binary-encoded labels for regression and introduce a general framework and a taxonomy for the design aspects of regression by binary classification . We propose desirable properties of encoding and decoding functions suitable for regression problems . • We present a series of suitable encoding , decoding , and loss functions for regression with BEL . We present an end-to-end learning approach and regression layer architecture for BEL . We combine BEL with task-specific feature extractors for four tasks and evaluate multiple encoding , decoding , and loss functions . BEL outperforms direct regression for all the problems and specialized approaches for several tasks . • We theoretically and empirically demonstrate the effect of different design parameters on the accuracy , how it varies across different tasks , datasets , and network architectures , and provide preliminary insights and motivation for further study . 2 RELATED WORK . Binary classification for regression : Prior works have proposed binary classification-based approaches for ordinal regression ( Crammer & Singer , 2001 ; Chu & Keerthi , 2005 ; Li & Lin , 2006 ) . Ordinal regression is a class of supervised learning problems , where the samples are labeled by a rank that belongs to an ordinal scale . Ordinal regression approaches can be applied to regression by discretizing the numeric range of the real-valued labels ( Fu et al. , 2018 ; Berg et al . ; Niu et al. , 2016 ) . In the existing works on ordinal regression by binary classification , N − 1 binary classifiers are used for target labels ∈ { 1 , 2 , ... , N } , where classifier-k predicts if the label is greater than k or not for a given input . Li & Lin ( 2006 ) provided a reduction framework and generalization bound for the same . However , the proposed binary classification formulation is restricted . It requires several binary classifiers if the numeric range of output is extensive , whereas reducing the number of classifiers by using fewer quantization levels increases quantization error . Thus , a more generalized approach for using binary classification for regression is desirable to allow flexibility in the design of classifiers . Binary classification for multiclass classification : Dietterich & Bakiri ( 1995 ) proposed the use of error-correcting output codes ( ECOC ) to convert a multiclass classification to a set of binary classification sub-problems . This improves the accuracy of the network as it introduces tolerance to binary classifiers ’ errors depending upon the hamming distance ( i.e. , number of bits changed between two binary strings ) between two codes . Allwein et al . ( 2001 ) provided a unifying framework and multiclass loss bounds in terms of binary classification loss . More recent works have also used Hadamard code , a widely used error-correcting code ( Song et al. , 2021 ; Verma & Swami , 2019 ) . Other works have focused on the problem of extreme classification and the use of ECOC and compact codes ( sublinear increase in the length of codes with the number of classes ) for a classification problem with a large number of classes ( Cissé et al. , 2012 ; Evron et al. , 2018 ) . However , the proposed encoding and decoding approaches do not consider the ordinality of labels for regression . Further , the binary classifiers possess distinct error probability distribution properties for regression problems as observed empirically ( Section 3.1 ) , which can be exploited to design codes suitable for regression . Multiclass classification and ordinal regression by binary classification can be viewed as special cases falling under the BEL framework . As shown in Section 4 , other BEL designs yield improvements in accuracy over these approaches . Task-specific regression techniques are well explored as summarized below ( see also Appendix D. While effective , task-specific approaches lack generality by design . Head pose estimation : SSR-Net ( Rothe et al. , 2015 ) and FSA-Net ( Yang et al. , 2019 ) used softmax to refine labels . HopeNet ( Ruiz et al. , 2018 ) used a combination of classification and regression loss . QuatNet ( Hsu et al. , 2019 ) used a combination of regression and ordinal regression loss . Facial landmark detection : Wang et al . ( 2020 ) minimize L2 loss between predicted and target 2D heatmaps with the latter formed using small variance Gaussians centered on ground truth landmarks . AWing ( Wang et al. , 2019 ) modified loss for different pixels in the heatmap . LUVLi ( Kumar et al. , 2020 ) proposed a landmark ’ s location , uncertainty , and visibility likelihood-based loss . Bulat & Tzimiropoulos ( 2016 ) used binary heatmaps with pixel-wise binary cross-entropy loss . Age estimation : OR-CNN ( Niu et al. , 2016 ) and CORAL-CNN ( Cao et al. , 2020 ) used ordinal regression via binary classification . MV-Loss ( Pan et al. , 2018 ) proposed to penalize the model output based on the age distribution ’ s variance , while Gao et al . ( 2018 ) proposed to use the KL-divergence between the softmax output and a generated label distribution for training . 3 BINARY-ENCODED LABELS FOR REGRESSION ( BEL ) . We consider regression problems where the goal is to minimize the error between real-valued target labels yi and predicted labels ŷi , over a set of training samples i . We transform this problem to a set of binary classification sub-problems by converting a real-valued label to a binary code . Figure 1 shows the training and inference flow for BEL . The red-colored blocks highlight functions that vary under BEL . A real-valued label yi ∈ R is quantized to level Qi ∈ { 1 , 2 , ... , N } . The quantized label is then converted to a binary sequence Bi ∈ { 0 , 1 } M using an encoding function E . The binary-encoded labels Bi are used to train M classifiers . In the prediction phase , the classifiers predict a binary code B̂i ∈ { 0 , 1 } M for an input xi . The predicted binary code ( B̂i ) or the real-valued confidence vector of classifiers ’ prediction ( Ẑi ) is then decoded to a predicted label ŷi ∈ R using a decoding function D. We explore decoding functions that yield either quantized or continuous predicted outputs . For the latter , we propose an expected correlation-based decoding function that decodes classifiers ’ output to a continuous value , reducing the quantization error ( Section 3.3 ) . There are five key design parameters when applying BEL : quantization , encoding , decoding , regressor network architecture , and training loss formulation yielding a large design space . In this work we consider only uniform quantization while leaving nonuniform quantization ( Fu et al. , 2018 ) to future work . Section 3.2 and 3.3 explore the characteristics of suitable encoding , decoding , and training loss functions . Section 3.4 explore the impact of regression network architecture . We find varying any of these aspects can improve accuracy . While BEL provides a framework and some codes appear generally better than others , the most suitable BEL code to employ appears to vary across task , dataset , and network architecture , as we show both theoretically ( Section 3.1 ) and empirically ( Section 4 ) .

Summary: This study presents a label encoding method (binary encoded labels) to transform a regression problem into multiple binary classifiation problems. The authors designed encoding and decoding functions to encode a real-value into multiple classes and decode the classes into the real value. The proposed demonstrated improved performance on benchmark datasets.