Detection Booster Training: A detection booster training method for improving the accuracy of classifiers.

Deep learning models owe their success at large , to the availability of a large1 amount of annotated data . They try to extract features from the data that contain2 useful information needed to improve their performance on target applications.3 Most works focus on directly optimizing the target loss functions to improve the4 accuracy by allowing the model to implicitly learn representations from the data.5 There has not been much work on using background/noise data to estimate the6 statistics of in-domain data to improve the feature representation of deep neural7 networks . In this paper , we probe this direction by deriving a relationship between8 the estimation of unknown parameters of the probability density function ( pdf ) 9 of input data and classification accuracy . Using this relationship , we show that10 having a better estimate of the unknown parameters using background and in-11 domain data provides better features which leads to better accuracy . Based on12 this result , we introduce a simple but effective detection booster training ( DBT ) 13 method that applies a detection loss function on the early layers of a neural network14 to discriminate in-domain data points from noise/background data , to improve15 the classifier accuracy . The background/noise data comes from the same family16 of pdfs of input data but with different parameter sets ( e.g. , mean , variance ) . In17 addition , we also show that our proposed DBT method improves the accuracy even18 with limited labeled in-domain training samples as compared to normal training.19 We conduct experiments on face recognition , image classification , and speaker20 classification problems and show that our method achieves superior performance21 over strong baselines across various datasets and model architectures.22 1 INTRODUCTION23 Modern pattern recognition systems achieve outstanding accuracies on a vast domain of challenging24 computer vision , natural language , and speech recognition benchmarks ( Russakovsky et al . ( 2015 ) ; 25 Lin et al . ( 2014 ) ; Everingham et al . ( 2015 ) ; Panayotov et al . ( 2015 ) ) . The success of deep learning26 approaches relies on the availability of a large amount of annotated data and on extracting useful27 features from them for different applications . Learning rich feature representations from the available28 data is a challenging problem in deep learning . A related line of work includes learning deep latent29 space embedding through deep generative models ( Kingma & Welling ( 2014 ) ; Goodfellow et al.30 ( 2014 ) ; Berthelot et al . ( 2019 ) or using self-supervised learning methods ( Noroozi & Favaro ( 2016 ) ; 31 Gidaris et al . ( 2018 ) ; Zhang et al . ( 2016b ) ) or through transfer learning approaches ( Yosinski et al.32 ( 2014 ) ; Oquab et al . ( 2014 ) ; Razavian et al . ( 2014 ) ) .33 In this paper , we propose to use a different approach to improve the feature representations of deep34 neural nets and eventually improve their accuracy by estimating the unknown parameters of the35 probability density function ( pdf ) of input data . Parameter estimation or Point estimation methods36 are well studied in the field of statistical inference ( Lehmann & Casella ( 1998 ) ) . The insights from37 the theory of point estimation can help us to develop better deep model architectures for improving38 the model ’ s performance . We make use of this theory to derive a correlation between the estimation39 of unknown parameters of pdf and classifier outputs . However , directly estimating the unknown40 pdf parameters for practical problems such as image classification is not feasible since it can sum41 up to millions of parameters . In order to overcome this bottleneck , we assume that the input data42 points are sampled from a family of pdfs instead of a single pdf and propose to use a detection43 based training approach to better estimate the unknowns using in-domain and background/noise data.44 One alternative is that we can use generative models for this task , however , they mimic the general45 distribution of training data conditioned on random latent vectors and hence can not be directly applied46 for estimating the unknown parameters of a family of pdfs . Our proposed detection method involves47 a binary class discriminator that separates the target data points from noise or background data . The48 noise or background data is assumed to come from the same family of distribution of in-domain49 data but with different moments ( Please refer to the appendix for more details about the family of50 distributions and its extension to a general structure ) . In image classification , this typically represents51 the background patches from input data that fall under the same distribution family . In speech domain,52 it can be random noise or the silence intervals in speech data . Collecting such background data to53 improve the feature representations is much simpler as compared to using labeled training data since54 it is time-consuming and expensive to collect labeled data . Since the background patches in images55 or noise in speech signals are used for binary classification in our method , we refer to such data56 as the noise of an auxiliary binary classification problem denoted by auxiliary binary classification57 ( ABC ) -noise dataset . An advantage of using ABC-noise data during training is that it can implicitly58 add robustness to deep neural networks against the background or noisy data.59 Since ABC-noise data can be collected in large quantities for free and using that data in our approach60 improves the classification benchmarks , we investigate whether this data can act as a substitute for61 labeled data . We conduct empirical analysis and show that using only a fraction of labeled training62 data together with ABC-noise data in our DBT method , indeed improves the accuracy as compared63 to normal training.64 To summarize , our contributions are threefold . First , we present a detailed theoretical analysis on65 the relation between the estimation of unknown parameters of pdf of data and classification outputs.66 Second , based on the theoretical analysis , we present a simple booster training method to improve67 classification accuracy which also doubles up as an augmented training method when only limited68 labeled data is available . Third , we consistently achieve improved performances over strong baselines69 on face recognition , image classification , and speaker recognition problems using our proposed70 method , showing its generalization across different domains and model architectures.71 2 RELATED WORK72 Notations and Preliminary : In this paper , vectors , matrices , functions , and sets are denoted by bold73 lower case , bold uppercase , lower case , and calligraphic characters , respectively . Consider a datapoint74 denoted by x . We assume that x belongs to a family of probability density functions ( pdf ’ s ) defined75 as P = { p ( x , θ ) , θ ∈ Θ } , where Θ is the possible set of parameters of the pdf . In general , θ is a real76 vector in higher dimensions . For example , in a mixture of Gaussians , θ is a vector containing the77 component weights , the component means , and the component covariance matrices . In this paper , we78 assume that θ is an unknown deterministic function ( There are other approaches such as bayesian79 that consider θ as a random vector ) . In general , although the structure of the family of pdfs is itself80 unknown , defining a family of pdfs such as P can help us to develop theorems and use those results81 to derive a new method . For the family of distribution P , we can define the following classification82 problem83 { C1 : θ ∈ Θ1 , C2 : θ ∈ Θ2 , · · · , Cn : θ ∈ Θn } ( 1 ) where set of Θi ’ s is a partition of Θ . The notation of ( 1 ) means that , class Ci deals with a set of84 data points whose pdf is p ( x , θi ) where θi ∈ Θi . A wide range of classification problems can be85 defined using ( 1 ) e.g. , ( ( Lehmann & Casella , 2006 , Chapter 3 ) ) and ( ( Duda et al. , 2012 , Chapter 4 ) ) .86 The problem of estimating θ comes under the category of parametric estimation or point estimation87 ( Lehmann & Casella ( 1998 ) ) . Estimating the unknown parameters of a given pdf p ( x , θ ) , have been88 extensively studied in the field of point estimation methods ( Lindgren ( 2017 ) ; Lee et al . ( 2018 ) ; 89 Lehmann & Casella ( 2006 ) ) . An important estimator in this field is the minimum variance unbiased90 estimator and it is governed by the Cramer Rao bound . The Cramer Rao bound provides the lower91 bound of the variance of an unbiased estimator ( Bobrovsky et al . ( 1987 ) ) . Let the estimation of92 θ be denoted by θ̂ , and assume that θ̂ is an unbiased estimator , i.e. , E ( θ̂ ) = θ . Its covariance93 matrix denoted by Σθ̂ satisfies Σθ̂ − I −1 ( θ ) 0 , where A 0 implies that A is a non-negative94 definite matrix ( ( Lehmann & Casella , 1998 , chapter 5 ) ) and I ( θ ) : = −E ( ∂2 log ( p ( x , θ ) ) /∂θ2 ) 95 is called the Fisher information matrix . For an arbitrary differentiable function g ( · ) , an efficient96 estimator of g ( θ ) is an unbiased estimator when its covariance matrix equals to I−1g ( θ ) , where I −1 g ( θ ) 97 is the fisher information matrix of g ( θ ) , i.e. , the efficient estimator achieves the lowest possible98 variance among all unbiased estimators . The efficient estimator can be achieved using factorization of99 ∂ log ( p ( x , θ ) ) /∂g ( θ ) = Ig ( θ ) ( ĝ ( x ) − g ( θ ) ) , if it exists ( Rao ( 1992 ) ; Lehmann & Casella ( 1998 ) ) .100 Based on these results , we derive a relationship between the efficient estimation of unknowns and101 maximum likelihood classifier of ( 1 ) and use auxiliary binary classifiers to apply that result in our102 proposed DBT method.103 Parameter Estimations : Independent component analysis ( Hyvärinen ( 1999 ) ) decomposes a multi-104 variate signal into independent non-Gaussian signals . ICA can extract non-Gaussian features from105 Gaussian noise . Additionally , there is a class of classifiers called generalized likelihood ratio functions106 that replaces the estimation of unknown parameters into the likelihood functions . This approach107 provides a huge improvement in the field of parametric classifiers , where the family of pdf of data108 is given ( Zeitouni et al . ( 1992 ) , Conte et al . ( 2001 ) , Lehmann & Casella ( 2006 ) ) . Noise-contrastive109 estimation ( NCE ) ( Gutmann & Hyvärinen ( 2010 ) ) involves training a generative model that allows110 a model to discriminate data from a fixed noise distribution . Then , this trained model can be used111 for training a sequence of models of increasing quality . This can be seen as an informal competition112 mechanism similar in spirit to the formal competition used in the adversarial networks game . In113 Bachman et al . ( 2019 ) , a feature selection is proposed by maximizing the mutual information of the114 difference between features extracted from multiple views of a shared context . In that work , it is115 shown that the best results is given by using a mutual information bound based on NCE . The key116 difference between our method and NCE is that , we do not construct a generative model for noise.117 Instead of estimating the pdf of noise in NCE , we estimate the parameters of pdf of in-domain dataset118 using an auxiliary class that has many common parameters in its pdf . Moreover , we show that the119 estimation of that parameters are sufficient statistic for a classifier . We assume that the noise dataset is120 not pure and it has some similarity with the in-domain dataset , where it can help the feature selection121 layers to select relevant ( in-domain ) features , e.g. , see Fig . 3 . Further , in our approach , we do not122 construct the pdf of noise or in-domain data , instead we estimate its parameters directly , which is123 more efficient in terms of training , computation and also dimensionality reduction.124 Auxiliary classifiers were introduced in inception networks ( Szegedy et al . ( 2015 ) ) and used in ( Lee125 et al . ( 2015 ) ; S. et al . ( 2016 ) ) for training very deep networks to prevent vanishing gradient problems.126 Further , auxiliary classifiers were also proposed for early exit schemes ( Teerapittayanon et al . ( 2016 ) ) 127 and self-distillation methods ( Zhang et al . ( 2019a ; b ) ) . Such auxiliary classifiers tackle different128 problems by predicting the same target as the final classification layer . In contrast , our proposed DBT129 method involves auxiliary binary classifiers that detect noise , interference , and/or background data130 from in-domain data points for improving the target classification accuracy.131 3 ESTIMATION OF PARAMETERS OF PDF AND CLASSIFICATION132 For ( 1 ) , we define a deterministic discriminative function of Θi , denoted by ti ( · ) such that the133 following conditions are satisfied:134 • ti ( · ) maps Θ to real numbers such that ti ( θ ) > 0 , if θ ∈ Θi and ti ( θ ) ≤ 0 for θ /∈ Θi.135 • ti ( · ) is a differentiable function almost everywhere and ∫ Θ |ti ( θ ) |dµl ( θ ) < ∞ , where µl denotes136 the Lebesgue measure.137 The following theorem shows the relationship of ti ( · ) and the log-likelihood ratio of class Ci versus138 other classes . The proofs of Theorems 1 , 2 and 3 are provided in the appendix.139 Theorem 1 Assume that the pdf p ( x , θ ) is differentiable with respect to θ almost everywhere . If the140 efficient minimum variance and unbiased estimation of a deterministic discriminative function of Θi141 exists , then the log likelihood ratio of class i against the rest of classes is an increasing function of142 the minimum variance and unbiased estimation of Θi.143 Directly from this theorem , it follows that the optimal classifier using the maximum likelihood for ( 1 ) 144 is given as follows d ( x ) = arg maxi∈ { 1 , ··· , n } ki ( t̂i ( x ) ) , where ki ’ s are some increasing functions and145 ti ( · ) ’ s are the deterministic discriminative function of Θi ’ s such that the efficient minimum variance146 and unbiased estimation for them exists . Based on this result , a set of minimum variance and unbiased147 estimation of deterministic discriminative functions of Θi ’ s leads us to the maximum likelihood148 classifier . One approach is to directly estimate the deterministic discriminative functions , instead of149 maximizing the likelihood function . However , finding deterministic discriminative functions that150 have efficient minimum variance and unbiased estimation may not be feasible in practical problems,151 especially when the dimension of θ increases . Theorems 2 and 3 study the same relationship between152 the estimation of unknown parameters and the accuracy of classifiers for sub-optimal estimators and153 classifiers.154 Theorem 2 Consider the output of two classifiers for the ith class as follows : rj ( x ) = i if hj ( x ) > τ155 and rj ( x ) = other classes if hj ( x ) < τ , where j ∈ { 1 , 2 } . where hj ( x ) is the estimation of a156 deterministic discriminative function and τ is a classification threshold . Assume that the cumulative157 distribution function of hj ( x ) ’ s have bounded inflection points , and also , the probability of true158 positive of rj ( x ) is an increasing function of d ( θ ) , which is the deterministic discriminative function159 of class i , for all i . Further assume that for each τ the probability of false positive of r1 ( x ) is less160 than the probability of false positive of r2 ( x ) and the probability of true positive of r1 ( x ) is greater161 than the probability of true positive of r2 ( x ) . Then , there exists a hmin such that for all d ( θ ) > hmin162 and all θ we have Pr ( |h1 ( x ) − d ( θ ) | < ) > Pr ( |h2 ( x ) − d ( θ ) | < ) .163 Theorem 2 shows that a better classifier leads to a better estimation of d ( θ ) . In the next theorem , we164 show the dual property of this result.165 Theorem 3 Let Θm be a Borel set with positive Lebesgue measure in ( 1 ) for all m ∈ { 1 , · · · , n } .166 Assume that r1 ( · ) and r2 ( · ) are given as follows r1 ( x ) = m , if θ̂1 ∈ Θm and r2 ( x ) = m , if θ̂2 ∈ Θm.167 Also , assume that Pr ( ‖θ̂1 − θ‖ ≤ ) ≥ Pr ( ‖θ̂2 − θ‖ ≤ ) , for all θ ∈ Θ = ∪nm=1Θm and > 0,168 then the probability of classification error r1 ( · ) is less than r2 ( · ) where θ̂1 and θ̂2 are two different169 estimators of θ ∈ Θ = ∪M−1m=0 Θm.170 Theorem 3 proves that a more accurate estimator leads to a classifier that has a lower probability171 of classification error . From Theorem 1 , we can infer that a sufficient statistic for developing the172 maximum likelihood classification is t̂i ( x ) , which is the efficient minimum variance and unbiased173 estimation of the deterministic discriminative functions of Θi ’ s denoted by ti ( θ ) . In other words , the174 maximum likelihood classifier is a function of x only via the efficient minimum variance and unbiased175 estimation ti ( θ ) . We can estimate ti ( θ ) by replacing the estimation θ in ti ( · ) , i.e. , t̂i ( θ ) ≈ ti ( θ̂ ) ,176 where θ̂ is a function of x . From the above theorems , we conclude that improving the estimation177 of unknown parameters of pdf of data can improve the accuracy of the classifier . On the other side,178 having a good classifier means having a good estimator of unknowns of the pdf of input data . In179 many practical problems , the optimal maximum likelihood classifier may not be achievable , but the180 likelihood function of the classifier provides an optimal bound of the probability of error . In such181 cases , we can improve the accuracy of sub-optimal classifiers and that is the main focus of this paper.182 Fig . 1 illustrates the proposed theorems visually.183 4 PROPOSED METHOD : DETECTION BOOSTER TRAINING ( DBT ) 184 In this section , we propose the detection booster training ( DBT ) method based on the achieved185 theorems in the previous section to improve the accuracy of deep networks . Specifically , we divide186 a deep model into two parts - early and later layers . We apply a detector ( detection here means187 detecting a target pattern from noise/background ) on the early layers of the neural network in order188 to improve the estimation of unknown parameters of the family of pdf ( based on Theorem 2 ) . A189 better estimation of unknown parameters corresponds to better feature representations in the early190 layers and these features are input to the rest of the layers to construct the deterministic discriminative191 functions ( DDF ) useful for the in-domain data classification ( based on Theorem 3 ) .192 A general schema for dividing a deep model into two sub-models namely PEF ( parameter estimator193 functions ) and DDF is depicted in Figure 2 . The early layers of the model estimate the unknown194 parameters of pdf of data while the later layers construct the discriminative functions essential for195 classification . Based on this scheme , we formally define the three main components of DBT as196 follows:197 • parameter estimator functions ( PEF ) : The sub-network from input layer to the kth layer , where k is198 a hyperparameter in the DBT approach.199 • auxiliary binary classification ( ABC ) : Some additional layers are attached to the end of PEF,200 mapping the output of the kth layer to a one-dimensional vector.201 • deterministic discriminative functions ( DDF ) : The sub-network from kth layer to the output of the202 model . The output of model is a vector equal to the length of the number of classes n.203 From Theorem 2 , we showed that unknown parameter estimation can be improved using a detection204 approach . During training , we apply a binary classification on the early layers ( PEF ) of the model to205 improve the estimation of unknown parameters of pdf and subsequently provide rich feature vectors206 for DDF . We define the auxiliary binary classification problem ( ABC problem ) as follows:207 • Class 1 ( alternative hypothesis ) of ABC problem denoted byH1 is set of all data points of classes208 of C1 to Cn , i.e . θ ∈ ∪ni=1Θi.209 • Class 0 ( null hypothesis ) of ABC problem denoted by H0 is a dataset of data points from same210 distribution p ( x , θ ) but θ /∈ ∪ni=1Θi . We also define the dataset of Class 0 of ABC as ABC-noise211 dataset , i.e. , the ABC is given by the following hypothesis testing problem : H1 : θ ∈ ∪ni=1Θi versus212 H0 : θ /∈ ∪ni=1Θi . In many practical problems , the noise , background or interference data related to213 the in-domain dataset have same type of probability distribution but different pdf parameters . Hence,214 using that dataset is a cheap and adept choice for the null hypothesis of ABC.215 The Auxiliary Binary Classification problem influences only the PEF and ABC units while the main216 classification problem with n classes updates the parameters of both PEF and DDF using in-domain217 data . Since the auxiliary classifier is only used during training , the inference model ( IM ) consists of218 only PEF and DDF and hence , there is no additional computation cost during inference . We formulate219 the aforementioned method in the following notations and loss functions . Assume that x is a data220 point that belongs to Class Ci , i ∈ { 1 , · · · , n } or Class H0 of ABC . Here , we define two type of221 labels denoted by lABC and lMC , where the subscription `` MC '' stands for multi-classes . So , if x222 belongs to class Ci , then lABC = 1 and lMC = i− 1 , else if x is a ABC-noise data point , lABC = 0223 and lMC is None . Therefore , the loss function is defined as:224 Ltot = LABC ( QABC ( QPEF ( x ) ) , lABC ) + λlABCLMC ( QDDF ( QPEF ( x ) ) , lMC ) , ( 2 ) where QPEF , QABC and QDDF are the functions of PEF , ABC and DDF blocks , respectively . We225 set the hyperparameter λ = 1 to balance the two loss terms . It is seen that , the second term of the226 total loss is zero if lABC = 0 . LABC and LMC are selected based on the problem definition and227 datasets . For classification , a simple selection for them can be binary cross-entropy and cross-entropy,228 respectively . For a given task and deep neural network , the choice of k and LABC influences the229 feature representation of early layers differently and consequently the accuracy of the model . We230 provide empirical studies in the next section to verify the same.231 5 EXPERIMENTAL STUDY OF DBT232 FACE RECOGNITION233 We conduct experiments on face recognition benchmarks and show that the DBT method learns rich234 features essential for face recognition . We also discover an important observation that current state-235 of-the-art ( SOTA ) face recognition models are very sensitive to non-face data , in particular , animal236 faces . Fig . 4 shows a few examples of misidentified faces and their corresponding animal distractors237 from the IJB-B dataset using the ArcFace ( Deng et al . ( 2019 ) ) model . We show that our DBT method238 not only improves the verification accuracy but also implicitly tackles this robustness issue of current239 models against non-face data . Implementation details are provided in the appendix.240 We consider the PEF discussed in Section 4 to be the first three layers of the model and DDF to be241 the rest of layers . Ablation studies on the choice of PEF and DDF are provided in the supplementary242 material . We define LMC in ( 2 ) as the SOTA ArcFace loss function proposed in ( Deng et al . ( 2019 ) ) .243 The ABC-noise is a non-face dataset containing 500K images that we collected from background244 patches of MS1MV2 ( Guo et al . ( 2016 ) ) ( More details in Appendix ) . We experimented with two245 different loss functions for LABC . For the first one , since popular face recognition models ( Deng et al.246 ( 2019 ) ; Wang et al . ( 2018 ) ) use normalized output features and compute the losses on a hypersphere,247 we select LABC as follows . Let pf ∈ Rd and pnf ∈ Rd denote the prototypes for faces and non-248 faces , respectively . Following ( Mettes et al . ( 2019 ) ) , we constrain the face/non-face prototypes on249 diametrically opposite directions i.e cos ( θpfpnf ) = −1 and normalize the output feature vectors for250 faces and non-faces such that ‖pfi‖ = ‖pnfi‖ = 1 . We then define the LABC as,251 LABC = − 1 N N∑ i=1 log ( es ( cos ( m1θyi+m2 ) −m3 ) es ( cos ( m1θyi+m2 ) −m3 ) + es cos θ2 ) + 1 N N∑ i=1 ( −1− |pfi .pnfi | ) 2 , ( 3 ) where θyi and θ2 correspond to the angles between the weights and the features for face and non-face252 labels , respectively ; m1 , m2 , m3 are the angular margins ; s denotes the radius of the hypersphere . For253 the second choice , we use simple binary cross entropy for LABC . Table 1 shows that the verification254 accuracy on LFW ( Huang et al . ( 2007 ) ) using ( 3 ) is 0.16 % higher than simple cross entropy loss . This255 also shows that choosing a task-specific LABC is essential in obtaining more accurate results . We use256 Eqn.1 as the default for LABC in all our face recognition experiments , unless otherwise stated.257 Table 3 compares the verification accuracy of our method versus the current SOTA method ArcFace258 on five different test sets , LFW , CPLFW ( Zheng & Deng ( 2018 ) ) , CALFW ( Zheng et al . ( 2017 ) ) ,259 CFP-FP ( Sengupta et al . ( 2016 ) ) and AgeDb-30 ( Moschoglou et al . ( 2017 ) ) . For the LFW test set,260 we follow the unrestricted with labeled outside data protocol to report the performance . We trained261 ResNet-50 and ResNet-100 using ArcFace and DBT approaches on CASIA ( small ) and MS1MV2262 ( large ) datasets , respectively . The results show that DBT method outperforms ArcFace on all datasets.263 Table 7 shows the angle statistics of the trained ArcFace and DBT models on the LFW dataset . Min.264 Inter and Inter refer to the mean of minimum angles and mean of all angles between the template265 embedding features of different classes ( mean of the embedding features of all images for each class ) ,266 respectively . Intra refers to the mean of angles between xi and template embedding feature for each267 class . From Table 7 , we infer that DBT extracts better face features and hence reduces the intra-class268 variations . Directly from Tables 3 and 7 , we infer that first , DBT consistently improves the accuracy269 on all test sets . Second , learning better features in the early layers is crucial to obtain rich face feature270 embeddings . Third , the achieved gain using DBT is more pronounced on models trained using a271 smaller ( CASIA ) dataset ( it has fewer identities and images ) . This shows that DBT can address the272 issue of the lack of in-domain data using cheap ABC-noise data.273 We also provide the results of training Inception-ResNet-V1 and ResNet-64 models using DBT on274 MS1MV2 to show the generalization capacity of the DBT method . For the Inception-ResNet-V1 and275 ResNet-64 , the PEF is set to be the first six layers and the DDF is the rest of the model . We use large276 margin cosine loss ( LMCL ) Wang et al . ( 2018 ) for LMC and Cross entropy ( CE ) for LABC . Table 4277 shows the verification accuracy on LFW for Inception-ResNet-V1 and ResNet-64 models trained278 on MS1MV2 with and without DBT . The results show that DBT method is independent of model279 depth or architectures or loss functions and thereby consistently improves the accuracy compared280 to baseline results . Table 4 also compares the DBT method with state-of-the-art methods on LFW281 and YTF datasets . DBT method notably improves the baselines that are comparable to ArcFace and282 superior to all the other methods . We were not able to reproduce the results of the ArcFace paper283 using our Tensorflow implementation and dataset . We believe that using the original implementation284 and dataset from ArcFace will achieve superior results over the baselines on the benchmark datasets285 as evident from the results of our implementation . Finally , we compare the result ArcFace and DBT286 on IJB-B and IJB-C , in Table 5 . It is seen that DBT provides a notable boost on both IJB-B and287 IJB-C by a considerable margin . DBT improves the verification accuracy as high as 1.94 % on IJB-B288 and 2.57 % on IJB-C dataset at 10−4 false alarm rate ( FAR ) . We plot the receptive fields of the top289 ten maximally activated neurons of an intermediate layer of the face recognition model to visualize290 the features learned using the DBT method . Fig . 3 shows that the receptive fields of layer 15 of291 the inception-resnet-v1 model trained using DBT attends to the regions of eyes , nose and mouth as292 compared to insignificant regions in the normal training method . This shows that DBT learns more293 discriminative features essential to face recognition , corroborating our theoretical claims.294 To show that current SOTA models are not robust to animal faces , we performed a 1 : N identification295 experiment with approximately 3000 animal distractors on the IJB-B ( Whitelam et al . ( 2017 ) ) dataset.296 We trained the face recognition model with about 500K non-face data which contains 200 animal297 faces . This is disjoint from the 3000 distractors used in the identification experiment . We collected the298 animal faces from web images using MTCNN ( Zhang et al . ( 2016a ) ) face detector which are the false299 positives from the face detector . Table 2 shows the Rank-1 identification accuracy of ResNet-100300 on IJB-B dataset , trained on MS1MV2 using the ArcFace loss ( ResNet-100-AF ) versus our DBT301 approach ( ResNet-100-DBT ) . The third column of Table 2 denotes the accuracy on a hard subset302 of images ( false positives from ArcFace model ) on the IJB-B dataset denoted by H-set . Results303 of Table 2 show that current face recognition models are unable to discriminate out-of-distribution304 ( non-face ) images from face images . Our ResNet-100-DBT significantly ( as high as 21 % ) reduces the305 misidentification rate as compared to the ArcFace model which shows that DBT method inherently306 overcomes this issue while also improving face recognition accuracy.307 IMAGE CLASSIFICATION308 In this section , we evaluate ResNet-110 and ResNext-101 models trained with and without DBT on309 image classification problem using CIFAR-10 , CIFAR-100 , and ImageNet . We also show the power310 of DBT to compensate for the smaller in-domain training set . For all implementations , PEF is defined311 to be the first three layers and DDF is the rest of the model . LABC and LMC are set to cross-entropy312 loss . ABC-noise is the same data used in face recognition experiments . We follow the same training313 configurations from ( He et al . ( 2016 ) ; Xie et al . ( 2017 ) ) .314 To study the efficacy of the DBT method in augmenting smaller in-domain training datasets , we315 also trained ResNet-100 and ResNext-101 using partial training data on CIFAR-10 and CIFAR-100.316 We randomly selected a fraction of the training data to be our training set , e.g. , k/5 of dataset317 means that we only used k fifth of total samples for training . From first row of Table 8 , we find that318 models trained with DBT show 0.59 % and 0.35 % improvement on CIFAR-10 , 0.62 % and 1.45 % 319 improvement on CIFAR-100 over baseline models for ResNet-110 and ResNext-101 architectures,320 respectively . Furthermore , using partial training data with our DBT method achieves superior results321 ( as high as 5.49 % on ResNext ( 1/5 ) CIFAR-100 ) as compared to normal training . Table 6 shows322 the results on Imagenet . We see that DBT improves the accuracy by 0.28 % on Top-1 accuracy . This323 shows that the DBT method consistently improves the results on both small and large datasets.324 SPEAKER IDENTIFICATION325 We consider the problem of speaker identification using the VGG-M ( Chatfield et al . ( 2014 ) ) model.326 We set PEF as the first two CNN layers and DDF as the remaining CNN layers . LABC and LMC327 are defined to be the cross-entropy loss . The ABC-noise is generated from the silence intervals of328 VoxCeleb ( Nagrani et al . ( 2017 ) ) augmented with Gaussian noise with variance one . The input to the329 model is the short-time Fourier transformation of speech signals with a hamming sliding window330 of width 25 ms and step 10 ms. Table 9 provides the accuracies of VGG-M model trained with and331 without DBT on VoxCeleb , Librispeech ( Panayotov et al . ( 2015 ) ) , VCTK ( Veaux et al . ( 2016 ) ) and332 ELSDR ( L. ( 2004 ) ) datasets . Table 9 shows that the trained models using DBT significantly improves333 the accuracy ( as high as 5.62 % ) for all datasets . Implementation details are provided in the appendix.334 335 MISCELLANEOUS EXPERIMENTS336 In this section , we experiment with the naive way of using background data by considering non-faces337 as a separate class in the final classification layer . For face recognition , Table 11 shows the results338 of training with an additional background class on MS1MV2 dataset with and without using DBT.339 ResNet+mod refers to a model trained with ArcFace loss and n + 1 classes where the additional class340 corresponds to the non-faces . ResNet-DBT+mod refers to a model trained with both DBT and the341 additional non-face class . We find that adding the additional non-face class hurts the performance342 of the model whereas ResNet-DBT+mod improves the results significantly relative to ResNet+mod343 model . Since the non-face dataset is sampled from a wide range of a family of distributions compared344 with faces , it has a larger range of unknown parameters , then the sufficient statistic of them should be345 larger than the sufficient statistics of face data . Thus , when we restrict faces and non-faces on the346 surface of a hypersphere , the non-face data is more spread on the surface compared with each of the347 other face classes . We demonstrate this effect with the help of a toy example in Fig . 6 in the appendix.348 We also conduct this experiment on CIFAR-10/CIFAR-100 and report it in Table 10 . We see that349 naively incorporating the background class is inferior to DBT showing that DBT is an effective350 technique to utilize background data to boost the performance of classification models.351 6 CONCLUSION352 In this paper , we presented a detailed theoretical analysis of the dual relationship between estimating353 the unknown pdf parameters and classification accuracy . Based on the theoretical study , we presented354 a new method called DBT using ABC-noise data for improving in-distribution classification accuracy.355 We showed that using ABC-noise data helps in better estimation of unknown parameters of pdf of356 input data and thereby improves the feature representations and consequently the accuracy in image357 classification , speaker classification , and face recognition benchmarks . It also augments the training358 data when only limited labeled data is available by improving accuracy . We showed that the concept359 of DBT is generic and generalizes well across domains through extensive experiments using different360 model architectures and datasets . Our framework is complementary to existing training methods and361 hence , it can be easily integrated with current and possibly future classification methods to enhance362 accuracy . In summary , the proposed DBT method is a powerful technique that can augment limited363 training data and improve classification accuracy in deep neural networks.364 REFERENCES365 M. Abadi , A. Agarwal , P. Barham , E. Brevdo , Z. Chen , C. Citro , G. S. Corrado , A. Davis , J. Dean,366 M. Devin , and S. Ghemawat and . TensorFlow : Large-scale machine learning on heterogeneous367 systems , 2015.368 Philip Bachman , R Devon Hjelm , and William Buchwalter . Learning representations by maximizing369 mutual information across views . In Advances in Neural Information Processing Systems , pp.370 15535–15545 , 2019.371 T. L. Berg , A. C. Berg , J. Edwards , and D. A. Forsyth . Who ’ s in the picture . NeurIPS , 2004.372 David Berthelot , Colin Raffel , Aurko Roy , and Ian Goodfellow . Understanding and improving373 interpolation in autoencoders via an adversarial regularizer . ICLR , 2019.374 Ben-Zion Bobrovsky , E Mayer-Wolf , and M Zakai . Some classes of global cramér-rao bounds . The375 Annals of Statistics , pp . 1421–1438 , 1987.376 Ken Chatfield , Karen Simonyan , Andrea Vedaldi , and Andrew Zisserman . Return of the devil in the377 details : Delving deep into convolutional nets . BMVC , 2014.378 Ernesto Conte , Antonio De Maio , and Giuseppe Ricci . Glrt-based adaptive detection algorithms for379 range-spread targets . IEEE transactions on signal processing , 49 ( 7 ) :1336–1348 , 2001.380 J. Deng , J. Guo , X. Niannan , and S. Zafeiriou . Arcface : Additive angular margin loss for deep face381 recognition . In Computer Vision and Pattern Recognition ( CVPR ) , 2019.382 Richard O Duda , Peter E Hart , and David G Stork . Pattern classification . John Wiley & Sons , 2012.383 Mark Everingham , SM Ali Eslami , Luc Van Gool , Christopher KI Williams , John Winn , and Andrew384 Zisserman . The pascal visual object classes challenge : A retrospective . International journal of385 computer vision , 111 ( 1 ) :98–136 , 2015.386 F. F. Li , R. Fergus , and P. Perona . Learning generative visual models from few training examples : An387 incremental bayesian approach tested on 101 object categories . In CVPR Workshop , pp . 178–178,388 2004.389 Spyros Gidaris , Praveer Singh , and Nikos Komodakis . Unsupervised representation learning by390 predicting image rotations . ICLR , 2018.391 Ian Goodfellow , Jean Pouget-Abadie , Mehdi Mirza , Bing Xu , David Warde-Farley , Sherjil Ozair,392 Aaron Courville , and Yoshua Bengio . Generative adversarial nets . NeurIPS , pp . 2672–2680 , 2014.393 Y. Guo , L. Zhang , Y. Hu , X . He , and J. Gao . Ms-celeb-1m : A dataset and benchmark for large-scale394 face recognition . ECCV , 9907:87–102 , 2016.395 Michael Gutmann and Aapo Hyvärinen . Noise-contrastive estimation : A new estimation principle396 for unnormalized statistical models . In Proceedings of the Thirteenth International Conference on397 Artificial Intelligence and Statistics , pp . 297–304 , 2010.398 K. He , X. Zhang , S. Ren , and J . Sun . Deep residual learning for image recognition . Computer Vision399 and Pattern Recognition , pp . 770–778 , 2016.400 G. B. Huang , M. Ramesh , T. Berg , and E. Learned-Miller . Labeled faces in the wild : A database for401 studying face recognition in unconstrained environments . In Technical Report , 2007.402 Aapo Hyvärinen . Survey on independent component analysis . 1999.403 S. Ioffe and C. Szegedy . Batch normalization : Accelerating deep network training by reducing404 internal covariate shift . International Conference on Machine Learning , 37:448–456 , 2015.405 Diederik P Kingma and Max Welling . Auto-encoding variational bayes . ICLR , 2014.406 Feng L. Speaker recognition , informatics and mathematical modelling . Technical University of407 Denmark , DTU , 2004.408 C-Y . Lee , S. Xie , P. Gallagher , Z. Zhang , and Z. Tu . Deeply-Supervised Nets . Proceedings of409 Machine Learning Research ( PMLR ) , 38:562–570 , 2015.410 Youngjo Lee , John A Nelder , and Yudi Pawitan . Generalized linear models with random effects:411 unified analysis via H-likelihood , volume 153 . CRC Press , 2018.412 E. L. Lehmann and G. Casella . Theory of point estimation , 1998 . 2ndn ed.413 Erich L Lehmann and George Casella . Theory of point estimation . Springer Science & Business414 Media , 2006.415 Tsung-Yi Lin , Michael Maire , Serge Belongie , James Hays , Pietro Perona , Deva Ramanan , Piotr416 Dollár , and C Lawrence Zitnick . Microsoft coco : Common objects in context . In European417 conference on computer vision , pp . 740–755 . Springer , 2014.418 Bernard Lindgren . Statistical theory . Routledge , 2017.419 B. Maze , J. Adams , J . A. Duncan , N. Kalka , T. Miller , C. Otto , A. K. Jain , W. T. Niggel , J. Anderson,420 J. Cheney , and P. Grother . Iarpa janus benchmark - c : Face dataset and protocol . International421 Conference on Biometrics , pp . 158–165 , 2018.422 P. Mettes , E. van der Pol , and C. Snoek . Hyperspherical prototype networks . NeuRIPS , 01 2019.423 S. Moschoglou , A. Papaioannou , C. Sagonas , J. Deng , I. Kotsia , and S. Zafeiriou . Agedb : the first424 manually collected , in-the-wild age database . CVPR Workshop , 2 ( 3 ) :5 , 2017.425 Arsha Nagrani , Joon Son Chung , and Andrew Zisserman . Voxceleb : a large-scale speaker identifica-426 tion dataset . arXiv preprint arXiv:1706.08612 , 2017.427 M. Noroozi and P. Favaro . Unsupervised learning of visual representations by solving jigsaw puzzles.428 ECCV , 2016.429 M. Oquab , L. Bottou , I. Laptev , and J. Sivic . Learning and transferring mid-level image representa-430 tions using convolutional neural networks . CVPR , pp . 1717–1724 , 2014.431 V. Panayotov , G. Chen , D. Povey , and S. Khudanpur . Librispeech : an asr corpus based on public432 domain audio books . International Conference on Acoustics , Speech and Signal Processing433 ( ICASSP ) , pp . 5206–5210 , 2015.434 BLS Prakasa Rao . Cramer-rao type integral inequalities for estimators of functions of multidimen-435 sional parameter . Sankhyā : The Indian Journal of Statistics , Series A , pp . 53–73 , 1992.436 Ali Razavian , Hossein Azizpour , Josephine Sullivan , and Stefan Carlsson . Cnn features off-the-shelf:437 an astounding baseline for recognition . CVPR Workshops , 2014.438 O. Russakovsky , J. Deng , H. Su , J. Krause , S. Satheesh , S. Ma , Z. Huang , A. Karpathy , A. Khosla,439 M. Bernstein , A. C. Berg , and F.F Li . ImageNet Large Scale Visual Recognition Challenge.440 International Journal of Computer Vision ( IJCV ) , 115 ( 3 ) :211–252 , 2015.441 Christian S. , Vincent V. , Sergey I. , Jon S. , and ZB W. Rethinking the inception architecture for442 computer vision . CVPR , 2016.443 S. Sengupta , J. Chen , C. Castillo , V. M. Patel , R. Chellappa , and D. W. Jacobs . Frontal to profile face444 verification in the wild . In Winter Conference on Applications of Computer Vision ( WACV ) , pp.445 1–9 , 2016.446 N. Srivastava , G. Hinton , A. Krizhevsky , I. Sutskever , and R. Salakhutdinov . Dropout : A simple way447 to prevent neural networks from overfitting . J. Mach . Learn . Res. , 15 ( 1 ) :1929–1958 , 2014.448 C. Szegedy , W. Liu , Y. Jia , P. Sermanet , S. Reed , D. Anguelov , D. Erhan , V. Vanhoucke , and449 A. Rabinovich . Going deeper with convolutions . CVPR , pp . 1–9 , 2015.450 S. Teerapittayanon , B. McDanel , and H. T. Kung . Branchynet : Fast inference via early exiting from451 deep neural networks . ICPR , 2016.452 Christophe Veaux , Junichi Yamagishi , Kirsten MacDonald , et al . Superseded-cstr vctk corpus:453 English multi-speaker corpus for cstr voice cloning toolkit . University of Edinburgh . The Centre454 for Speech Technology Research ( CSTR ) , 2016.455 H. Wang , Y. Wang , Z. Zhou , X. Ji , D. Gong , J. Zhou , Z. Li , and W. Liu . Cosface : Large margin456 cosine loss for deep face recognition . CVPR , pp . 5265–5274 , 2018.457 C. Whitelam , E. Taborsky , A. Blanton , B. Maze , J. Adams , T. Miller , N. Kalka , A. K. Jain , J. A.458 Duncan , K. Allen , J. Cheney , and P. Grother . Iarpa janus benchmark-b face dataset . CVPR459 Workshops , pp . 592–600 , 2017.460 L. Wolf , T. Hassner , and I. Maoz . Face recognition in unconstrained videos with matched background461 similarity . CVPR , pp . 529–534 , 2011.462 S. Xie , R. Girshick , P. Dollár , Z. Tu , and K. He . Aggregated residual transformations for deep neural463 networks . CVPR , pp . 5987–5995 , 2017.464 D. Yi , Z. Lei , S. Liao , and S. Z. Li . Learning face representation from scratch . arXiv , abs/1411.7923,465 2014.466 J. Yosinski , J. Clune , Yoshua Bengio , and Hod Lipson . How transferable are features in deep neural467 networks ? NIPS , 2014.468 Ofer Zeitouni , Jacob Ziv , and Neri Merhav . When is the generalized likelihood ratio test optimal ? 469 IEEE Transactions on Information Theory , 38 ( 5 ) :1597–1602 , 1992.470 K. Zhang , Z. Zhang , Z. Li , and Y. Qiao . Joint face detection and alignment using multi-task cascaded471 convolutional networks . Signal Processing Letters , 23 ( 10 ) :1499–1503 , 2016a.472 L. Zhang , J . Song , A. Gao , J. Chen , C. Bao , and K. Ma . Be your own teacher : Improve the473 performance of convolutional neural networks via self distillation . ICCV , 2019a.474 Linfeng Zhang , Zhanhong Tan , Jiebo Song , Jingwei Chen , Chenglong Bao , and Kaisheng Ma . Scan:475 A scalable neural networks framework towards compact and efficient models . NeurIPS , 2019b.476 Richard Zhang , Phillip Isola , and Alexei Efros . Colorful image colorization . ECCV , 2016b.477 T. Zheng and W. Deng . Cross-pose lfw : A database for studying cross-pose face recognition in un-478 constrained environments . Technical Report , Beijing University of Posts and Telecommunications,479 2018.480 T. Zheng , W. Deng , and J. Hu . Cross-age lfw : A database for studying cross-age face recognition in481 unconstrained environments . arXiv , abs/1708.08197 , 2017.482 APPENDIX483 IN-DOMAIN FAMILY OF PDFS AND THE EXTENDED FAMILY OF DISTRIBUTIONS484 In this section , we discuss about background/noise and in-domain data points and their corresponding485 distributions to clarify the definition of those concepts in this paper . Consider a random vector denoted486 by s. Assume that the corresponding distribution is Gaussian with mean and variance given by α 6= 0487 and σ = 1 , respectively . Now , assume that we observed x = s + n , where the pdf of n is assumed to488 be Guassian with zero mean and variance σ2n , hence the pdf of x is Gaussian with mean α and variance489 1 + σ2n . Here , n is the background or noise data and the vector of unknowns is given by , θ = [ α , σ 2 n ] .490 The in-domain family of pdfs for x is then given by Px = { N ( α , 1 + σ2n ) |α 6= 0 , σ2n > 0 } . If we491 include the family of pdf of n to Px , then we can extend Px as P = { N ( α , 1 +σ2n ) |α ∈ R , σ2n > 0 } .492 So P is the union of family of pdfs of in-domain data points and noise/background data . From493 estimation theory , we know that the sufficient statistics and the unknown parameters of P can also494 represent the sufficient statistics and the unknown parameters of Px . In other words , an estimation of495 α can help us detect if the observed data point is from s + n or n by comparing it with a threshold.496 Thus , estimating the unknown parameters of the family of pdfs using P can provide more information497 about the observed data useful for tasks such as classification.498 In general , we can assume that a generalized family of pdfs is given by the family of pdf of noise or background along with the family of pdfs of in-domain data . Hence , estimating from the extended family of distribution can provide more information about the in-domain distribution . Let us consider that the pdf of in-domain data points is given by px ( x , [ θs , θn ] ) and the pdf of noise/background is given by pn ( x , θn ) , so the extended pdf can be represented by h ( pn ( x , θn ) , px ( x , [ θs , θn ] ) ) , where h is a function that combines two pdfs in a general structure . So a general family of distribution can be denoted as follows : P = { h ( pn ( x , θn ) , px ( x , [ θs , θn ] ) ) |θ : = [ θs , θn ] ∈ Θs , n } , where θ is defined as a new set of parameters in a higher dimension and Θs , n are set of all possible499 [ θs , θn ] that belongs to pn and px . The extended family of pdf provides more information about500 the nuisance parameters of pdf of in-domain datapoints . Inspired by this observation , we develop501 our detection booster training method using background/noise data . Figure 5 shows an example of502 background and in-domain data point.503 PROOF OF THEOREM 1504 Let ti ( · ) denote deterministic discriminative function of Θi . Since the efficient minimum variance505 and unbiased estimation of ti ( θ ) exists , we have506 ∂ ln ( p ( x , θ ) ) ∂ti ( θ ) = Iti ( θ ) ( t̂i ( x ) − ti ( θ ) ) , ( 4 ) where t̂i ( x ) is the minimum variance and unbiased estimation of ti ( θ ) using the data point x and Iti ( x ) is the Fisher information function of ti ( θ ) , which is given by Iti ( θ ) = ∂ti ( θ ) ∂θ T I ( θ ) ∂ti ( θ ) ∂θ ≥ 0 , where T denotes the transpose and I ( θ ) is the Fisher information matrix of θ . Now we show that507 the log-likelihood ratio is an increasing function in t̂i ( x ) . Note that Iti ( θ ) ≥ 0 ( Lehmann & Casella508 ( 2006 ) ) .509 On the other hand , we have d ln ( p ( x , θ ) ) = ∑ j ∂ ln ( p ( x , θ ) ) ∂θj dθj , therefore,510 ln ( p ( x , θ ) ) + k ( x ) = ∑ j ∫ ∂ ln ( p ( x , θ ) ) ∂θj dθj = ∑ j ∫ ∂ ln ( p ( x , θ ) ) ∂ti ( θ ) ∂ti ( θ ) ∂θj dθj =∫ ∂ ln ( p ( x , θ ) ) ∂ti ( θ ) ∑ j ∂ti ( θ ) ∂θj dθj = ∫ ( Iti ( θ ) ( t̂i ( x ) − ti ( θ ) ) ) ∑ j ∂ti ( θ ) ∂θj dθj = α ( θ ) t̂i ( x ) − β ( θ ) ( 5 ) where the third equality is archived based on the third property of ti ( · ) in its definition and the forth equality is given by replacing ( 4 ; k ( x ) is the constant of integration . Finally , the last equality is given by defining the following terms α ( θ ) : = ∫ Iti ( θ ) ∑ j ∂ti ( θ ) ∂θj dθj , β ( θ ) : = ∫ Iti ( θ ) ti ( θ ) ∑ j ∂ti ( θ ) ∂θj dθj , ( 6 ) thus dα ( θ ) dti ( θ ) = Iti ( θ ) ≥ 0 , i.e. , α ( θ ) is increasing in ti ( θ ) . Since , ti is a deterministic discriminative511 function of Θi , so for each j 6= i and θi ∈ Θi and θj ∈ Θj , we have ti ( θi ) > ti ( θj ) , therefore512 α ( θi ) ≥ α ( θj ) . The later inequality is achieved based on the increasing property of α ( θ ) with513 respect to ti ( θ ) .514 Using ( 5 ) , the log likelihood ratio of class i against the rest of classes is given by LLR : =515 ln ( p ( x , θi ) ) − ln ( p ( x , θj ) ) , so we have LLR = ( α ( θi ) − α ( θi ) ) t̂i ( x ) − ( β ( θi ) − β ( θj ) ) . LLR516 depends on x only via t̂i ( x ) and since for each j 6= i and θi ∈ Θi and θj /∈ Θi , α ( θi ) − α ( θi ) > 0,517 then LLR is increasing in t̂i ( x ) . 518 PROOF OF THEOREM 2519 The probability of true positive of class i of rj is given by Ptp , i , j = Prθ ( hj ( x ) > τ ) = 1− Fjθ ( τ ) , where Fiθ ( · ) denotes the Cumulative distribution function ( CDF ) of hj . Since the probability of true positive of class i of r1 is greater than r2 for all τ , F1θ ( τ ) < F2θ ( τ ) , for all τ . Now we define a function as follows u ( τ , θ ) : = F2θ ( τ ) − F1θ ( τ ) . Since the CDFs are increasing in τ and tend to 1 and the number of inflection points of these CDFs are bounded , there is an hmin such that , for τ > hmin , such that u ( τ , θ ) is a monotonically decreasing function in τ . Thus for any θ that satisfies d ( θ ) > hmin we have u ( d ( θ ) + , θ ) < u ( d ( θ ) − , θ ) . Replacing u ( h , θ ) = F2θ ( h ) − F1θ ( h ) in the last inequality , we have520 F2θ ( d ( θ ) + ) − F1θ ( d ( θ ) + ) < F2θ ( d ( θ ) − ) − F1θ ( d ( θ ) − ) ⇒ ( 7 ) F2θ ( d ( θ ) + ) − F2θ ( d ( θ ) − ) < F1θ ( d ( θ ) + ) − F1θ ( d ( θ ) − ) . ( 8 ) Based on the definition of CDF , we have521 Prθ ( |h2 ( x ) − d ( θ ) | < ) = Prθ ( d ( θ ) − < h2 ( x ) < d ( θ ) + ) < Prθ ( d ( θ ) − < h1 ( x ) ) < d ( θ ) + ) = Prθ ( |h1 ( x ) − d ( θ ) | < ) . ( 9 ) 522 PROOF OF THEOREM 3523 First , we prove the following claim,524 Claim : For any open set , there exists a set of disjoint countable open balls such that their union equals525 the origin open set.526 Proof of claim : Consider an open set O , and also consider x0 ∈ O , such that B ( x0 , r0 ) ⊆ O527 and r0 is the greatest possible radius between all possible open balls in O , where B ( x0 , r0 ) is the528 open ball with radius r0 at point x0 . Now , we define x1 ∈ O − B ( x0 , r0 ) , where B ( x0 , r0 ) is529 the closure of B ( x0 , r0 ) , as the point with greatest radius in O − B ( x0 , r0 ) and similarly xi ∈530 O − ∪i−1k=0B ( xk , rk ) such that B ( xi , ri ) provides the greatest radius in O − ∪ i−1 k=0B ( xk , rk ) . So531 we have O = ∪∞k=0B ( xk , rk ) . This is because , if the latest equality is not valid , then there exists532 an open ball in O − ∪∞k=0B ( xk , rk ) hence another open ball with greatest radius will be added to533 ∪∞k=0B ( xk , rk ) , which has a contradiction with the definition of ∪∞k=0B ( xk , rk ) . The claim is proven534 at this point.535 Now , we show the true positive probability of r1 is greater than r2 . Let Θ′m be the set of interior points of Θm , then , there exists a union of disjoint open balls such that Θ′m = ∪∞k=0B ( xk , rk ) . From assumptions in the theorem , we have Pr ( ‖θ̂1 − θ‖ ≤ ) ≥ Pr ( ‖θ̂2 − θ‖ ≤ ) , then Prθ ( θ̂1 ∈ B ( xk , rk ) ) ≥ Prθ ( θ̂2 ∈ B ( xk , rk ) ) , where θ ∈ Θm . Based on the claim we have536 Prθ ( θ̂1 ∈ Θ′m ) ≥ Prθ ( θ̂2 ∈ Θ′m ) . ( 10 ) Moreover , based on definition of ri , the true positive probability of class m is given by ptp , i = Prθ ( θ̂i ∈ Θm ) = Prθ ( θ̂i ∈ Θ′m ) + Prθ ( θ̂i ∈ Θm −Θ′m ) , for i = 1 , 2 . Additionally , from the Cauchy–Schwarz inequality , we have Prθ ( θ̂i ∈ Θm −Θ′m ) ≤ µl ( Θm −Θ′m ) = 0 , So , ptp , i = Prθ ( θ̂i ∈ Θ′m ) and from ( 10 ) the true positive probability of class i of r1 is greater than537 r1.538 The error probability of rj is given by per , j = 1− ∑n i=1 PiPtp , i , j , where Pi is the prior probability539 of class i. Therefor , per,1 ≤ per,2 . 540 541 CONNECTING THE THEOREMS WITH THE PROPOSED METHOD542 Fig . 6 shows the connection between the proposed theorems and the approach . In part 1 , Theorem543 2 connects the estimation of unknown parameters to the auxiliary classifier . In part 2 , the learned544 features are passed to a decision making network ( result of Theorem 2 ) . In part 3 , Theorem 3545 guarantees that the multi-class classifier outperforms other classifiers , because it is using the features546 from a better estimation of unknown parameters of pdf.547 548 TOY EXAMPLE:549 We demonstrate the effect of adding background class to the original classifier with a toy example550 and visualize it in Fig . 7 . In this example , the input is a sequence of binary bits ( +1 and −1 ) with551 length 3 in white Gaussian noise . the classifier is constructed using two fully connected layers with552 sigmoid and the last layer is normalized on unit circle . As seen from Fig . 7 , adding an additional553 noise class visibly reduces the feature separation between all the other classes.554 IMPLEMENTATION DETAILS555 FACE RECOGNITION556 We use Tensorflow ( Abadi et al . ( 2015 ) ) to conduct all our experiments . We train with a batch557 size of 256 on two NVIDIA TeslaV100 ( 32G ) GPUs . We train our models following small ( less558 than 1M training images ) and large ( more than 1M training images ) protocol conventions . We use559 CASIA-Webface ( Yi et al . ( 2014 ) ) dataset for small protocol and MS1MV2 dataset for the large560 protocol . We use ResNet-50 ( He et al . ( 2016 ) ) and ResNet-100 models for small and large protocols,561 respectively . The PEF is selected as the first three layers . Following ( Deng et al . ( 2019 ) ) , we apply562 BN ( Ioffe & Szegedy ( 2015 ) ) , dropout ( Srivastava et al . ( 2014 ) ) to the last feature map layer followed563 by a fully connected layer and batch normalization to obtain the 512-D embedding vector . We set564 the feature scale s parameter to 64 following ( Wang et al . ( 2018 ) ; Deng et al . ( 2019 ) ) and set the565 margin parameters ( m1 , m2 , m3 ) to ( 1 , 0.5 , 0 ) , respectively . For small scale protocol , we start the566 learning rate at 0.01 and divide the learning rate by 10 at 40K , 80K , and 100K iterations . We train for567 120K iterations . For large scale protocol , we start the learning rate at 0.01 and divide the learning568 rate by 10 at 80K , 100K , and 200K iterations . We train for 240K iterations . We use Momentum569 optimizer and set the momentum to 0.9 and weight decay to 5e-4 . We use the feature centre of all570 images from a template or all frames from a video in order to report the results on IJB-B , IJB-C and571 YTF datasets . For ABC-noise data , we cropped background images patches from MS1MV2 ( Guo572 et al . ( 2016 ) ) dataset and cropped hard examples from the Caltech-101 ( F. F. Li et al . ( 2004 ) ) dataset573 plus a few open sourced images ( animal faces ) using MTCNN ( Zhang et al . ( 2016a ) ) face detector.574 We generated roughly 500K non-face images for training the ABCs.575 SPEAKER IDENTIFICATION576 L2 loss and dropout with a rate of 0.2 are applied during training for generalization . The ABC-noise577 is collected form silence intervals of the VoxCeleb dataset , where an energy-based voice activity578 detection ( VAD ) is applied to detect the silence intervals . To augment the ABC-noise , Gaussian579 noise is added to the silence intervals . Each batch size is set to 64 and the optimizer is ADAM with580 a learning rate of 0.001 . The VoxCeleb dataset is trained for 11 epochs and the other datasets are581 trained for 6 epochs.582 LFW AND YTF DATASETS583 LFW database contains the annotations for 5171 faces in a set of 2845 images taken from the Faces584 in the Wild data set ( Berg et al . ( 2004 ) ) . YouTubeFaces ( Wolf et al . ( 2011 ) ) contains 3,425 videos of585 1,595 people . Following the standard convention , we report the results on 5000 video pairs using586 unrestricted with labeled outside data protocol.587 IJB-B AND IJB-C DATASETS588 The IJB-B contains 1,845 subjects with 21.8K still images and 55K frames from 7,011 videos . In589 total , there are 12,115 templates with 10,270 genuine matches and 8M impostor matches . The IJB-C590 dataset ( Maze et al . ( 2018 ) ) is a further extension of IJB-B , having 3,531 subjects with 31.3K still591 images and 117.5K frames from 11,779 videos . In total , there are 23 , 124 templates with 19,557592 genuine matches and 15 , 639K impostor matches.593

Summary: This paper proposes a training method for classification, with the goal of training with less data. The proposal is to train an auxiliary classifier at the same time. The auxiliary classifier and the main classifier share the early layers. The auxiliary classifier is a binary classifier that discriminates training data versus background/noise data. The proposed method is evaluated on image and speech classification tasks.