Detection Booster Training: A detection booster training method for improving the accuracy of classifiers.
Authors: Ali Ghobadzadeh
Deepak Sridhar
Juwei Lu
Wei Li
Venue:  (2021)
Keywords: Representation Learning
Deep Learning Theory
Face Recognition
Image Classification
Parametric Estimation Theory

Abstract: Deep learning models owe their success at large, to the availability of a large amount of annotated data. They try to extract features from the data that contain useful information needed to improve their performance on target applications. Most works focus on directly optimizing the target loss functions to improve the accuracy by allowing the model to implicitly learn representations from the data. There has not been much work on using background/noise data to estimate the statistics of in-domain data to improve the feature representation of deep neural networks. In this paper, we probe this direction by deriving a relationship between the estimation of unknown parameters of the probability density function (pdf) of input data and classification accuracy. Using this relationship, we show that having a better estimate of the unknown parameters using background and in-domain data provides better features which leads to better accuracy. Based on this result, we introduce a simple but effective detection booster training (DBT) method that applies a detection loss function on the early layers of a neural network to discriminate in-domain data points from noise/background data, to improve the classifier accuracy. The background/noise data comes from the same family of pdfs of input data but with different parameter sets (e.g., mean, variance). In addition, we also show that our proposed DBT method improves the accuracy even with limited labeled in-domain training samples as compared to normal training. We conduct experiments on face recognition, image classification, and speaker classification problems and show that our method achieves superior performance over strong baselines across various datasets and model architectures.

40 of unknown parameters of pdf and classifier outputs. However, directly estimating the unknown pdf parameters for practical problems such as image classification is not feasible since it can sum Under review as a conference paper at ICLR 2021
DBT: A DETECTION BOOSTER TRAINING METHOD FOR IMPROVING THE ACCURACY OF CLASSIFIERS
Anonymous authors Paper under double-bling review
ABSTRACT
Deep learning models owe their success large, the availability of large amount of annotated data. They try to extract features from the data that contain useful information needed to improve their performance
2
target applications. Most works focus on directly optimizing the target loss functions to improve the accuracy by allowing the model implicitly learn representations from the data. There has not been much work on using background/noose data to estimate the statistics of in-domain data to improve the feature representation of deep neural networks. In this paper, we probe this direction by deriving a relationship between the estimation of unknown parameters of the probability density function (pdf) of input data and classification accuracy. Using this relationship, we show that having a better estimate of the unknown parameters using background and domain data provides better features which leads to better accuracy. Based on this result, we introduce a simple but effective detection booster training (DBT) method that applies detection loss function the early layers of neural network to discriminate in-domain data points from noise/background data, to improve the classifier accuracy. The background/noise data comes from the same family of pdfs of input data but with different parameter sets (e.g., mean, variance). In addition, we also show that our proposed DBT method improves the accuracy even with limited labeled in in-domain training samples as compared to normal training. We conduct experiments on face recognition, image classification, and speaker classification problems and show that our method achieves superior performance over strong baselines across various datasets and model architectures.
11
13
15
16
17
18
19
20
21
22
23
INTRODUCTION
24
Modern pattern recognition systems achieve outstanding accuracies on vast domain of challenging computer vision, natural language, and speech recognition benchmarks (Russakovsky et al. (2015); Lin al. 2014); Everingham et al. (2015); Panayotov et al. (2015)). The success of deep learning approaches relies on the availability of a large amount of annotated data and on extracting useful features from them for different applications. Learning rich feature representations from the available data is a challenging problem in deep learning. A related line of work includes learning deep latent space embedding through deep generative models (Kingma & Welling (2014); Goooffeloo al. (0014): Berthelot et al. (2019) or using self-supervised learning methods (Noroozi & Favaro (2016);
25
26
27
29
31
32
Gidaris et (2018): Zhang et al. (2016b)) or through transfer learning approaches Yosiiski et al. (2014) Oquab et al. (014) Razavian el al. (2014)).
33
34 In this paper, we propose to a different approach improve the feature representations of deep neural nets and eventually improve their accuracy by estimating the unknown parameters of the probability density function (pdf) of input data. Parameter estimation or Point estimation methods are well studied in the field of statistical inference (Lchmann & Casella (1998). The insights from
35
36
37
38 the theory of point estimation can help us to develop better deep model architectures for improving
39 the model's performance. We make use of this theory to derive correlation between the estimation
42 up to millions of parameters. In order to overcome this bottleneck, we assume that the input data points sampled from family of pdfs instead of single pdf and propose to detection
44 based training approach to better estimate the unknowns using in-domain and background/noise data. One alternative is that use generative models for this task, however, they mimic the general

---

Under review as a conference paper at ICLR 2021
46
distribution of training data conditioned random latent vectors and hence cannot be directly applied for estimating the unknown parameters of a family of pdfs. Our proposed detection method involves binary class discriminator that separates the target data points from noise or background data. The noise or background data is assumed to come from the same family of distribution of in-domain data but with different moments (Please refer to the appendix for more details about the family of distributions and its extension to a general structure). In image classification, this typically represents the background patches from input data that fall under the same distribution family. In speech domain, it can be random noise or the silence intervals in speech data. Collecting such background data to improve the feature representations is much simpler as compared to using labeled training data since it is time-consuming and expensive to collect labeled data. Since the background patches in images
47
49
50
51
52
53
54
55
56 or noise in speech signals are used for binary classification in our method, we refer to such data as the noise of an auxiliary binary classification problem denoted by auxiliary binary classification (ABC)-noise dataset. An advantage of using ABC-noise data during training is that it can implicitly add robustness to deep neural networks against the background or noisy data.
57
58
60
Since ABC-noise data can be collected in large quantities for free and using that data in our approach improves the classification benchmarks, we investigate whether this data can act as a substitute for labeled data. We conduct empirical analysis and show that using only a fraction of labeled training data together with ABC-noise data in our DBT method, indeed improves the accuracy as compared to normal training.
61
62
63
64
65 To summarize, our contributions are threefold. First, we present a detailed theoretical analysis on
66 the relation between the estimation of unknown parameters of pdf of data and classification outputs. Second, based on the theoretical analysis, we present a simple booster training method to improve classification accuracy which also doubles up as augmented training method when only limited 69 labeled data is available. Third, we consistently achieve improved performances over strong baselines
67
70 on face recognition, image classification, and speaker recognition problems using our proposed method, showing its generalization across different domains and model architectures.
71
72
2 RELATED WORK
73
In this paper, vectors, matrices, functions, and sets are denoted by bold lower case, bold uppercase, lower case, and calligraphic characters, respectively. Consider a datapoint denoted by x. We assume that x belongs to family of probability density functions (pdf"s) defined {p(x, (x, 0). 0 E θ), where θ the possible set of parameters of the pdf. In general, @ a real vector in higher dimensions. For example, in mixture of Gaussians, O is a vector containing the component weights, the component means, and the component covariance matrices. In this paper, we assume that 0 is an unknown deterministic function (There are other approaches such as bayesian that consider 0 as a random vector). In general, although the structure of the family of pdfs is itself unknown defining a family of pdfs such P can help us to develop theorems and use those results 82 to derive a new method. For the family of distribution P, we can define the following classification problem
Notations and Preliminary:
74
75
76 as P
77
78
80
81
83
1 0 E C,. 0 E E
(1)
84
where set of O,'s 's is partition of θ The notation of ( means that, class C₁ deals with set of data points whose pdf is p(x, 0,) where 0, E O A wide range of classification problems can be defined using (1) e.g.. ((Lehmann& & Casella 2006 Chapter 3)) and ((Duda et al. [012] Chapter 4)).
85
87
The problem of estimating comes under the category of parametric estimation or point estimation (Lehmann & Casella (1998)). Estimating the unknown parameters of given pdf p(x, 0), have been extensively studied in the field of point estimation methods (Lindgren) (2017): [ee et al. (2018);
88
89
90
Lehmann& & Casella (006)). An important estimator in this field the minimum variance unbiased estimator and it is governed by the Cramer Rao bound. The Cramer Rao bound provides the lower bound of the variance of an unbiased estimator (Bobrovsky et al. (1987) )) Let the estimation of be denoted by 0, and assume that Ø is an unbiased estimator, i.c., E ()) = 0. Its covariance matrix denoted by Σg satisfies Σ3 I 1-' (() > O, where A > 0 implies that A is non-negative definite matrix (Lehmann & Casella, 1998 chapter 5)) and I(0) = - -E(?² log(p(x, 0))
91
92
93
96 is called the Fisher information matrix. For an arbitrary differentiable function g(-). an efficient
37 estimator of g(0) is an unbiased estimator when its covariance matrix equals I (0), where I='
98 is the fisher information matrix of g(0), i.e., the efficient estimator achieves the lowest possible
2

---

Under review as a conference paper at ICLR 2021
variance among all unbiased estimators. The efficient estimator can be achieved using factorization of (0) = Lg (0) - g())). if it exists (Rao) ( Lehmann & Casella (1998)) Based on these results, we derive relationship between the efncient estimation of unknowns and maximum likelihood classifier of ① and use auxiliary binary classifiers to apply that result in our
100
i log(p(xx /))/
102
103
proposed DBT method.
104
Parameter Estimations: Independent component analysis (Hyvärinen (1999) decomposes a multi- variate signal into independent non-Gaussian signals. ICA can extract non- non-Gaussian features from Gaussian noise. Additionally, there is a class of classifiers called generalized likelihood ratio functions that replaces the estimation of unknown parameters into the likelihood functions. This approach provides a huge improvement in the field of parametric classifiers, where the family of pdf of data given (Zeitouni al. (1992), Conte et al. (200T), Lehmann & Casella (2006)). Noise-contrastive estimation (NCE) (Gutmann & Hyvärinen (2000) involves training a generative model that allows model to discriminate data from fixed noise distribution. Then, this trained model can be used for training sequence of models of increasing quality. This can be seen as an informal competition mechanism similar in spirit to the formal competition used in the adversarial networks game. In Bachman et al. (211)) feature selection is proposed by maximizing the mutual information of the difference between features extracted from multiple views of a shared context. In that work, it is shown that the best results is given by using a mutual information bound based on NCE, The key difference between our method and NCE is that, we do not construct a generative model for noise. Instead of estimating the pdf of noise in NCE, we estimate the parameters of pdf of in in-domain dataset using an auxiliary class that has many common parameters in its pdf. Morcover, we show that the estimation of that parameters are sufficient statistic for classifier. We assume that the noise dataset is not pure and it has some similarity with the in-domain dataset, where it can help the feature selection layers select relevant (in-domain) features, e-g., see Fig. 3 Further, in our approach, do not construct the pdf of noise or in-domain data, instead we estimate its parameters directly, which is more efficient in terms of training, computation and also dimensionality reduction.
105
106
108
109
110
111
112
113
114
115
116
117
119
120
121
122
123
124
125
Auxiliary classifiers were introduced in inception networks (Szegedy et al. (2015) and used in (Lee) et al. (2015); (S. et al. (2016) for training very deep networks to prevent vanishing gradient problems.
126
127
Further, auxiliary classifiers were also proposed for early exit schemes (Teerapittayanon et al. (2016)) and self-distillation methods (Zhang el al. (201a b)). Such auxiliary classifiers tackle different problems by predicting the same target the final classification layer. In contrast, our proposed DBT method involves auxiliary binary classifiers that detect noise, interference, and/or background data from in-domain data points for improving the target classification accuracy.
128
130
131
ESTIMATION OF PARAMETERS OF PDF AND CLASSIFICATION
132
For (I. we define a deterministic discriminative function of O₁ denoted by t((-) such that the following conditions are satisfied:
133
135
4()) maps θ to real numbers such that (0) > 0, if 0 E O and t, (0) < 0 for 0 & O2
136
(-) is a differentiable function almost everywhere and Se (0) (0) < co, where iu denotes
137
the Lebesgue measure.
138
The following theorem shows the relationship of t.(.) and the log- ligllikel ratio of class C, versus
other classes. The proof' of Theorems 2 and 3 provided in the appendix.
139
140
Assume that the pdf p(x, 0) is differentiable with respect to f almost everywhere. If the efficient minimum variance and unbiased estimation of a deterministic discriminative function of O exists, then the log likelihood ratio of class against the rest of classes is an increasing function of the minimum variance and unbiased estimation O
Theorem
142
143
144
Directly from this theorem, it follows that the optimal classifier using the maximum likelihood for ( is given as follows d(x) = arg max (1.-...) k; (x)), where k 's are some increasing functions and (-)ss are the deterministic discriminative function of O,'s 's such that the efficient minimum variance and unbiased estimation for them exists. Based on this result, set of minimum variance and unbiased estimation of deterministic discriminative functions of O leads us to the maximum likelihood classifier. One approach is to directly estimate the deterministic discriminative functions, instead of maximizing the likelihood function. However, finding deterministic discriminative functions that have efficient minimum variance and unbiased estimation may not be feasible in practical problems,
145
146
148
150
3

---

Under review as a conference paper at ICLR 2021
Multi- Culss Class
Cramer Rao Bound
Better Estingtio
Theorem 2
Probs of Erree
In-domain
IM
Var O)
Clappfer
Butter
inguls
PEF
DDF
Theurem
ABC- noise inputs
Outputs
Binary Classifier output
The Theorem
ABC
Error that is achieved by the true likelihood function assuming that the unknown parameters are known
Figure 1: Visualizing Theorems 1,2 and 3
Figure A general schema of our proposed DBT method with PEF, DDF and ABC blocks
152
especially when the dimension of 0 increases. Theorems 2 and study the same relationship between the estimation of unknown parameters and the accuracy of classifiers for sub-optimal estimators and classifiers.
154
Theorem 2 Consider the output two classifiers for the ith class as follows: xx(x)= > 156
and (x) = other classes if 1x(x) < T, where j E {1,2}. where h;(x) is the estimation of deterministic discriminative function and is classification threshold. Assume that the cumulative distribution function of hy(x)'s have bounded inflection points, and also, the probability of true positive of (x) an increasing function of d(0), which is the deterministic discriminative function of class i, for all i. Further assume that for each the probability of false positive of rı (x) is less than the probability of false positive of r2(x) and the probability of true positive of Tr rx(x) is greater than the probability of true positive of (2)).. Then, there exists a hmin such that for all d(0) > hmin and all we have (x(()r(x)- d(O)) < <e) Pr([ba(x) - d(0)) < 6).
157
158
159
160
161
163
164
Theorem2 shows that a better classifier leads to a better estimation of d(0). In the next theorem, we show the dual property of this result.
186
Theorem Let O. be a Borel set with positive Lebesgue measure in ① for all m E {1. ... nn. (-) are given as follows r1 (x) m. i E Om and r2(x) = m, 02 E Om-
167
Assume that r-) and (-)
Also, assume that Pr( 0|| S <) > Pr[1 0 c), for all @ e - On and
168
169
then the probability of classification error r((-) is less than r2(()) where 0, and 02 are two different estimators of 0 E θ - θ m-
170
171
Theorem 3 proves that a more accurate estimator leads to a classifier that has a lower probability of classification error. From Theorem [ we can infer that sufficient statistic for developing the maximum likelihood classification is (x), which is the efficient minimum variance and unbiased estimation of the deterministic discriminative functions of O,'s denoted by (0)). In other words, the maximum likelihood classifier is function of only via the efficient minimum variance and unbiased
172
174
175
176
estimation (0). We can estimate t, (0) by replacing the estimation 0 in f-),, i.e., t,()) = (0))
where 0 is function of x. From the above theorems, we conclude that improving the estimation of unknown parameters of pdf of data can improve the accuracy of the classifier. On the other side, having a good classifier means having good estimator of unknowns of the pdf of input data. In many practical problems, the optimal maximum likelihood classifier may not be achievable, but the likelihood function of the classifier provides an optimal bound of the probability of error. In such cases, we can improve the accuracy of sub-optimal classifiers and that is the main focus of this paper. Fig. illustrates the proposed theorems visually.
178
179
180
181
182
183
PROPOSED METHOD: DETECTION BOOSTER TRAINING (DBT)
184
185
In this section, we propose the detection booster training (DBT) method based on the achieved theorems in the previous section improve the accuracy of deep networks. Specifically, we divide a deep model into two parts early and later layers. We apply a detector (detection here means detecting target pattern from noise/background) on the early layers of the neural network in order
187
188
4

---

Under review as a conference paper at ICLR 2021
Loss
Ver. Acc. (%) 98.96
Loss ResNet-100-P 100-AF ResNet-1000 100-DBT
Acc.
Acc. on H-set 00.04 21.00
ResNet-50-DBT (CE)
78.85
ResNet-50-DBT
99.12
81.11
Table 1:
Verification accuracy on LFW dataset for two different LaBC trained using CASIA Yi et al. (2014) dataset.
Table 2: Comparison of Rank-1 identification accuracy the IJB-B, with animal distrac- tors.
to improve the estimation of unknown parameters of the family of pdf (based on Theorem 2 A better estimation of unknown parameters corresponds to better feature representations in the early layers and these features are input to the rest of the layers to construct the deterministic discriminative functions (DDF) useful for the in-domain data classification (based on Theorem 3)
189
190
191
193
general schema for dividing a deep model into two sub-models namely PEF (parameter estimator functions) and DDF is depicted in Figure [ The early layers of the model estimate the unknown parameters of pdf of data while the later layers construct the discriminative functions essential for classification. Based on this scheme, we formally define the three main components of DBT as
194
195
197
follows:
parameter estimator functions (PEF): The sub-network from input layer to the kehh layer, where k is hyperparameter in the DBT approach.
189
200
auxiliary binary classification (ABC): Some additional layers are attached the end of PEF,
201
mapping the output of the kth layer to a one-dimensional vector.
202
deterministic discriminative functions (DDF): The sub-network from kth layer to the output of the
203
model. The output of model is a vector equal to the length of the number of classes n.
204
From Theorem we showed that unknown parameter estimation can be improved using a detection approach. During training, we apply binary classification on the early layers (PEF) of the model to improve the estimation of unknown parameters of pdf and subsequently provide rich feature vectors
205
206
for DDF. We define the auxiliary binary classification problem (ABC problem) as follows: Class (alternative hypothesis) of ABC problem denoted by Hi is set of all data points of classes 208
of C1 to Cz, i.c. 0 E - O,
210
Class 0 (null hypothesis) of ABC problem denoted by Ho is a dataset of data points from same distribution p(x, 0) but 0 & I We also define the dataset of Class 0 of ABC ABC-noise dataset, i.e., the ABC is given by the following hypothesis testing problem: H1 : E U O versus e,
212
Ho 0 & O In many practical problems, the noise, background or interference data related to the in-domain dataset have same type of probability distribution but different pdf parameters. Hence, using that dataset is cheap and adept choice for the null hypothesis of ABC.
214
215
216
The Auxiliary Binary Classification problem influences only the PEF and ABC units while the main classification problem with n classes updates the parameters of both PEF and DDF using in-domain data. Since the auxiliary classifier is only used during training, the inference model (IM) consists of only PEF and DDF and hence, there no additional computation cost during inference. We formulate the aforementioned method in the following notations and loss functions. Assume that x is data
217
218
219
220
221
point that belongs to Class C,, i E {1,-.. n) or Class Ho of ABC. Here, we define two type of labels denoted by LABC and IMC. where the subscription "MC" stands for multi-classes. So, if
223
and ICC = 1, else if x is ABC-noise data point, IABC = and IuC is None. Therefore, the loss function is defined as:
belongs to class C₁2, then ABCC
Liot LABC (QABC (x))))))))))))))e (ABC) + AAACC (MQDDDD (QPEr(x)),
(2)
225
where QPEP, ABC and QDDF are the functions of PEF, ABC and DDF blocks, respectively. We set the hyperparameter x to balance the two loss terms. It is seen that, the second term of the total loss is zero if /ABC = 0. LABC and LM are selected based on the problem definition and datasets. For classification, a simple selection for them can be binary cross-entropy and cross-entropy,
226
227
228
229
respectively. For a given task and deep neural network, the choice of & and LABC influences the
230
feature representation of early layers differently and consequently the accuracy of the model. We provide empirical studies in the next section to verify the same.
5

---

Under review as a conference paper at ICLR 2021
Figure 3: Maximally activated receptive fields of layer 15 of Inception-Reste ResNet-vl with (top row) and without (bottom row) DBT.
Figure 4: Examples of mis-identified faces along with their corresponding animal distrac- tors on the IJBB for AreFace.
EXPERIMENTAL STUDY OF DBT
232
233
FACE RECOGNITION
We conduct experiments on face recognition benchmarks and show that the DBT method learns rich features essential for face recognition. We also discover an important observation that current state- of-the-an art (SOTA) face recognition models very sensitive to non-face data, in particular, animal faces. Fig. shows a few examples of misidentified faces and their corresponding animal distractors from the UJBB dataset using the ArcFace (Deng al. (2019) model. We show that our DBT method not only improves the verification accuracy but also implicitly tackles this robustness issue of current models against non-face data. Implementation details are provided in the appendix.
235
237
239
241
We consider the PEF discussed in Section to be the first three layers of the model and DDF to be the of layers. Ablation studies the choice of PEF and DDF are provided in the supplementary material. We define MM in (2) as the SOTA ArcFace loss function proposed in (Deng et al. (2019). The ABC-noise is a non-face dataset containing 500K images that we collected from background patches of MS MV2 (Guo et al. (2016) (More details in Appendix). We experimented with two different loss functions for LABC. For the first one, since popular face recognition models (Deng et al.)
243
244
245
246
247
(0019; : Wang et al. (2018) use normalized output features and compute the losses on a hypersphere,
we select LABC as follows. Let py E R2d and Pas Rd denote the prototypes for faces and non- faces, respectively. Following (Mettes et al. (2019),, we constrain the face/non-face prototypes on
248
250
and normalize the output feature vectors for faces and non-faces such that [|p IP 1. We then define the LABC as,
diametrically opposite directions i.e cos (6 pppp,))
251
N
N --1- lp Png, N
CBBC = N log (cas((m)) Dy, +ma) ma) e cas 0g
1
Σ
Σ
where fy, and 82 correspond to the angles between the weights and the features for face and non-face labels, respectively; m1, m2, my are the angular margins; a denotes the radius of the hypersphere. For the second choice,
253
simple binary cross entropy for LABC. Table T shows that the verification accuracy on LFW (Huang et (200)) using (3) is 0.16% higher than simple cross entropy loss. This also shows that choosing task- aask-specific CBCC is essential in obtaining more accurate results. We use Eqn as the default for LABC in all our face recognition experiments, unless otherwise stated.
255
256
257
258
Table commares the verification accuracy of method versus the current SOTA method ArcFace on five different test sets, LFW, CPLFW (Zheng & Deng (2018)) CALFW (Zheng et al. (017))) CFP-FP (Sengupta et al. (2016)) and AgeDb-30 (Moschoglou al. (2017)). For the LFW test set,
259
260
261
we follow the unrestricted with labeled outside data protocol to report the performance. We trained Re and ResNet- 100 using ArcFace and DBT approaches on CASIA (small) and MSIMV2 (large) datasets, respectively. The results show that DBT method outperforms ArcFace on all datasets. Table 7 shows the angle statistics of the trained ArcFace and DBT models on the LFW dataset. Min. Inter and Inter refer to the mean of minimum angles and mean of all angles between the template embedding features of different classes (mean of the embedding features of all images for each class), respectively. Intra refers to the mean of angles between 2; and template embedding feature for each class. From Table 7 we infer that DBT extracts better face features and hence reduces the intra-class variations. Directly from Tables 3 and
262
263
264
286
288
289
we infer that first, DBT consistently improves the accuracy

---

Under review as a conference paper at ICLR 2021
Method
LFW
CALFW
CPLFW
CFPFP
AgeDb-30
ResNet-50-AF (ArcFace)
98.46
89.48
80.88
86.74
88.98
ResNet-50-DBBT
99.12
91.38
87.10
94.95
91.23
ResNet-110-AF (AreFace)
99.61
94.50
89.35
96.14
95.33
ResNet-100-DBT
99.75
95.13
90.70
96.90
96.16
Table 3: ArcFace vs. DBT-Arcaace: (rification(%) accuracy on LFW, CALFW, CPLFW, CFP-FP and AgeDb-30 of models ResNet-100 and ResNet-50.
on all test sets. Second, learning better features in the early layers is crucial to obtain rich face feature embeddings. Third, the achieved gain using DBT is more pronounced on models trained using a smaller (CASIA) dataset (it has fewer identities and images). This shows that DBT can address the issue of the lack of in-domain data using cheap ABC-noise data.
271
272
273
274
We also provide the results of training Inception- Inception-ReNNet-V and ResNet-64 models using DBT on MS MMV2 to show the generalization capacity of the DBT method. For the Inception-ResNet-V and ResNet-64, the PEF set to be the first six layers and the DDF is the rest of the model. We use large margin cosine loss (LMCL) Wang et al. (2018) for LMc and Cross entropy (CE) for LABC, Table 4 shows the verification accuracy on LFW for Inception-ResNet-VV and ResNet-64 models trained on MS1MV2 with and without DBT. The results show that DBT method is independent of model depth or architectures or loss functions and thereby consistently improves the accuracy compared to bascline results. Table 4 also compares the DBT method with state-of-the-art methods on LFW and YTF datasets. DBT method notably improves the baselines that are comparable to ArcFace and superior all the other methods. We were not able reproduce the results of the ArcFace paper using our Tensorflow implementation and dataset. We believe that using the original implementation and dataset from ArcFace will achieve superior results over the baselines on the benchmark datasets as evident from the results of our implementation. Finally, we compare the result ArcFace and DBT on UB-B and JJBC. in Table [5] It is seen that DBT provides notable boost on both IJB-B and UB-C C by considerable margin. DBT improves the verification accuracy as high as 1.94 % on IJB-B and 2.57 % on IJBC dataset at 10-4 false alarm rate (FAR). We plot the receptive fields of the top ten maximally activated neurons of an intermediate layer of the face recognition model to visualize the features learned using the DBT method. Fig. 3 shows that the receptive fields of layer 15 of the inception-resnet-v model trained using DBT attends to the regions of eyes, nose and mouth as compared to insignificant regions in the normal training method. This shows that DBT learns more theoretical claims.
275
276
278
280
282
283
284
285
286
287
288
289
290
291
293
discriminative features essential to face recognition, corroborating
295
To show that current SOTA models are not robust to animal faces, we performed a 1:N identification experiment with approximately 3000 animal distractors on the IJB-B (Whitclam al. (2017) dataset. We trained the face recognition model with about 500K non-face data which contains 200 animal faces. This is disjoint from the 3000 distractors used in the identification experiment. We collected the animal faces from web images using MTCNN (Zhang et al. (2016))) face detector which are the false positives from the face detector. Table 2 shows the Rank- identification accuracy of ResNet- 100
296
297
298
299
300
on IJB-B dataset, trained on MSIMV2 using the ArcFace loss (ResNet-100-AIl versus our DBT approach (Re-NEt-)00-DBBT, The third column of Table 2 denotes the accuracy on a hard subset of images (false positives from ArcFace model) on the IJB-B dataset denoted by H-set. Results of Table 2 show that current face recognition models are unable to discriminate ofitt--diitriuution (non- face) images from face images. Our M significantly (as high 21%) reduces the misidentification rate as compared to the ArcFace model which shows that DBT method inherently overcomes this issue while also improving face recognition accuracy.
302
304
306
307
308
IMAGE CLASSIFICATION
309
In this section, we evaluate ResNet- 110 and ResNext-101 models trained with and without DBT on image classification problem using 10, CIFAR-100, and ImageNet. We also show the power
310
311
of DBT to compensate for the smaller in-domain training set. For all implementations, PEF is defined to be the first three layers and DDF is the rest of the model. CABC and ExC are set to cross-entropy
313 loss. ABC-noise is the same data used in face recognition experiments. We follow the same training
configurations from (He al. 2016); Xie et al. (2017).
315 To study the efficacy of the DBT method in augmenting smaller in-domain training datasets, we
316 also trained ResNet- 100 and ResNext-101 using partial training data on CIFAR-10 and 100.
7

---

Under review as a conference paper at ICLR 2021
Model
Loss
LFW
Method
LFW
YTF
Inception Resnet
CE
99.45
Center Loss
99.28
94.9
Inception Resnet-DBT
99.50
Range Loss
99.52
93.7
Inception Resnet
LMCL
99.55
SphereFace
99.42
95.0
Inception Resnet-DBT
LMCL
99.60
SphereFace+
99.47
Resnet 64
CE 99.55
CosFace
99.73
97.6
Resnet 64-DBT
CE
99.63
ArcFace
99.82
98.02
Resnet 64
LMCL
99.65
ArcFace
99.61
97.31
Resnet 64-DBT
LMCL
99.68
ResNet-100-DBT
99.75
97.67
Table 4: Comparison of DBT models with SOTA methods on L.W and YTF. ArcFace refers to our areface implementation.
IJBB
IJB-C
Method
10 10-6
10
10-4
10-3 3
10
10-1
10
10
10-4
10
10
10
10-1
ArcFace 38.47 65.60
82.97
91.11
96.01
98.91
61.96
73.22
83.84
91.85
96.51
99.08
DBT
47.01
72.70
84.91
91.92
96.37
99.03
67.42
77.33
86.41
92.75
96.66
99.06
Table 5: 1:1 verification: ResNet-100: DBT vs. ArcFace on the IJB-B and IJB-C datasets
Method Top-1
Top-5
Method ArcFace ResNet-DBT 52.96
Min. Inter
Intra
Inter
ResNet
22.10
6.15
53.23
7.2
88.73
ResNet-DBT
21.82
6.02
7.16
88.52
Table 6: Top-1 and Top-5 error rates (%) on
Table 7: Comparison of inter and intra angles (degrees) for different methods on LFW.
ILSVRC15 benchmark for ResNet w/o DBT.
ResNet Models
CIFAR-I0
CIFAR-I00
ResNext Models
CIFAR-10
CIFAR-
He et al. (
5.84
22.15
Xie et al.
(017)
5.03
21.24
DBT (575)
5.25
21.53
DBT (5/5)
4.68
19.79
ResNet (4/5)
5.89
24.23
ResNext (4/5)
4.93
23.52
DBT (4/5)
5.36
23.98
DBT (4/5)
4.76
22.56
ResNet (3/5)
6.61
27.99
ResNext (3/5)
5.38
27.25
DBT (3/5)
5.44
26,81
DBT (3/5)
4.77
26.04
ResNet (2/5)
7.06
33.81
ResNext (2/5)
5.85
33.62
DBT (2/5)
5.94
31.95
DBT (2/5)
5.05
30.73
ResNet (1/5)
8.20
47.43
ResNext (1/5)
7.24
48.05
DBT (1/5)
6.86
43.65
DBT (1/5)
6.05
42.56
Table 8: Comparison of Top-1 error rates (%) for CIFAR-10 and CIFAR- datasets w/o DBT." denotes our implementation. (x/5) denotes the fraction of training data used for training that model.
Method
VoxC (top 1)
VoxC (top 5)
Librispeech
VCTK
ELSDSR
VGG-M CNN
80.5
92.1
93.12
82.52
79.98
VGG-M CNN-DBT
82.3
95.8
95.62
88.14
81.56
Table 9: Accuracy speaker identification (%) for different datasets.
Method
CIFAR-10
CIFAR-TOW
ResNet-Back 5.65
21.84
ResNet-DBT
5.25
21.53
ResNext-Bace
4.97
21.65
ResNext-DBT
4.68
19.79
Table 10: Comparison of top-1 error rates on CIFAR-10 and CIFAR-100 using an additional back- ground class vs DBT.
8

---

Under review as a conference paper at ICLR 2021
LFW 99.16 ResNet-DBT+mod 99.65
Method ResNet+mod
CALFW
CPLFW
CFP-HP
AgeDb-30
91.46 95.05
86.11 90.08
93.81 96.20
92.71
95.87
Table 11: Ablation study on the verification performance of adding background class to the model on MSIMV2 dataset.
317
We randomly selected a fraction of the training data to be our training set, c.g., k/5 of dataset means that we only used & fifth of total samples for training. From first row of Table we find that models trained with DBT show 0.59% and 0.35% improvement on CIFAR-10, 0.62% and 1.45% improvement on CIFAR-100 over baseline models for ResNet- 10 and ResNext- 101 architectures, respectively. Furthermore, using partial training data with our DBT method achieves superior results (as high as 5.49 % on ResNext (1/5) CIFAR- --00 as compared to normal training. showes the results on Imagenet. We see that DBT improves the accuracy by 0.28% on Top-1 accuracy. This shows that the DBT method consistently improves the results on both small and large datasets.
318
319
320
322
324
325
SPEAKER IDENTIFICATION
326
We consider the problem of speaker identification using the VGG-M (Chatfield et al. (2014) model.
We set PEF as the first two CNN layers and DDF as the remaining CNN layers. LABC and LMe are defined to be the cross-entropy loss. The ABC-noise is generated from the silence intervals of
328
329
VoxCeleb (Nagrani et al. (2017)) augmented with Gaussian noise with variance one. The input the
330
model is the short-time Fourier transformation of speech signals with a hamming sliding window the accuracies of VGG-M model trained with and without DBT on VoxCeleb, Librispeech (Panayotov et al. ))015), VCTK (Veaux et al. (2016)) and
331
of width 25 ms and step 10 ms.
332
333
ELSDR LL. 2004)) datasets. Table shows that the trained models using DBT significantly improves the accuracy (as high as 5.62 %) for all datasets. Implementation details are provided in the appendix.
334
335
336
MISCELLANEOUS EXPERIMENTS
337 In this section, we experiment with the naive way of using background data by considering non-faces as a separate class in the final classification layer. For face recognition, Table II shows the results of training with an additional background class on MS IMV2 dataset with and without using DBT. med refers to model trained with ArcFace loss and + classes where the additional class corresponds to the non-faces. ResNet-DBi -mod refers to a model trained with both DBT and the additional non-face class. We find that adding the additional non-face class hurts the performance of the model whereas ResNet-DBT+
339
341
342
343
improves the results significantly relative to ResNet+mod model. Since the non-face dataset is sampled from wide range of family of distributions compared with faces, it has a larger range of unknown parameters, then the sufficient statistic of them should be larger than the sufficient statisties of face data. Thus, when we restrict faces and non-faces on the surface of hypersphere, the non-face data is more spread on the surface compared with each of the
344
345
346
347
348
other face classes. We demonstrate this effect with the help of a toy example in Fig. Gin the appendix.
We also conduct this experiment on CIFAR- 10RRIO//IFA---OO and report it in Table 10 We see that naively incorporating the background class is inferior to DBT showing that DBT is an effective technique to utilize background data to boost the performance of classification models.
350
CONCLUSION
352
353
In this paper, we presented a detailed theoretical analysis of the dual relationship between estimating the unknown pdf parameters and classification accuracy. Based on the theoretical study, we presented
354
355
a new method called DBT using ABC-n data for improving in-distribution classification accuracy.
356
We showed that using ABC-noise data helps in better estimation of unknown parameters of pdf of
357
input data and thereby improves the feature representations and consequently the accuracy in image classification, speaker classification, and face recognition benchmarks. It also augments the training
358
359
data when only limited labeled data is available by improving accuracy. We showed that the concept of DBT is generic and generalizes well across domains through extensive experiments using different model architectures and datasets. Our framework is complementary to existing training methods and
361
hence, it can be casily integrated with current and possibly future classification methods to enhance
363
accuracy. In summary, the proposed DBT method a powerful technique that can augment limited
364
training data and improve classification accuracy in deep neural networks.

---

Under review as a conference paper at ICLR 2021
REFERENCES
386
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. Corrado, A. Davis, J. Dean, M. Devin, and S. Ghemawat and. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015.
368
369
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pp. 15535-15544 2019.
370
T. L. Berg, A. C. Berg, J. Edwards, and D. A. Forsyth. Who's 's in the picture. NeurlPS, 2004.
David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow. Understanding and improving interpolation in autoencoders via an adversarial regularizer. ICLR, 2019.
374
375
Ben-Zion Bobrovsky, E Mayer-Wolf, and M Zakai. Some classes of global cramér-rao bounds. The Annals of Statissicss, pp. 1421-1438, 1987.
378
377
Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Return of the devil in the details: Delving deep into convolutional nets. BMVC, 2014.
Emesto Conte, Antonio De Maio, and Giuseppe Ricci. Girt-based adaptive detection algorithms for range-spread targets. IEEE Iransactions on signal processing, 49(7): 133)-1333-,338, 2001.
380
381
J. Deng, J. Guo, X. Niannan, and S. Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Computer Vision and Pattern Recognition (CVPR), 2019.
382
383
Richard Duda, Peter Hart, and David Stork. Puttern classification. John Wiley & Sons, 2012.
384
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams,Johe Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111 (1)::98-136, 2015.
386
387
F. F. Li, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshop, pp. 178-178. 2004.
386
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. ICLR, 2018.
391
392
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurlPS, pp. 2672-2680, 2014.
393
394
Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao. Ms-celeb- 1m: A dataset and benchmark for large-scale face recognition. ECCV, 9907:87-102, 2016.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
397
389 K. He. X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. Computer Vision and Pattern Recognition, pp. 770-778, 2016.
G. B. Huang, M. Ramesh, T. Berg, and F. Learned-Miller Labeled faces in the wild: A database for studying face recognition in unconstrained environments. In Technical Report, 2007.
402
403
Aapo Hyvärinen. Survey on independent component analysis. 1999.
404
S. loffe and C. Szegedy. Batch normalization: internal covariate shift. International Conference on Machine Learning, 37:448 456, 2015.
Accelerating deep network training by reducing
405
406
Diederik Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014.
407
Feng L. Speaker recognition, informatics and mathematical modelling. Technical University of Denmark, DTU, 2004.
10

---

Under review as a conference paper at ICLR 2021
C-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu. Seeply-Supervied Nets. Proceedings of Machine Learning Research (PMLR), 38:562-570, 2015.
410
Youngjo Lee, John A Nelder, and Yudi Pawitan. Generalized linear models with random effects: unifted analysis via H-likelihood, volume 153. CRC Press, 2018.
412
413
E. L. Lehmann and G. Casella. Theory of point estimation, 1998. 2ndn ed.
414
Erich L Lehmann and George Casella. Theory of point estimation. Springer Science & Business Media, 2006.
415
416
Tsung-Vo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
418
Bernard Lindgren. Statistical theory. Routledge, 2017.
B. Maze, J. Adams, J. A. Duncan, N. Kalka, T. Miller, C. Otto, A. K. Jain, W. T. Niggel, J. Anderson, Cheney, and P. Grother. larpa janus benchmark c: Face dataset and protocol. International Conference Biometrics, pp. 158-165, 2018.
421
423
Mettes, E. van der Pol, and Snoek. Hyperspherical prototype networks. NeuRIPS, 01 2019.
424
S. Moschoglou, A. Papaioannou, C. Sagonas, J. Deng, I. Kotsia, and S. Zafeiriou. Agedb: the first manually collected, in-the-wild age database. CVPR Workshop, 2(3):5, 2017.
425
426
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: large-scale speaker identifica- tion dataset. arXiv preprint arXiv:1706.08617 2017.
428
M. Noroozi and Favaro. Unsupervised learning visual representations by solving jigsaw puzzles. ECCV, 2016.
429
M. Oquab. L. Bottou, I. Laptev, and J. Sivic. Learning and transferring mid-level image representa- tions using convolutional neural networks. CVPR, pp. 1717-1724, 2014.
431
432
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: an asr corpus based on public domain audio books. International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210, 2015.
433
BLS Prakasa Rao. Cramer-rao type integral inequalities for estimators of functions of multidimen- sional parameter. Sankhyā: The Indian Journal of Statistics, Series A, pp. 53-73, 1992.
436
437
Ali Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off-the- shelff an astounding baseline for recognition. CVPR Workshops, 2014.
438
439
O. Russakovsky, J. Deng, H. Su, Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and FFF Li. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJC), 115(3):111-252, 2015.
440
441
Christian S., Vincent V., Sergey I,, Jon S., and ZB W. Rethinking the inception architecture for computer vision. CVPR, 2016.
443
444
S. Sengupta. J. Chen, C. Castillo, M. Patel, R. Chellappa, and D. W. Jacobs. Frontal to profile face verification in the wild. In Winter Conference on Applications of Computer Vision (WACV), pp. 1-9, 2016.
445
446
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929-1958, 2014.
448
449
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and
450
A. Rabinovich. Going deeper with convolutions. CVPR, pp. 1-9, 2015.
451
Tecrapitayanon, McDanel, and H. T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. ICPR, 2016.
11

---

Under review as a conference paper at ICLR 2021
Christophe Veaux, Jumichi Yamagishi, Kirsten MacDonald, al. Superseded-cstr vetk corpus: English speakes corpus for cstr voice cloning toolkit. University of Edinburgh. The Centre for Speech Technology Research (CSTR), 2016.
454
H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li, and W. Liu. Cosface: Large margin cosine loss for deep face recognition. CVPR, pp. 5265-5274, 2018.
457
458
C. Whitelam, E. Taborsky, A. Blanton, B. Maze, J. Adams, T. Miller, N. Kalka, A. K. Jain. J. A. Duncan, K. Allen, J. Cheney, and P. Grother. larpa janus benchmark-b face dataset. CVPR Workshops, pp. 592-600, 2017.
460
461
L. Wolf, T. Hassner, and Maoz. Face recognition in unconstrained videos with matched background similarity. CVPR, pp. 529-534, 2011.
463
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. CVPR, pp. 5987-5995, 2017.
464
465
D. Yi, Z. Lei, Liao, and Z. Li. Learning face representation from scratch. arXiv, abs/ 1.7923, 2014.
466
467
J. Yosinski, J. Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? NIPS, 2014.
468
469
Ofer Zeitouni, Jacob Ziv, and Neri Merhav. When is the generalized likelihood ratio test optimal? IEEE Transactions on Information Theory, 197)) 1992.
470
471
K. Zhang, 7. Zhang, Z. Li, and Y. Qiao. Joint face detection and alignment using multi-task cascaded convolutional networks. Signal Processing Letters, 199901)449--503 2016a.
472
473
L. Zhang, J. Song, A. Gao. J. Chen, C. Bao, and K. Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. ICCV, 2019a.
474
475
Linfeng Zhang, Zhanhong Tan, Jiebo Song, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Scan: A scalable neural networks framework towards compact and efficient models. NeurlPS, 2019b.
Richard Zhang, Phillip Isola, and Alexci Efros. Colorful image colorization. ECCV, 2016b.
T. Zheng and W. Deng. Cross-pose Ifw: A database for studying cross-pose face recognition in un- constrained environments. Technical Report, Beijing University of Posts and Telecommunicattons, 2018.
479
T. Zheng, W. Deng, and J. Hu. Cross-age Ifw: A database for studying cross-age face recognition in unconstrained environments. arXiv, abs// 0897778888.,,,.... 2017.
482
483
APPENDIX
484
IN-DOMAIN FAMILY OF PDES AND THE EXTENDED FAMILY OF DISTRIBUTIONS
In this section, diseuss about background/noise and in-domain data points and their corresponding distributions to clarify the definition those concepts in this paper. Consider a random vector denoted by Assume that the corresponding distribution is Gaussian with mean and variance given by & and 0 = 1, respectively. Now, assume that we observed x = S + n. where the pdf of n is assumed to be Guassian with zero mean and variance o hence the pdf of is Gaussian with mean a and variance
486
487
488
489
1 + o² Here, n is the background or noise data and the vector of unknowns is given by, O = [a. o. > 0}. If
490
The in-domain family of pdfs for x is then given by Px (a 1 1+ aa
491
include the family of pdf of to Px, then we can extend Px as P = (N(a, 1+l,, o RR of 0). So P the union of family of pdfs of in-domain data points and noise/background data. From estimation theory, know that the sufficient statistics and the unknown parameters of P can also represent the sufficient statistics and the unknown parameters of Px. In other words, an estimation of by comparing it with threshold.
493
495
help detect if the observed data point is from Thus, estimating the unknown parameters of the family of pdfs using P can provide more information about the observed data useful for tasks such as classification.
497
In general, we can assume that a generalized family of pdfs is given by the family pdf of noise or background along with the family of pdfs of in-domain data. Hence, estimating from the extended
12

---

Under review as a conference paper at ICLR 2021
Figure 5: In-domain data point versus background data point. The background is cropped from the in-domain image and provides complementary information to the main data, thereby we can provide better estimation of the pdf parameters of in-domain data.
family of distribution can provide more information about the in-domain distribution. Let us consider that the pdf of in-domain data points is given by Px (x [0., 0,,]) and the pdf of noise/background is given by Pm (x, ),), so the extended pdf can be represented by
h(pn (x, 0n), (x 10., 0nl)),
where is a function that combines two pdfs general structure. So a general family of distribution can be denoted as follows:
P {h(pn (x, 0n), Pxx [e., 0,))) (0,, 0nl Or.n.
where 0 is defined as a new set of parameters in a higher dimension and Os.n are set of all possible 489
a, 0n] that belongs Pn and Px- 'hee extended family of pdf provides more information about the nuisance parameters of pdf of in-domain datapoints. Inspired by this observation, we develop our detection booster training method using background/noise data. Figure [ shows example of background and in in-domain data point.
501
502
503
504
PROOF OF THEOREM 1
505
Let f.( denote deterministic discriminative function of θ, Since the efficient minimum variance and unbiased estimation of (e))) exists, we have
506
i (n(p(x 0)) (0)()()))) - 4())),
(4)
a1.
where (x) is the minimum variance and unbiased estimation of (0) using the data point x and Ie, (x) is the Fisher information function of (0). which is given by
(0) (0) T
(
I
0,
00
0e
where T' denotes the transpose and I(0) is the Fisher information matrix of 0. Now we show that
507
508 the likelioel ratio is an increasing function in (x).. Note that 4r, (0) 2 0 Leehmann & Casella
509
(006
510
On the other hand, we have d d 0)) =-
2 Unipix 01) 86,
d0,, therefore,
a ln((px 0))
(n(p, 0)) (0)
In(p(x,)). -xxxx)=))
Σ
d0; = Σ
d0, =
00,
(t:(0) 00,
Aln(p(x, 0)) (t)
(0) d0, / (I,(0) - (()))) Σ
(l2
a0, (x) ((0) (5)
08,
00,
where the third equality is archived based on the third property of 4.()) in its definition and the forth equality is given by replacing [ k((x) is the constant of integration. Finally, the last equality is given by defining the following terms
(0)) / f L. (0) Σ
(
3(0) =- / Lt, (0),, Σ
04.(0) 00,
do,,
do,,
(6)
002
511 thus do(l) (
Lr, (0) > 0, i.c. a0)) is increasing in & (0). Since, t; is a deterministic discriminative function of e; so for each j # i and 0, O and Og Oj, have (, 1400),, therefore
13

---

Under review as a conference paper at ICLR 2021
(0,)) a(O,). The later inequality is achieved based on the increasing property of a()) with respect to (0).
514
Using
5 the log likelihood ratio of class i against the rest of classes is given by LLR In(p(x, 0.)) In(p(x, 0j)), so we have LLR (a(O.) a(0.)){i(x) (3(0.) 3(0))) )) LLR
depends on x only via (x) and since for each j # and 0g E θ and 02 & θ a(,)) a >
517
then LLR is increasing in (x). O
518
519
PROOF OF THEOREM
The probability of true positive of class of r is given by
Pep.i.j = (ra(h)(x) > 7) 1 (t,)))
where Fin (-) denotes the Cumulative distribution function (CDF) of h2g Since the probability of true positive of class of r, is greater than r2 for all т, (T, < F2. (T), for all r. Now we define a function as follows
u(T, 0) = (T) - Fia
Since the CDFs are increasing are bounded, there is an Amin such that, for 5 > hmins such that u(T, 0) is monotonically decreasing function in r. Thus for any O that satisfies d())) ) Amin we have
and tend and the number of inflection points of these CDFs
u(d(0) e, 0) < u(d(O) e, 0). Replacing u(h, = F20 (b) Fio in the last inequality, we have
520
F2, (d(()) + c) - Fi, (d( + c) < F2, (d(0) - c) Fi, (d(0) - -c c) - F2. (d(() + c) F2. (d( c) < (d + <) (d(0)) -c).
(7)
(8)
521
Based on the definition of CDF, we have
Pro (12(x) d(0)) <x) 6) = Pro (d(0) c < h2(xx) < d(0) +0)
Pro (d(0) < hx)) < d(0) +e) Pro - -xx <e).
(9)
522
523
PROOF OF THEOREM
524
First, we prove the following claim,
525
Claim: For any open set, there exists a set of disjoint countable open balls such that their union equals
526
the origin open set.
527
Proof of claim: Consider an open set O, and also consider To E O, such that B( ro) C O and ro is the greatest possible radius between all possible open balls in O, where B( ro) the
528
529
open ball with radius To at point Ty- Now, we define I1 E O B(Io, ru), where B( ro] is
530
the closure of To), as the point with greatest radius in O - B(x,, ro) and similarly 24 E U B(a rk) such that B(I;, rz) provides the greatest radius in O - Uf B(ra, rx). So we have = o B(rr, rk). This is because, the latest equality not valid, then there exists an open ball in O Up B(xs, rk) hence another open ball with greatest radius will be added to UP o Bar, rx). which has a contradiction with the definition of U (x rk). The claim is proven
532
533
534
535
at this point.
Now, we show the true positive probability of r1 is greater than r2. Let O be the set of interior points of Om, then, there exists union of disjoint open balls such that O',, = U (T)) From assumptions in the theorem, we have Pr(|0, 0|l < <) Pr( 01 S e), then
(0 E B( rk)) 2 Pro (02 E B( rw)):
536 where 0 E Om Based on the claim we have
(ro E 6%,,) 2 Pro E ('m))
(10)
Moreover, based definition of r;, the true positive probability of class m is given by
Ptp.i Pro @m) Pro 6%,)) Pro O. Omm),
14

---

Under review as a conference paper at ICLR 2021
AsC-taise Ipptt
PEF
rechs OPS
AfC-nnttn Inpute
PEF
heury
Theemn hete cleaifer the orizetioo panrecter of pdf
ABC
cul dls
The output here the in that Theoren: 2 guartnteed
Theoren 5 guaranties that the multidees, desufier has bette fommrncc,,. constructed from better estimetion unenown paremeters
Ourpea
PEF
DDF
Ourps
ABC nolse inputs
A0C Inpur....
ABC
Ourel
ABC
Nerr
Figure 6: Relationship between the theorems in Section and the proposed method in Section 4
100 175-
1.0
The noise class
Figure 7: Feature distance between different classes with and without additional background class for a toy example. Left: Contains classes and the feature separation is visibly larger; Right: Contains an additional noise class that decreases the feature distance for all the other classes.
for i 1,2. Additionally, from the Cauchy-Schwarz inequality, we have
Pro(O, E mm O'm)) & ( - (m=)=0)
537
So, Ptp. (ro E O')) and from 10 the true positive probability of class of ri is greater than
538 F1.
The error probability of ry is given by Penj = - E PiPtpsij. where P: is the prior probability of class i. Therefor, Per,1 S Per.2-
539
540
542
CONNECTING THE THEOREMS WITH THE PROPOSED METHOD
543
Fig. shows the connection between the proposed theorems and the approach. In part 1, Theorem 2 connects the estimation of unknown parameters to the auxiliary classifier. In part 2, the learned
545
features are passed to decision making network (result of Theorem 2). In part 3, Theorem guarantees that the multi-class classifier outperforms other classifiers, because is using the features from a better estimation of unknown parameters of pdf.
546
547
548
519
TOY EXAMPLE:
550
We demonstrate the effect of adding background class to the original classifier with a toy example and visualize it in Fig. In this example, the input is sequence of binary bits (+1 and -1) with length 3 in white Gaussian noise, the classifier is constructed using two fully connected layers with sigmoid and the last layer is normalized unit circle. As seen from Fig. 7 adding additional noise class visibly reduces the feature separation between all the other classes.
552
554
IMPLEMENTATION DETAILS
556
FACE RECOGNITION
557
We use Tensorflow (Abadi et al. (0055) to conduct our experiments. We train with batch size of 256 on two NVIDIA Tesla V100 (32G) GPUs. We train our models following small (less than IM training images) and large (more than 1M training images) protocol conventions. We use (Yi et al. (2014)) dataset for small protocol and MSIMV2 dataset for the large protocol. We use ResNet-50 (He et al. (2016)) and ResNet- 100 models for small and large protocols, respectively. The PEF is selected the first three layers. Following (Deng al. (2019)), apply
559
581
15

---

Under review as a conference paper at ICLR 2021
BN (loffe & Szegedy (2015) ) dropout (Srivastava al. (2014) to the last feature map layer followed by a fully connected layer and batch normalization to obtain the 512-D embedding vector. We set the feature scale parameter to 64 following (Wang et al. (0118); Deng et al. (2019) and set the margin parameters (m1 M2, ma) to (1, 0.5, 0), respectively. For small scale protocol, we start the learning rate at 0.01 and divide the learning rate by 10 at 40K, 80K, and 100K iterations. We train for 120K iterations. For large scale protocol, we start the learning rate at 0.01 and divide the learning rate by 10 at 80K, 100K, and 200K iterations. We train for 240K iterations. We use Momentum optimizer and set the momentum to 0.9 and weight decay to 5e-4. We use the feature centre of all images from a template or all frames from a video in order report the results on IJB-B, B. IJB-C and
684
566
588
589
570
571
YTF datasets. For ABC-noise data, we cropped background images patches from MS MSV (Guo)
573
et al. (2016)) dataset and cropped hard examples from the Caltech-101 (F. Li et al. 2004)) dataset
574
plus few open sourced images (animal faces) using MTCNN (Zhang et al. (20T 6a
face detector.
575
We generated roughly 500K non-face images for training the ABCs.
576
SPEAKER IDENTIFICATION
577 L2 loss and dropout with a rate of 0.2 are applied during training for generalization. The ABC-eoise is collected form silence intervals of the VoxCeleb dataset, where an energy-based voice activity detection (VAD) applied to detect silence intervals. To augment the ABC-noise, Gaussian noise is added to the silence intervals. Each batch size is set to 64 and the optimizer is ADAM with a learning rate of 0.001. The VoxCeleb dataset is trained for epochs and the other datasets are trained for epochs.
579
581
583
LFW AND YTF DATASETS
LFW database contains the annotations for 5171 faces in set of 2845 images taken from the Faces in the Wild data set (Berg et al. (2004). YouTubeFaces (Wolf et al. (011)) contains 3,425 videos of 1,595 people. Following the standard convention, report the results 5000 video pairs using unrestricted with labeled outside data protocol.
585
587
586
IJB-B B AND IJB-C DATASETS
589
The IJB-B contains 1,845 subjects with 21.8K still images and 55K frames from 7,011 videos. In total, there are 12,1 templates with 10,270 genuine matches and 8M impostor matches. The IJB-C
580
581
dataset (aaze et al. (0188) is a further extension of JJ-B, having 3,531 subjects with 31.3K still
592
images and 117.5K frames from 11,779 videos. In total, there are 23, 124 templates with 19,557
593
genuine matches and 15, 639K impostor matches.
16


Reviews:
[{"review_id": "hUDJrjl8ve", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission proposes a training strategy that leverages background/noise data to learn robust representations. Experimental results show that the use of this strategy generally leads to improved performance. \n\nQuality: The quality of this submission is lacking in several respects. \n\n1) You abstract states that there has been not much work done on using background/noise data whilst estimating model parameters. This is not true. What about NCE and all the follow up work?\n2) What you are describing is a feature extractor that is using one objective and classifier that uses another objective. There is prior work in this area. However, your related work section suggests something entirely different. \n3) You are cramming way too much into 8 pages. \n4) Section 3, 4 and 5 do not work together. Section 3 talks about Cramer-Rao bounds and Lebesgue measures while section 4 shows something that looks like a multi-task training objective. Section 5 does not have baselines other than the trivial one of not using your approach. \n\nClarity: The clarity in this submission is lacking. Given Section 4, the point of sections 2 and 3 is very unclear. \n\nOriginality: It is hard to judge originality of this submission given the way it is presented. Which theorems are your contribution? Which of many statements you make in sections 2, 3 and 4 are your contributions? \n\nSignificance: It is hard to judge the significance of this submission given the way it is presented. \n\nPros: This submission contains evaluation across many modalities. \n\nCons: One-sided presentation with links to existing works missing. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "WiIvy1uzJ3Q", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a training method for classification, with the goal of training with less data. The proposal is to train an auxiliary classifier at the same time. The auxiliary classifier and the main classifier share the early layers. The auxiliary classifier is a binary classifier that discriminates training data versus background/noise data. The proposed method is evaluated on image and speech classification tasks.\n\nOn the positive side, the experimental results show consistent benefit across tasks. The facial recognition comparisons should be taken with a grain of salt, because of how the background data were collected (see page 7 and appendix). Other than that, the results seem quite consistent.\n\nHowever, I have substantial concerns. The experiments are not sufficient to support the claims: there is little results on training with less data; certain critical experiments are missing; some comparisons are not fair. The theoretical content is detached from the proposed method and hence not important. I'll elaborate in the following.\nThere are three angles to look at the proposed method:\n\n1. Data augmentation. Part of the proposed method is building an extra class of training data that are background or noise. One example is given in Figure 5. It seems feasible for image and speech tasks.\n\n   A critical point, that is missing from this paper, is how would this data augmentation work by itself without the auxiliary classifier. In other words, what if we just train a classifier with n+1 labels, where the extra label's training data are the background or noise data points? At inference time, one simply uses the first n logits. How much of the reported benefits would this simplified method achieve? This would show how important is the data augmentation versus how important is the auxiliary classifier.\n\n   The appendix stated that the background/noise data include animal faces collected from open source images. Therefore, the discussion around Figure 4 and Table 2, which claims robustness advantage over ArcFace, is invalid. It's not a fair comparison when the training data is expanded with a particular emphasis.\n\n2. Semi-supervised learning with less labeled data. It seems that the most natural and meaningful task is where the auxiliary classifier is trained on all in-domain data, both labeled and unlabeled, while the main classifier is trained on a small number of labeled data. It's important to compare with existing methods in this setup.\n\n   Unfortunately this paper does not include such experiments. Table 8 includes reduced-training-set experiments on CIFAR, however 3/5 of the training set is still way too big, and the remaining 2/5 are not utilized as unlabeled in-domain data.\n\n3. Regularization of the early layers. This is the stated rationale for the proposed method. The auxiliary classifier is designed to encourage the early layers to learn more meaningful features, and Section 3 supports this relation.\n\n   Section 3 is very difficult to read. The conclusion is that the cross entropy loss of the auxiliary classifier is positively correlated with the quality of the estimation of unknown parameters of the data distribution, and therefore training with the joint loss of equation (2) is implicitly related to a good estimation of parameters, which in turn is related to better feature representations in the early layers (an implicit assumption here is that good features for the auxiliary binary classification are also good features for the multi-class classification, which could be a question by itself).\n\n   This relation, however, is qualitative rather than quantitative. In other words, neither training nor inference of the proposed method utilizes the content of Section 3. Section 3 mostly serves as a motivational argument. The auxiliary classifier idea is intuitive by itself, and it seems that Section 3 can be removed without affecting the core content of this paper.\n\n   The stated goal of regularization is to learn with less data. However the only results on this subject seems in Table 8 where 3/5 of CIFAR data are used. Table 8 is not the best setup, as mentioned earlier in the second point. Even not considering semi-supervised learning, 3/5 of the CIFAR training data are too many for a training-with-less-data experiment.\n\nMinor points:\n-- Figure 3 is not a fair comparison. With the joint training, the roles of ResNet blocks are likely shuffled. Some information that gets processed in later layers during normal training may get moved to early layers in joint training, and vice versa. Hence comparing at a fixed layer is not meaningful. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8-PLtgkXG18", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "To improve multi-class classification problem using DNNs, authors propose to do multi-task learning of solving another auxiliary tasks which tells if data points are just noise/distractors or not. It is common to have some data which is non-representative of good training samples but they can be used (under DBT) to make first few layers (and thus all) of network more robust.\nResults are satisfactory but:\n1. Can this idea not be done simply by having one more classification unit which detects if x is in-domain or noise? I believe that experiment should be done. That way, whole network can be trained for this. I believe my suggestion is different from \"DBT(ABC-last)\" of appendix.\n2. The multi-task formulation should have a hyper-parameter lambda in Equation 2. Is there a reason it is set to 1? I think it is standard to experiment with that. Perhaps equal weightage to two tasks is not optimal.\n3. It is not clear if I use a regularization technique like MixUp or label smoothing, will I still see improvement?\n\nI am not much aware of related work BTW.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper the authors propose an approach to improving the accuracy of the classification problem based on deep neural networks by detecting the in-domain data from background/noise.  The strategy is designed in such a way that the detector and the classifier share the bottom layers of the network.  Theoretical proof is given and experiments are conducted on a variety of datasets.  The novelty of the work is to come up with a better estimate the pdf of the data and use it to help the classification based on the deep neural networks.   There are concerns raised by the reviewers regarding the related work, the exposition and the experimental design.  After the rebuttal from the authors, which is meticulous, some of the issues unfortunately still stand.  The paper needs to make a stronger case in order to be accepted, especially, for instance, the theoretical and empirical comparison with the existing techniques sharing the similar idea. 
Decision: Reject