Complement Objective Training

1 INTRODUCTION . Statistical learning algorithms work by optimizing towards a training objective . A dominant principle for training is to optimize likelihood ( Mitchell et al. , 1997 ) , which measures the probability of data given the model under a specific set of parameters . The popularity of deep neural networks has given rise to the use of cross entropy ( Kullback & Leibler , 1951 ) as its primary training objective , since minimizing cross entropy is essentially equivalent to maximizing likelihood for disjoint classes . Cross entropy has become the standard training objective for many tasks including classification ( Krizhevsky et al. , 2012 ) and sequence generation ( Sutskever et al. , 2014 ) . Let yi ∈ { 0 , 1 } K be the label of the ith sample in one-hot encoded representation and ŷi ∈ [ 0 , 1 ] K be the predicted probabilities , the cross entropy H ( y , ŷ ) is defined as : H ( y , ŷ ) = − 1 N N∑ i=1 yTi · log ( ŷi ) = − 1 N N∑ i=1 log ( ŷig ) ( 1 ) where ŷig represents the predicted probability of the ground-truth class for the i th sample . Train- ing with cross entropy as the primary objective aims at finding θ̂ = argminθ H ( y , ŷ ) , where ŷ = hθ ( x ) , hθ is a neural network and x is a sample . Although training using the cross entropy as the primary objective has achieved tremendous success , we have observed one limitation : it exploits mostly the information from the ground-truth class as Eq ( 1 ) shows ; the information from complement classes ( i.e. , incorrect classes ) has been largely ignored , since the predicted probabilities other than ŷig are zeroed out due to the dot product calculation with the one-hot encoded yi . Therefore , for classes other than the ground truth , the model behavior is not explicitly optimized — their predicted probabilities are indirectly minimized when ŷig is maximized since the probabilities sum up to 1 . One way to utilize the information from the complement classes is to neutralize their predicted probabilities . To this end , we propose Complement Objective Training ( COT ) , a new training paradigm that achieves this optimization goal without compromising the model ’ s primary objective . Figure 1 illustrates the comparison between Figure 1a : the predicted probability ŷ from the model trained with just cross entropy as the primary objective , and Figure 1b : ŷ from the model trained with both primary and complement objectives . Training with the complement objective finds the parameters θ that evenly suppress complement classes without compromising the primary objective ( i.e. , maximizing ŷg ) , making the model more confident of the ground-truth class . Figure 2 further illustrates the embeddings of CIFAR10 images calculated from ResNet-110 using two training paradigms : cross entropy and COT . An embedding of an image is the vector representation computed by the ResNet-110 model , before taking the softmax operation . Compared to Figure 2a , the clusters in Figure 2b seem to have clean and separable boundaries , leading to more accurate and robust classification results . The experimental results later in Section 3 further confirm this observation . Complement objective training requires a function that complements the primary objective . In this paper , we propose “ complement entropy ” ( defined in Section 2 ) to complement the softmax cross entropy for neutralizing the effects of complement classes . The neural net parameters θ are then updated by alternating iteratively between ( a ) minimizing cross entropy to increase ŷg , and ( b ) maximizing complement entropy to neutralize ŷj 6=g . Experimental results ( in Section 3 ) confirm that COT improves the accuracies of the state-of-the-art methods for both ( a ) the image classification tasks on ImageNet-2012 , Tiny ImageNet , CIFAR-10 , CIFAR-100 , and SVHN , and ( b ) language understanding tasks on machine translation and speech recognition . Furthermore , experimental results also show that models trained by COT are more robust to adversarial attacks . 2 COMPLEMENT OBJECTIVE TRAINING . In this section , we first define “ Complement Entropy ” as the complement objective , and then provide a new training algorithm for updating neural network parameters θ by alternating iteratively between the primary objective and the complement objective . 2.1 COMPLEMENT ENTROPY . Conventionally , training with cross entropy as the primary objective aims at maximizing the predicted probability of the ground-truth class ŷg in Eq ( 1 ) . As mentioned in the introduction , the proposed COT also maximizes the complement objective for neutralizing the predicted probabilities of the complement classes . To achieve this , we propose “ complement entropy ” as the complement objective ; complement entropy C ( · ) is defined to be the average of sample-wise entropies over complement classes in a mini-batch : C ( ŷc̄ ) = 1 N N∑ i=1 H ( ŷic̄ ) = − 1 N N∑ i=1 K∑ j=1 , j 6=g ( ŷij 1− ŷig ) log ( ŷij 1− ŷig ) ( 2 ) H ( · ) is the entropy function . All the symbols and notations used in this paper are summarized in Table 1 . One thing worth noticing is that this sample-wise entropy is calculated by considering only the complement classes other than the ground-truth class g. The sample-wise predicted probability ŷij is normalized by one minus the ground-truth probability ( i.e. , 1−ŷig ) . The term ŷij/ ( 1−ŷig ) can be understood as : conditioned on the ground-truth class g not happening , the predicted probability to see the class j for the ith sample . Since the entropy is maximized when the events are equally likely to occur , optimizing on the complement entropy drives ŷij to ( 1− ŷig ) / ( K − 1 ) , which essentially neutralizes the predicted probability of complement classes as K grows large . In other words , maximizing the complement entropy “ flattens ” the predicted probabilities of complement classes ŷj 6=g . We conjecture that , when ŷj 6=g are neutralized , the neural net hθ generalizes better , since it is less likely to have an incorrect class with a sufficiently high predicted probability to “ challenge ” the ground-truth class . 2.2 TRAINING WITH COMPLEMENT OBJECTIVE . Given a training procedure using a primary objective , such as softmax cross entropy , one can easily adopt the complement entropy to turn the procedure into a Complement Objective Training ( COT ) . Algorithm 1 describes the new training mechanism by alternating iteratively between the primary and complement objectives . At each training step , the cross entropy is first calculated as the loss value to update the model parameters ; next , the complement entropy is calculated as the loss value to perform the second update . Therefore , additional forward and backward propagation are required in each iteration when using the complement objective , making the total training time empirically 1.6 times longer . 3 EXPERIMENTS . We perform extensive experiments to evaluate COT on tasks in domains ranging from computer vision to natural language understanding and compare it with the baseline algorithms that achieve state-of-the-art in the respective domains . We also perform experiments to evaluate the robustness of the model trained by COT when attacked by adversarial examples . For each task , we select a stateof-the-art model that has an open-source implementation ( referred to as “ baseline ” ) and reproduce their results with the hyper-parameters reported in the paper or code repository . Our code is available at https : //github.com/henry8527/COT . 3.1 BALANCING TRAINING OBJECTIVES . In theory , the loss values between the primary and the complement objectives can be in different scales ; therefore , additional efforts for tuning learning rates might be required for optimizers to achieve the best performance . Empirically , we find the complement entropy in Eq ( 2 ) can be modified as follows to balance the losses between the two objectives : C ′ ( ŷc̄ ) = 1 K − 1 · C ( ŷc̄ ) = 1 K − 1 · 1 N N∑ i=1 H ( ŷic̄ ) ( 3 ) where K is the number of classes . This modification can be treated as the complement entropy C ( · ) being “ normalized ” by ( K − 1 ) . For all the experiments conducted in this paper , we use this normalized complement entropy as the complement objective to improve the baselines without further tuning of learning rates . 3.2 IMAGE CLASSIFICATION . We consider the following datasets for experiments with image classification : CIFAR-10 , CIFAR100 , SVHN , Tiny ImageNet and ImageNet-2012 . For CIFAR-10 , CIFAR-100 and SVHN , we choose the following baseline models : ResNet-110 ( He et al. , 2016b ) , PreAct ResNet-18 ( He et al. , 2016a ) , ResNeXt-29 ( 2×64d ) ( Xie et al. , 2017 ) , WideResNet-28-10 ( Zagoruyko & Komodakis , 2016 ) and DenseNet-BC-121 ( Huang et al. , 2017b ) with a growth rate of 32 . For those five models , we use a consistent set of settings below , which is described in ( He et al. , 2016b ) . Specifically , the models are trained using SGD optimizer with momentum of 0.9 . Weight decay is set to be 0.0001 and learning rate starts at 0.1 , then being divided by 10 at the 100th and 150th epoch . The models are trained for 200 epochs , with mini-batches of size 128 . The only exception here is for training WideResNet-28-10 , we follow the settings described in ( Zagoruyko & Komodakis , 2016 ) , and the learning rate is divided by 10 at the 60th , 120th and 180th epoch . In addition , no dropout ( Srivastava et al. , 2014 ) is applied to any baseline according to the best practices in ( Ioffe & Szegedy , 2015 ) . For Tiny ImageNet and ImageNet-2012 , the baseline models are slightly different : we follow the settings from ( Zhang et al. , 2018 ) , and the details are described in the corresponding paragraphs . CIFAR-10 and CIFAR-100 . CIFAR-10 and CIFAR-100 are datasets ( Krizhevsky , 2009 ) that contain colored natural images of 32x32 pixels , in 10 and 100 classes , respectively . We follow the baseline settings ( He et al. , 2016b ) to pre-process the datasets ; both datasets are split into a training set with 50,000 samples and a testing set with 10,000 samples . During training , zero-padding , random cropping , and horizontal mirroring are applied to the images with a probability of 0.5 . For the testing images , we use the original images of 32x32 pixels . A comparison between the models trained using the primary objective and the COT model is illustrated in Figures 3a and 4a for CIFAR-10 and CIFAR-100 respectively . We show that COT consistently outperforms the baseline models . Some of the models , for example , ResNetXt-29 , achieves a significant performance boost of 12.5 % in terms of classification errors . For some other models such as WideResNet-28-10 and DenseNet-BC-121 , the improvements are not as significant but are still large enough to justify the differences . Similar conclusions can be observed from the CIFAR-100 dataset . In addition to the comparisons of the performance , we also present the change of testing errors over the course of the training in Figures 3b and 4b for the ResNet-110 model . Following the standard training practice , learning rates drop after the 100th epoch , which corresponds to a drop in testing errors . As we can see from the plot , COT outperforms consistently compared to the baseline models when the models are close to the convergence . Street View House Numbers ( SVHN ) . The SVHN dataset ( Netzer et al. , 2011 ) consists of images extracted from Google Street View . We divide the dataset into a set of 73,257 digits for training and a set of 26,032 digits for testing . When pre-processing the training and validation images , we follow the general practice to normalize pixel values into [ -1,1 ] . Table 2 shows the experimental results and confirms that COT consistently improves the baseline models with the biggest improvement being the ResNet-110 with 11.7 % reduction on the error rate . Tiny ImageNet . Tiny ImageNet1 dataset is a subset of ImageNet ( Deng et al. , 2009 ) , which contains 100,000 images for training and 10,000 for testing images across 200 classes . In this dataset , each image is down-sampled to 64x64 pixels from the original 256x256 pixels . We consider four state-of-the-art models as baselines : ResNet-50 , ResNet-101 ( He et al. , 2016b ) , ResNeXt-50 ( 32×4d ) and ResNeXt-101 ( 32×4d ) ( Xie et al. , 2017 ) . During training , we follow the standard data-augmentation techniques , such as random cropping , horizontal flipping , and normalization . For each model , the stride of the first convolution layer is modified to adapt images of size 64x64 ( Huang et al. , 2017a ) . For evaluation , the testing data is only augmented with 56x56 central cropping . The rest of the experimental details are the same as the ones described at the beginning of Section 3.2 . Table 3 provides the experimental results , which demonstrate that COT consistently improves the performance of all baseline models . ImageNet . ImageNet-2012 dataset ( Russakovsky et al. , 2015 ) is one of the largest datasets for image classification , which contains 1.3 million images for training and 50,000 images for testing with 1,000 classes . Random crops and horizontal flips are applied during training ( He et al. , 2016b ) , while images in the testing set use 224x224 center crops ( 1-crop testing ) for data augmentation . ResNet-50 is selected as the baseline model , and we follow ( Goyal et al. , 2017 ) for the experimental 1https : //tiny-imagenet.herokuapp.com/ setup : 256 minibatch size , 90 total training epochs , and 0.1 as the initial learning rate starting that is decayed by dividing 10 at the 30th , 60th and 80th epoch . Table 4 shows ( a ) the error rate2 of baseline reported by ( He et al. , 2016b ) and ( b ) the error rate of baseline model trained by COT , which confirms COT further improves the baseline performance .

Summary: The paper deals with the training of neural networks for classification or sequence generation tasks, using a cross-entropy loss. Minimizing the cross-entropy means maximizing the predicted probabilities of the ground-truth classes (averaged over the samples). The authors introduce a "complementary entropy" loss with the goal of minimizing the predicted probabilities of the complementary (incorrect) classes. To do that, they use the average of sample-wise entropy over the complement classes. By maximizing this entropy, the predicted complementary probabilities are encouraged to be equal and therefore, the authors claim that it neutralizes them as the number of classes grows large. The proposed training procedure, named COT, consists of alternating between the optimization of the two losses.