Filter Training and Maximum Response: Classification via Discerning

1 INTRODUCTION . Suppose given a task of sorting a bunch of colored balls , we can usually do it in two ways . One is to randomly pick up a ball and deposit it to one of the groups according to its color . Or we can collect balls of the same color at a time until the last two colors are separated . Standard neural networks for classification tasks work in the former manner , for which it is an implied principle that every class should be on an equal footing and biases are harmful . This is the reason why the training data contain subsets of identical sample number or close in size for each class . The scheme introduced in this work is an analogue of the latter , in which the networks do not respond equally to features of different classes . Rather , they function more like filters , and ascription of an object is inferred from how strong their responses are . The filter networks are obtained by filter training , from which we have a group of networks or more specifically a batch of parameter values , number of which is equal to the number of classes . Ensemblization appears to be a built-in trait of this recognition scheme . Since a prediction should not refer to a preassigned label , a quantitative evaluation of how each filter responds to an input should be defined on an equal footing . As long as one of the alternative filters scores higher than the correct one , the prediction is wrong . Fluctuation makes the correct filter the weaker in this one to many contest . To do it a favour we introduce another hierarchy of ensemblization that is a batch of versions for the filters , in the hope that the correct filter could be the overall winner in this tournament . These three steps constitute a classification procedure of discerning , maximum response , and multiple check ( DMM ) . The DMM scheme can improve accuracy of mediocre networks , networks already having high accuracy , and those trained with small-scale data . Fundamental reason for the increase is that in the filter training a multiclass problem is reduced to a pseudo binary classification , the class of interest and the others . Intuitively , we have the feeling that telling a component from a mixture is easier than sorting all the ingredients . The pseudo reduction mitigates workload . Depending on capacity of networks and amount of training samples , the increase of accuracy varies . Nevertheless , as performance improvement due to the mitigation is almost sure , the scheme can be a general route to enhance feedforward networks . Moreover , since a filter is specifically trained for one class , the filter training is a feature abstraction procedure in itself . In the following , we first investigate the mechanism of filter training and how it works through a toy model . From this classification of points in a 2D plane , we can clearly view how the decision boundaries are reshaped . Then , we give a probabilistic argument why the maximum response is a proper criterion to infer classification . After remarking on its relation with related works , we experiment MMD on the CIFAR-10 and MNIST datasets . 2 FILTER TRAINING . The training dataset of our toy model consists of randomly scattered points around four centers at ( 0 , 0 ) , ( 0 , 3 ) , ( 3 , 0 ) , ( 3 , 3 ) , with a quota of 100 , 70 , 30 , 100 points , respectively . All directions are equally probable , and the radiuses obey a normal distribution with standard deviation of 2.0 . Points around ( 0 , 0 ) is labelled red , green for ( 0 , 3 ) , ( 3 , 0 ) , and blue for ( 3 , 3 ) , so that each class has 100 points . This simple classification is handled by a network made of an input and an output layer , which are fully connected layers . Size of the input layer is set as 128 nodes , and the ReLU activation is used . The Adam optimizer with a learning rate of 10−4 and random shuffling of samples are adopted for all the trainings in this work . The ordinary training is run for 104 iterations with batch size of 100 to build the original classification network . By filter training we want to make features pertaining to a specified class stand out and those related to the others suppressed . The most direct attempt is to train/retrain the network with the sub dataset of the class . The resulting model , however , is a nearly constant mapping to that class label , no matter what the real ascription is . In Fig . 1 are results of retraining the network with the blue subset . Fig . 1 ( a ) is the portion of samples assigned to their original class ( training accuracy ) , and Fig . 1 ( b ) shows portion of the red and green data points that are assigned to blue ( misassignment ) . In Fig . 1 ( c ) the light blue background fully covers the dataset , indicating that no decision boundary remains . Indeed , this retrain procedure leads to a constant mapping , which blurs all the features , and the model totally loses recognition capacity . The above results are conceptually reasonable , since filtering makes no sense without unwanted components as reference . The lesson is that for filter training the targeted class along is not enough and the alternative classes should be taken into account . Accordingly , we found two viable ways to prepare the training dataset : 1 ) perform a random label shuffling among samples in the unspecified classes ; 2 ) each sample , except those in the specified class , is assigned labels of all the other unspecified classes beside its own . If the random sample shuffling is used during training and the label shuffling in preparation-1 is redone at each epoch , the two are equivalent . Datasets for training a filter of the blue are illustrated in Fig . 2 . An obvious advantage of preparation1 is smaller data size , which is equal to that of the original . For our toy model , there are only two alternative classes , so the data size is only doubled in Fig . 2 ( c ) . Size enlargement with preparation-2 is much severe when the number of classes increases . For a training set having Nc classes , under each label of the alternative classes all the samples except those of the specified class should be included , so every subclass dataset is augmented by a factor of Nc − 1 . So is the total training dataset , if Nc− 2 copies of the specified class is included to keep its sample density , as what is done for the toy model . For this reason , we use the equivalent reshuffling to implement preparation-2 in our experiments . Preparation-2 has its advantage that the data are better randomized by multiple labelling while preparation-1 is a dataset with definite labelling . Fig . 3 shows results of two independent filter trainings using preparation-1 and -2 , respectively . Both the preparation procedures lead to increased accuracy of the targeted class , while accuracy for the alternative classes are decreased . More importantly , the filter training does not suffer from decision boundary destruction . A point of possible use is the similarity in the two trainings using preparation2 , where both the training accuracies and trained decision boundaries follow a similar pattern . This implies that we can use the filter training to abstract features and construct innate metrics . To see more clearly the effect of filter training , we trained a filter for the red using preparation-2 and plot in Fig . 4 snapshots of decision boundary reshaping . From comparison with Fig . 1 , although the accuracy for the targeted class is less close to 100 % , considerable increase results . In contrast to Fig . 1 ( b ) , filter training only add a little misassignment , and this small increase is understandable , since boundary expansion as well as other adaptations is a reasonable strategy for the specified class to increase its accuracy . Difference between the increase of accuracy ( ≈ 0.2 ) and that of misassignment ( ≈ 0.1 ) indicates that the drawings-in of data are purposeful and selective . From the dynamics of decision boundary construction , we can understand how and why the filter training works . As is known , in each training step the boundary is adjusted a little bit according to the gradients to lower the cost . The gradients can be imagined as instructions encoding direction and strength of forces acting on the decision boundary . When a training batch is fed , each involved sample exerts a pull or push force on the boundary , based on its own location in the configuration space , its label , and the boundary position . Since the cost function is a mean value , forces from the samples in the unspecified classes are diminished by the random labelling , and the samples in the specified class is dominant . Still owing to the randomness , small movements by the unspecified samples in different training steps result in mutual cancelling other than increment . So only the class of interest accounts for effective movement of the decision boundary and adapts the boundary to a shape that well divides itself from the others . 3 MAXIMUM RESPONSE . Because every filter gives a softmax evaluation , we must define a selection rule to infer classification . In the ideal case , we expect only one filter has a strong response at the corresponding element , while responses of the other elements and of the other filters are suppressed . For an input belonging to the first class , for instance , it is the ideal output for the first filter that its first digit is close to 1.0 and consequently the other digits are minimal , and meanwhile for the other filters the softmax slightly fluctuates around 1/Nc . In reality , however , this is hardly the case . When more than one filters give positive indication , i.e. , the corresponding digit is the biggest , we choose the one having the strongest response . This criterion seems good , since it always yields a result . Nevertheless , our choice is faced with a dilemma of superiority , when outputs have a pattern as shown in Fig . 5 ( a ) , where r12 > r11 > r22 > r21 ( r23 ) . The first filter has the strongest response at the corresponding digit ( r11 > r22 ) but the indication is negative ( r11 < r12 ) , while the second filter makes a positive indication ( r22 > r21 ( r23 ) ) . Which is superior , the maximum response ( MR ) or the positive indication ( PI ) ? Since switch statements can not be generalized to a batch of inputs , instead of using them , we compute a prediction under the positive indication superiority by defining a score vector s = p + p ∗m + m , ( 1 ) p = agrmax ( R ) == agrmax ( I ) , ( 2 ) m = 1 Nc ∑ j ( R ∗ I ) ij , ( 3 ) where the multiplications are elementwise . Here , R is the matrix whose rows are the softmax evaluations , and I is an Nc × Nc unit matrix whose nonzeros mark the corresponding digits . The argmax operates on the rows , so p is a vector recording the indications with 1 and 0 , and m records softmax values at the corresponding digits . The second term p∗m facilitates the maximum response criterion in the case of multiple positive indications . The multiplication nulls the value of filters that give negative indications and only positive indications matter , so the superiority is untouched . The third term is added to ensure a result when no positive indication is made ( p = 0 ) . Since any positive indication scores 1 in p and m < 1.0 , it does not alter the superiority . If only the third term presents , the maximum element of s dictates the classification , which is nothing but the maximum response criterion . It can be readily checked by substitution that only the pattern in Fig , 5 ( a ) can trigger the difference between the two criteria . Beside the maximum response , there can be other ways to solve the problem of multiple positive indications . When we are indecisive , we may consult an expert . Similarly , we can train such expert networks with only subclass datasets that are concerned , and refer to the corresponding one according to the positive indications . Another resort is the peer review like checking . Similarity in background and diversity in specialty may compensate the weakness and bias . Since this multiple check procedure is more pertinent to filtering in the sense that it can suppress fluctuations , we focus our study on it . Here two types of fluctuations should be distinguished . One is fluctuation in the softmax evaluations , which is harmful in that it can cause false positive indications . When they occurs , however , it is desirable that distribution of the false positive indications fluctuates in different filter versions . With the latter , incremental status of the real positive indication could be built up . This is the mechanism of the multiple check , and the motivation of introducing randomness . As randomness has already been there in the training process and data preparation , in the multiple check we just retrain the network to build new versions , and scores of different versions are averaged . For the toy model , we use the training data also as test data and plot in Fig . 5 the test accuracy up to 10 filter versions . Better performance for preparation-2 ( Fig . 5 ( c ) ) suggests that fluctuation is indeed harmful . Stronger fluctuations are also the cause for the bigger difference in Fig . 5 ( b ) , since we can see from Fig . 5 ( a ) that occurrence of the pattern requires strong fluctuations . In both results , the maximum response is superior to the positive indication . There could be a reason . Let us consider a point located near the decision boundary , where improvements are actually made by DMM ( see Fig . 5 ( d ) ) . As the softmax is unitary , we assume the Dirichlet distribution for output of every filter , when randomness are introduced into filter versions by the stochastic operations such as the stochastic gradient descent , the random shuffling of training samples , and the random label assignment in filter training . Note that the fixated point as input is unchanged while the outputs in different versions are stochastic due to the randomness . For a point exactly at the ideal boundary separating class-1 and class-2 , D01 = Dir ( α , β , β , · · · , β ) , ( 4 ) D02 = Dir ( β , α , β , · · · , β ) , ( 5 ) · · · · · · · · · · · · · · · · · · , ( 6 ) D0Nc = · · · · · · · · · · · · , ( 7 ) is a proper assumption by symmetry argument . The less relevant distributions are denoted in ellipsis , for which the probability at the corresponding digit is a little bit biased , since filter training tends to expand the decision boundary enclosing the class of interest . Compared to other r1j ( j 6= 1 ) , r12 is not endowed with specialty , because in the filter training for class-1 , these classes ( j 6= 1 ) are treated equally . It is the same for r21 . When the point shifts away form class-2 and into class-1 , a bias is raised between the two filters . The probability density peak of filter-1 should move toward the high end of r11 , and vice versa for filter-2 , so we can expect D1 = Dir ( α+ ∆ , β , β , · · · , β ) , D2 = Dir ( β , α−∆ , β , · · · , β ) ( 8 ) with ∆ > 0 , and for shift into class-2 the sign is reversed , ∆ → −∆ . Here , we do not alter β and parameters of the remaining distributions ( so Dn = D0n , n > 2 ) , which implies that we attribute variation of distribution on these digits to secondary effect of varied α . This is plausible , since ∆ is a function of the direction and distance the point away from the boundary , and meanwhile labels for the alternative classes are randomly shuffled , which diminishes dependence on position and direction . Of course , weak dependence could remain , since the training data themselves are not evenly distributed in the configuration space . For clarity and without loss of generality , we can assume rik > rij for k > j , beside r12 > r11 > r22 > r21 . Then , incidence of the pattern is given by integral P = ∫ D1D2 · · ·DNcdR over the assumed region subject to unity ∑ j rij = 1 . Now , the problem is a Bayesian like inference—shift into which class yields higher incidence . As what really matters is the trend , we can avoid tedious calculation for an analytic formulation by taking the derivative with respect to ∆ . At the boundary ∆ = 0 , and presence of both ∆ and −∆ cancels the derivatives of the normalization factors . The resulting derivative reads dP d∆ ∣∣∣ ∆=0 = ∫ ln ( r11 r22 ) D01D 0 2 · · ·D0NcdR , ( 9 ) which is positive due to r11 > r22 . So a positive ∆ , corresponding to shift into class-1 , increases the incidence . This means that an input having the output in Fig . 5 ( a ) should be ascribed to class-1 and hence maximum response is superior to positive indication . Conceptually , the enhancement by DMM is due to the mitigation of workload , so that more delicate decision boundaries can be constructed . With the wrongs and rights with respect to the original results as shown in Fig . 5 ( d ) , we can go to more details . Near where DMM makes a mistake , a correction is made to compensate , which ensures an accuracy not going lower . The three close points around ( −0.5 , −1.5 ) is an instance that one wrong is traded for two rights and has one net gain . The closeness of the points implies that high expense of capacity is required to make a delicate discrimination . It is like that a person needs to spend considerable brain power to obtain relatively small gain , which is uneconomic if the brain power is not abundant . This economic consideration also shows up in the sparse regions , where DMM adds rights but no wrongs . Due to the sparsity , a gain requires paying more attention to adapt the decision boundary more , which is also an uneconomic deployment without the release of brain power .

Summary: The author proposed a novel classification procedure of discerning, maximum response, and multiple check (DMM) to improve accuracy of mediocre network and enhance feedforward network. The author used a toy model with randomly scattered points around four centres to demonstrate the dynamics of decision boundary construction and show how and why the filter training had worked when a multiclass problem is reduced to a pseudo binary classification. The literature review is rather short, some discussion on how the proposed work advances the latest deep neural networks should be added. The experiments are carried out with a simple CNN, it will be better if the author demonstrate its integration to more state-of-art network and make a comparison to their performance.