Neural Random Projection: From the Initial Task To the Input Similarity Problem

1 INTRODUCTION . Evaluating object similarity is an important area in machine learning literature . It is used in various applications such as search query matching , image similarity search , recommender systems , clustering , classification . In practice , the quality of similarity evaluation methods depends on the data representation . For a long time neural networks show successful results in many tasks and one such task is obtaining good representations . Many of these methods can be considered in terms of domain and task . The first case is when we have only unlabeled dataset . Then we can use autoencoders ( Bank et al. , 2020 ) or self-supervised approaches ( Chen et al. , 2020 ; Devlin et al. , 2018 ; Doersch et al. , 2015 ; Dosovitskiy et al. , 2014 ; Gidaris et al. , 2018 ; Noroozi & Favaro , 2016 ; Noroozi et al. , 2017 ; Oord et al. , 2018 ; Peters et al. , 2018 ) , which require formulation of a pretext task , which in most cases depends on the data type . These methods can be called explicit because they directly solve the problem of representation learning . Moreover , these models can be used for transfer knowledge when we have labeled data only in the target domain . The second case is where we have labeled data in the source and target domains . Then we can apply a multi-task learning approach ( Ruder , 2017 ) or fine-tune the models ( Simonyan & Zisserman , 2014 ; He et al. , 2016 ) trained on a large dataset like ImageNet . Finally , there is the domain adaptation approach ( Wang & Deng , 2018 ) where we have a single task but different source and target domains with labeled data in the target domain ( Hu et al. , 2015 ) or with unlabeled data ( Li et al. , 2016 ; Yan et al. , 2017 ; Zellinger et al. , 2017 ) . In our study the target task is to measure similarity between objects and to define hidden classes based on it . We are interested in studying the issue of implicit learning of representations . Can the neural networks store information about subcategories if we don ’ t explicitly train them to do this ? More formally , we have the same source and target domains but different tasks and we don ’ t have labeled data in the target domain . That makes our case different from the cases of transfer learning . A solution to this problem could be useful in many practical cases . For example , we could train a model to classify whether messages are spam or not and then group spam campaigns or kind of attacks ( phishing , spoofing , etc . ) based on similarity measuring by trained neural network . Similar cases could be in the medicine ( classifying patients into healthy/sick and grouping them by the disease ) or in financial ( credit scoring ) area . The benefits are that we do not depend on the data type and , more importantly , we use only one model for different tasks without fine-tuning , which significantly reduces time for developing and supporting of several models . Similar study was done in ( Hanawa et al. , 2020 ) , where authors proposed evaluation criteria for instance-based explanation of decisions made by neural network and tested several metrics for measuring input similarity . In particular , they proposed the Identical subclass test which checks whether two objects considered similar are from the same subclass . According to the results of their experiments , the most qualitative approach is the approach presented in ( Charpiat et al. , 2019 ) , which proposed to measure similarity between objects using the gradients of a neural network . In experiments , the authors applied their approach to the analysis of the self-denoising phenomenon . Despite the fact that this method has theoretical guaranties and does not require to modify the model to use it , in practice , especially in real-time tasks , using gradients tends to increase the computation time and size of vector representation . This approach will be described in more detail in Section 2 . To avoid these problems , we propose a method that only uses outputs from the last hidden layer of a neural network and does not use a backward step to vectorize the input . In our research , we found that a correlation of neurons and insufficient width of the last hidden layer influence on the quality of representations obtained in implicit way . To solve these issues , we propose several modifications . First , we show that the weight matrix should be orthogonal . Second , we modify Batch-Normalization ( Ioffe & Szegedy , 2015 ) to obtain the necessary mathematical properties , and use it with dropout ( Srivastava et al. , 2014 ) to reduce the correlation caused by nonlinear activation functions . Using orthogonal weight matrices allows us to consider the neural network in terms of Random Projection method and evaluate the lower bound of the width of the last hidden layer . Our approach will be discussed in detail in Section 3 . Finally , in Section 4 we perform experiments on MNIST dataset and physical examination dataset ( Maxwell et al. , 2017 ) . We used these datasets to show that our approach can be applied for any type of data and combined with different architectures of neural networks . In both experiments , we split a set of labels into two disjoint subsets to train a neural network for binary classification problem , and then use this model to measure the similarity between input data and define hidden classes . We also cluster the inputs to evaluate how well objects from the same class are grouped together . Our experimental results show that the proposed approach achieves competitive results on the input similarity task while reducing both computation time and the size of the input representation . 2 RELATED WORKS . Using a trained neural network to measure similarity of inputs is a new research topic . In ( Charpiat et al. , 2019 ) the authors introduce the notion of object similarity from the neural network perspective . The main idea is as follows : how much would parameter variation that changed the output for x impact the output for x′ ? In principle , if the objects x and x′ are similar , then changing parameters should affect the outputs in a similar way . The following is a formal description for one- and multidimensional cases of the output value of a neural network . One-dimensional case Let fθ ( x ) ∈ R be a parametric function , in particular a neural network , x , x′ be input objects , θ ∈ Rnθ - model parameters , nθ - number of parameters . The authors proposed the following metric : ρθ ( x , x ′ ) = ∇θfθ ( x′ ) ∇θfθ ( x ) ||∇θfθ ( x′ ) || ||∇θfθ ( x ) || ( 1 ) In this way , the object similarity is defined as the cosine similarity between the gradients computed at these points . Multi-dimensional case Let fθ ( x ) ∈ Rd , d > 1 . In this case , the authors obtained the following metric : ρθ , d ( x , x ′ ) = 1 d Tr ( Kθ ( x , x ′ ) ) , ( 2 ) where Kθ ( x , x′ ) = K −1/2 x′ , x′Kx , x′K −1/2 x , x , and Kx′ , x = ∂fθ∂θ ∣∣∣ x′ ∂fθ ∂θ T ∣∣∣ x is calculated using the Jaco- bian matrix ∂fθ∂θ ∣∣∣ x . Summary Unlike using third-party models for vectorization , such as VGG , this approach allows us to use any pre-trained neural network to calculate the similarity of input data . To achieve this , authors use gradients of neural network as illustrated in equation 1 and equation 2 . This gradientbased solution takes into account all activations , which does not require the selection of a hidden layer for vectorization . However , it has a number of drawbacks . First of all , fast computation of gradients require additional computational resources . Second , the size of objects representation is nθ for one-dimensional output and nθ ∗d for multi-dimensional case . This means that increasing model complexity increases representation size and , as a result , can lead to large memory consumption . Motivation of this research is to develop an approach that reduces the size of the representation and uses only the forward pass of a trained neural network to work . In the next section , we will discuss the proposed approach in detail . 3 PROPOSED METHOD . In this paper , we suggest using outputs from the last hidden layer of a trained neural network to encode inputs . This representation has several advantages . First , the output of the last layer is usually low-dimensional , which allows us to get a smaller vector dimension . Second , and more importantly , on the last hidden layer the semantics of the original problem is taken into account to the greatest extent . For example , for a classification problem , data is theoretically linearly separable on the last hidden layer . Informally , this allows us to measure similarities within each class , which reduces false cases in terms of the original problem . This is extremely important for clustering spam campaigns , because this representation reduces the probability to group spam and legitimate messages together . However , as already mentioned in the section 1 , a neural network that is trained to solve a simple problem does not retain the quality of representaions needed to solve a more complex one . Due to this fact , it ’ s impossible to apply this representation as is . We have identified the main causes of this : a strong correlation of neurons and an insufficient size of the last hidden layer . Our goal is to avoid these in order to make the proposed vectorization reasonable . In practice , last hidden layers are usually fully connected . For this reason , we consider only this type of the last layer . In 3.1 , we show how to reduce correlation between neurons , and in 3.2 we offer an estimate of the lower bound of the size of the last hidden layer and prove that the proposed representation can be used for the input similarity problem . We introduce the following notation : 1. l ∈ { 0 , 1 , . . . , L } - layer number , where l = 0 - input layer ( source space ) , l = L - output layer , other - hidden layers . 2 . Nl - number of units in layer l 3. hl ∈ RNl - pre-activation vector 4 . Φ ( hl ) = ( φ ( hl,1 ) , . . . , φ ( hl , Nl ) ) - activation vector 5 . Wl ∈ RNl−1×Nl , bl ∈ RNl - weight matrix and bias vector 3.1 NEURON CORRELATION . The benefits of decorrelated representations have been studied in ( LeCun et al. , 2012 ) from an optimization viewpoint and in ( Cogswell et al. , 2015 ) for reducing overfitting . We consider the decorrelated representations from information perspective . Basically , correlation of neurons means that neurons provide similar information . Therefore , we gain less information from observing two neurons at once . This phenomenon may occur due to the fact that the neural network does not retain more information than is necessary to solve a particular task . Thus , only important features are highlighted . The example in Fig . 1A illustrates that the output values of two neurons are linearly dependent , which entails that many objects in this space are indistinguishable . On the contrary ( Fig . 1B ) , decorrelation of neurons provides more information and , as a result , the ability to distinguish most objects . In the following paragraphs , we identify the main causes of the correlation of neurons and suggest ways to prevent it . We decided to consider the correlations before and after activation of neurons separately . Reducing correlation of neurons before activation Statement 1 explains the main reason for the correlation that occurs before activation . Statement 1 . Suppose that for some layer l the following conditions are satisfied : 1 . E [ Φ ( hl ) ] = 0 2 . E [ Φ ( hl ) TΦ ( hl ) ] = σ2l I - covariance matrix as we required E [ Φ ( hl ) ] = 0 Then , in order for correlation not to occur on the layer l + 1 , it is necessary that the weight matrix Wl+1 be orthogonal , that is , satisfy the conditionW Tl+1Wl+1 = I The statement 1 shows that the first correlation factor on a fully connected layer is a non-orthogonal weight matrix , given that , if the input neurons do not correlate . See Appendix A.1 for proof of the statement 1 . During training , the network does not try to maintain this property , solving the problem at hand . Later in this paper , we will add regularization to penalize the loss function if the weights are not orthogonal . The corollary follows from the statement 1 , which gives the second condition for preventing correlation . This corollary states that the dimension of the layer l + 1 should be no greater than on the layer l. Otherwise , this leads to a correlation and does not increase information . See Appendix A.1 for proof this corollary . Corollary 1 . Suppose that the conditions of statement 1 are satisfied , then if the dimension Nl+1 > Nl , then there is certainly a pair of neurons hl+1 , i , hl+1 , j ; i 6= j : Cov ( hl+1 , i , hl+1 , j ) 6= 0 It should be noted that there are also the studies addressing orthogonal weight matrices ( Huang et al. , 2017 ; Jia et al. , 2019 ; Xie et al. , 2017 ) . However all of these works consider this topic from optimization perspective . In particular , in ( Cho & Lee , 2017 ) was proposed an approach for optimization of loss on the Stiefel manifold Vp ( Rn ) = { W ∈ Rn×p|W TW = I } to ensure orthonormality of weight matrices throughout training . To achieve this , they applied the orthogonality regularization ( 3 ) to require the Gram matrix of the weight matrix to be close to identity matrix . In our study we also use regularization ( 3 ) to ensure orthonormality of weight matrices . 1 2 L∑ l=1 λl ∣∣∣∣W Tl Wl − I∣∣∣∣2F , ( 3 ) Providing the necessary moments of neurons after activation In the statement 1 , we relied on the zero expected value and the same variance of units in the layer . But the nonlinear activation function does not guarantee the preservation of these properties . Due to this fact , we can not reason in the same way for the following layers . Therefore , we propose using an activation normalization approach similar to Batch-Normalization : φ̂ ( hl , i ) = γl φ ( hl , i ) − µφ ( hl , i ) √ σ2φ ( hl , i ) + , ( 4 ) where γl is a trainable scale parameter , µφ ( hl , i ) , σ 2 φ ( hl , i ) - parameters that are evaluated , as inIoffe & Szegedy ( 2015 ) . The difference compared to the standard Batch-Normalization is that the γl is the same for all neurons and we removed the βl , i parameters . This leads to an expected value of zero and the same variance γ2l of each unit in the layer . Reducing correlation of neurons after activation It should be noted that an activation function can also impact the formation of redundant features ( Ayinde et al. , 2019 ) . In particular , in this work we use tanh ( x ) ∈ ( −1 , 1 ) as the activation function . There are several methods that prevent formation of redundant features . In ( Cogswell et al. , 2015 ) was proposed DeCov loss which penalizes non-diagonal elements of estimated covariance matrix of hidden representation . In ( Desjardins et al. , 2015 ; Blanchette & Laganière , 2018 ; Huang et al. , 2018 ) were proposed approaches for learning decorrelation layers that perform the following transformation : Φ̃ ( hl ) = ( Φ ( hl ) − µΦ ( hl ) ) Σ − 12 Φ ( hl ) . All of these methods have a common drawback : they require estimating covariance matrices . Often in practice the size of mini-batch is much smaller than is needed for estimating covariance matrices . Therefore , the covariance matrix is often singular . Moreover , methods that use the decorrelation layer are computationally expensive when it comes to high-dimensional embeddings , since it is necessary to calculate the square root of the inverse covariance matrix . This is especially evident in wide neural networks . Besides , these techniques add a significant amount of parameters ∑L−1 l=1 N 2 l . As an alternative , we suggest using Dropout ( Srivastava et al. , 2014 ) , which prevents units from co-adapting too much and reduces the correlation between neurons during training stage in the layer in proportion to p - the probability of retaining a unit in the network . See Appendix A.2 for the proof of this statement . It is important to note that we apply normalization to the input data ( input layer l = 0 ) as well as Dropout , since it is not always possible to decorrelate data so that this does not affect the quality of training . Moreover , fully-connected layers are often used in more complex architectures , such as convolutional neural networks . Obviously , after convolution operations , transformed data will be correlated . In this case , we must apply dropout after the convolutional block in order to reduce the correlation of neurons and use the proposed approach for vectorization .

Summary: The paper studies the usage of the representations developed in the last layer of a neural network as a way to measure the similarity between input patterns. The fundamental idea revolves around the concept of orthogonal weight matrices, to decorrelate the activations of the neurons, and which would definitely enrich the internal representations developed by the neurons in the last hidden layer. In the paper, it is suggested to regularize during training to maintain the orthogonality of the weight matrices. Moreover, a variant of Batch Normalization is proposed. The method proposed in the paper is then experimentally applied and evaluated on two benchmark datasets (MNIST and  Henan Renmin).