k-Nearest Neighbors by Means of Sequence to Sequence Deep Neural Networks and Memory Networks

1 INTRODUCTION . Recently , neural networks have been attracting a lot of attention among researchers in both academia and industry , due to their astounding performance in fields such as natural language processing Turian et al . ( 2010 ) Mikolov et al . ( 2013 ) and image recognition Krizhevsky et al . ( 2012 ) Deng et al . ( 2009 ) . Interpretability of these models , however , has always been an issue since it is difficult to understand the performance of neural networks . The well-known manifold hypothesis states that real-world high dimensional data ( such as images ) form lower-dimensional manifolds embedded in the high-dimensional space Carlsson et al . ( 2008 ) , but these manifolds are tangled together and are difficult to separate . The classification process is then equivalent to stretching , squishing and separating the tangled manifolds apart . However , these operations pose a challenge : it is quite implausible that only affine transformations followed by pointwise nonlinear activations are sufficient to project or embed data into representative manifolds that are easily separable by class . Therefore , instead of asking neural networks to separate the manifolds by a hyperplane or a surface , it is more reasonable to require points of the same manifold to be closer than points of other manifolds Olah ( 2014 ) . Namely , the distance between manifolds of different classes should be large and the distance between manifolds of the same class should be small . This distance property is behind the concept of k-Nearest Neighbor ( kNN ) Cover & Hart ( 1967 ) . Consequently , letting neural networks mimic kNN would combine the notion of manifolds with the desired distance property . We explore kNN through two deep neural network models : sequence to sequence deep neural networks Sutskever et al . ( 2014 ) and memory networks Sukhbaatar et al . ( 2015 ) . A family of our models are based on a sequence to sequence network . The new sequence to sequence model has the input sequence of length one corresponding to a sample , and then it decodes it to predict two sequences of output , which are the classes of closest samples and neighboring samples not necessarily in the training data , where we call the latter as out-of-sample feature vectors . We also propose a family of models built on a memory network , which has a memory that can be read and written to and is composed of a subset of training samples , with the goal of using it for predicting both classes of close samples and out-of-sample feature vectors . With the help of attention over memory vectors , our new memory network model generates the predicted label sequence and out-of-sample feature vectors . Both families of models use loss functions that mimic kNN . Computational experiments show that the new sequence to sequence kNN model consistently outperforms benchmarks ( kNN , a feed-forward neural network and a vanilla memory network ) . We postulate that this is due to the fact that we are forcing the model to ‘ work harder ’ than necessary ( producing out-of-sample feature vectors ) . Different from general classification models , our models predict not only labels , but also out-ofsample feature vectors . Usually a classification model only predicts labels , but as in the case of kNN , it is desirable to learn or predict the feature vectors of neighbors as well . Intuitively , if a deep neural network predicts both labels and feature vectors , it is forced to learn and capture representative information of input , and thus it should perform better in classification . Moreover , our models also function as synthetic oversamplers : we add the out-of-sample feature vectors and their labels ( synthetic samples ) to the training set . Experiments show that our sequence to sequence kNN model outperforms Synthetic Minority Over-sampling Technique ( SMOTE ) Chawla et al . ( 2002 ) and Adaptive Synthetic sampling ( ADASYN ) He et al . ( 2008 ) most of the times on imbalanced datasets . Usually we allow models to perform kNN searching on the entire dataset , which we call the full versions of models , but kNN is computationally expensive on large datasets . We design an algorithm to resolve this and we test our models under such an ‘ out-of-core ’ setting : only a batch of data can be loaded into memory , i.e . kNN searching in the entire dataset is not allowed . For each such random batch , we compute the K closest samples with respect to the given training sample . We repeat this R times and find the closestK samples among theseKR samples . These closestK samples provide the approximate label sequence and feature vector sequence to the training sample based on the kNN algorithm . Computational experiments show that sequence to sequence kNN models and memory network kNN models significantly outperform the kNN benchmark under the out-of-core setting . Our main contributions are as follows . First , we develop two types of deep neural network models which mimic the kNN structure . Second , our models are able to predict both labels of closest samples and out-of-sample feature vectors at the same time : they are both classification models and oversamplers . Third , we establish the out-of-core version of models in the situation where not all data can be read into computer memory or kNN can not be run on the entire dataset . The full version of the sequence to sequence kNN models and the out-of-core version of both sequence to sequence kNN models and memory network kNN models outperform the benchmarks , which we postulate is because learning neighboring samples enables the model to capture representative features . We introduce background and related works in Section 2 , show our approaches in Section 3 , and describe datasets and experiments in Section 4 . Conclusions are in Section 5 . 2 BACKGROUND AND LITERATURE REVIEW . There are several works trying to mimic kNN or applying kNN within different models . Mathy et al . ( 2015 ) introduced the boundary forest algorithm which can be used for nearest neighbor retrieval . Based on the boundary forest model , in Zoran et al . ( 2017 ) , a boundary deep learning tree model with differentiable loss function was presented to learn an efficient representation for kNN . The main differences between this work and our work are in the base models used ( boundary tree vs standard kNN ) , in the main objectives ( representation learning vs classification and oversampling ) and in the loss functions ( KL divergence vs KL divergence components reflecting the kNN strategy and L2 norm ) . Wang et al . ( 2017 ) introduced a text classification model which utilized nearest neighbors of input text as the external memory to predict the class of input text . Our memory network kNN models differ from this model in 1 ) the external memory : our memory network kNN models simply feed a random batch of samples into the external memory without the requirement of nearest neighbors and thus they save computational time , and 2 ) number of layers : our memory network kNN models have K layers while the model proposed by Wang et al . ( 2017 ) has one layer . A higher-level difference is that Wang et al . ( 2017 ) considered a pure classification setting , while our models generate not only labels but out-of-sample feature vectors as well . Most importantly , the loss functions are different : Wang et al . ( 2017 ) used KL divergence as the loss function while we use a specially designed KL divergence and L2 norm to force our models to mimic kNN . The sequence to sequence model , one of our base models , has recently become the leading framework in natural language processing Sutskever et al . ( 2014 ) Cho et al . ( 2014 ) . In Cho et al . ( 2014 ) an RNN encoder-decoder architecture was used to deal with statistical machine translation problems . Sutskever et al . ( 2014 ) proposed a general end-to-end sequence to sequence framework , which is used as the basic structure in our sequence to sequence kNN model . The major difference between our work and these studies is that the loss function in our work forces the model to learn from neighboring samples , and our models are more than just classifiers - they also create out-of-sample feature vectors that improve accuracy or can be used as oversamplers . In summary , the main differences between our work and previous studies are as follows . First , our models predict both labels of nearest samples and out-of-sample feature vectors rather than simply labels . Thus , they are more than classifiers : the predicted label sequences and feature vector sequences can be treated as synthetic oversamples to handle imbalanced class problems . Second , our work emphasizes on the out-of-core setting . All of the prior works related to kNN and deep learning assume that kNN can be run on the entire dataset and thus can not be used on large datasets . Third , our loss functions are designed to mimic kNN , so that our models are forced to learn neighboring samples to capture the representative information . SEQUENCE TO SEQUENCE MODEL . A family of our models are built on sequence to sequence models . A sequence to sequence ( Seq2seq ) model is an encoder-decoder model . The encoder encodes the input sequence to an internal representation called the ‘ context vector ’ which is used by the decoder to generate the output sequence . Usually , each cell in the Seq2seq model is a Long Short-Term Memory ( LSTM ) cell Hochreiter & Schmidhuber ( 1997 ) or a Gated Recurrent Unit ( GRU ) cell Cho et al . ( 2014 ) . Given input sequence x1 , ... , xT , in order to predict output Y P1 , ... , Y P K ( where the superscript P denotes ‘ predicted ’ ) , the Seq2seq model estimates conditional probability P ( Y P1 , ... , Y P t |x1 , ... , xT ) for 1 ≤ t ≤ K. At each time step t , the encoder updates the hidden state het , which can also include the cell state , by het = f e h ( xt , h e t−1 ) where 1 ≤ t ≤ T . The decoder updates the hidden state hdt by hdt = f d h ( Y P t−1 , h d t−1 , h e T ) where 1 ≤ t ≤ K. The decoder generates output yt by yt = g ( Y P t−1 , h d t , h e T ) , ( 1 ) and Y Pt = q ( yt ) with q usually being softmax function . The model calculates the conditional probability of output Y P1 , ... , Y P K by Pr ( Y P1 , ... , Y P K |x1 , ... , xT ) = K∏ t=1 Pr ( Y Pt |Y P1 , ... , Y Pt−1 ) . END TO END MEMORY NETWORKS . The other family of our models are built on an end-to-end memory network ( MemN2N ) . This model takes x1 , ... , xn as the external memory , a ‘ query ’ x , a ground truth Y GT and predicts an answer Y P . It first embeds memory vectors x1 , ... , xn and query x into continuous space . They are then processed through multiple layers to generate the output label Y P . MemN2N has K layers . In the tth layer , where 1 ≤ t ≤ K , the external memory is converted into embedded memory vectors mt1 , ... , m t n by an embedding matrix A t. The query x is also embedded as ut by an embedding matrix Bt . The attention scores between embedded query ut and memory vectors ( mti ) i=1,2 , ... , n are calculated by p t = softmax ( ( ut ) Tmt1 , ( u t ) Tmt2 , ... , ( u t ) Tmtn ) . Each xi is also embedded to an output representation cti by another embedding matrix C t. The output vector from the external memory is defined as ot = n∑ i=1 ptic t i . By a linear mapping H , the input to the next layer is calculated by ut+1 = Hut + ot . Sukhbaatar et al . ( 2015 ) suggested that the input and output embeddings are the same across different layers , i.e . A1 = A2 = ... = AK and C1 = C2 = ... = CK . In the last layer , by another embedding matrix W , MemN2N generates a label for the query x by Y P = softmax ( W ( HuK + oK ) ) .

Summary: To exploit the near neighbor/manifold features, this paper proposes to combine k-nearest neighbors of each training data point into the neural network models.  Specifically, the authors propose two families of models built on the popular sequence to sequence neural network models and memory network models, which mimic the k-nearest neighbors model in model learning. Besides, the final label of the classification task will be learned, a sequence of nearest neighbor labels and a sequence of out-of-sample feature vectors (for oversampling) will be also learned in the same time, similar with the multi-task approaches. Since the proposed models are based k-nearest neighbor calculations, which is time-consuming, they also design an algorithm for the ‘out-of-core’ situation, say load a small portion of data each time to approximately calculate the neighbors. Experiments show that some proposed models work better than baselines in classification and oversampling.