Alignment Based Mathching Networks for One-Shot Classification and Open-Set Recognition

1 INTRODUCTION . Convolutional neural networks ( CNNs ) in various realizations have come to dominate image , speech , and even text processing . Performance gains afforded by such architectures , with learned , multi-stage image features , emerge from large amounts of supervised training data . In addition to being data hungry , the networks are also challenging to interpret after the fact , potentially yielding confident yet unwarranted predictions for tailored examplesYosinski et al . ( 2015b ) ; Szegedy et al . ( 2013 ) . Our goal in contrast is to learn from very few examples , and specifically tailor the models to emphasize interpretability . Deep , complex architectures , despite large numbers of parameters involved , are not incompatible with one-shot or open-set learning . Instead of predicting categories directly , one can set up the learning problem to recognize sameness between pairs of images as in Siamese networks Koch et al . ( 2015 ) or between a set of support images as in Matching Networks Vinyals et al . ( 2016 ) . In this case , a relatively large dataset of pairs is required to help the network learn features that summarize images in a manner that is tailored for assessing category differences . Such a network can succeed in predicting sameness for entirely new categories or comparing new images to a set of given reference sets . We depart from this view by including a learned alignment step in the comparison . Each new image is aligned to a reference image and it is the resulting learned alignment score that guides the selection of which reference image it is most related to . Our approach of learning to align images to reference objects gives a degree of freedom to select reference objects that may differ substantially from the test images ( by shape , style or even type ) . By construction , the resulting alignment is also interpretable about the relation , and can be examined after the fact . The explicit alignment also allows a channel of feedback for learning beyond the label classification . We learn multi-scale image features , possibly different representations for reference and target images , and use them to learn point-to-point matches . These point matches which , are known for a self-alignment ( aligning an image to itself ) , can be used to learn with a much higher bandwidth signal than the traditional label-only signal . The method is specifically geared towards one-shot learning where a new set of reference objects are given , one example per category , and the new images are to be classified accordingly without additional labels . As our method computes an alignment between images , we call our model Alignment Based Matching Networks ( ABM Nets ) . We demonstrate that the matching images through ABM Nets can yield state-of-the-art accuracies for the one-shot learning task on Omniglot and MiniImageNet datasets . Further , we show that ABM networks outperforms these other state-of-the-art models on the open-set recognition task while in the one-shot setting by achieving a high accuracy of matching the non-open-set classes while maintaining a high F1 score for identifying samples in the open-set . Additionally , we show how ABM Nets provide for free , some understanding into why our network selects a label for a target image by querying individual point match likelihoods for pairs of points selected between the target and support images . Point matchings allow us to quickly determine the strength of the semantic information learned by the network . We contrast the strength of the semantic information learned for handwritten digit recognition to the more superficial information learned for real world images . Our main contributions can be summarized as follows : • We cast the one-shot learning task as the outcome of an alignment task and extend the Matching Network model for learning these explicit alignments • We introduce self-alignment based regularization as a mechanism of providing a high- bandwidth signal to an otherwise low-bandwidth task • We demonstrate state-of-the-art performance on the one-shot learning task for Omniglot and MiniImageNet tasks • We show how the same ABM network architecture can be used to perform open-set recogni- tion in the one-shot setting , outperforming existing alternatives The next section describes related work . We will then introduce the ABM model and then evaluate the model on the one-shot learning and one-shot open-set recognition tasks . 2 RELATED WORK . Matching points from one image to another has been traditionally performed with robust descriptors such as SIFT Lowe ( 1999 ) , HOG and their variants . A neural network can even be trained to learn the descriptor and orientation from dataYi et al . ( 2016 ) or to detect keypoints for patch-matching Altwaijry et al . ( 2016 ) . These methods enable us to identify distinctive objects under many different conditions such as affine transformations , orientation changes , different illuminations , etc . These descriptors can however , be sensitive to different realizations of an object type making them less suitable for object classification than CNNs . CNNs have dominated the visual classification tasks over the past few years and have grown increasingly complex and deep . Understanding why these networks produce specific results has not been an easy task . Most venues of explanations are provided either through visualizing the activation of various layers of a network or by generating images from the network Zeiler & Fergus ( 2014 ) ; Mahendran & Vedaldi ( 2015 ) ; Yosinski et al . ( 2015a ) . These give us insight into properties of the network such as the areas of visual attention of the network or types of properties captured by those nodes . However , the correspondence between parts of images and finer grained information is lost as we go to the higher layers of the network . Instead of finding correspondences between parts of images , image pairs can be directly embedded into a space where their similarity can be queried . Such models are used in one-shot learning settings to match pairs of images from previously unseen tasks . Networks like the Siamese neural networks Koch et al . ( 2015 ) encode images into a feature representation given by the top layer of a CNN and compute similarity in the CNN space . Matching networks Vinyals et al . ( 2016 ) take this a step further by comparing sets of labeled support images to unlabeled target images in the top level feature encoding space to find the closest match among the support images . They also design a training procedure to specifically train networks for one-shot learning . This has been extended by Prototypical Networks Snell et al . ( 2017 ) to use a prototype per class , corresponding to the mean embedded vector per class , rather than the mean distances to the embedded vectors to improve the performance of these networks in the few-shot learning setting . Finding the right training procedure specifically for one-shot learning has been abstracted out in meta-learning schemes which learn how to perform gradient descent to best optimize the models and how to best initialize them to good starting points . Meta-learning LSTMsRavi & Larochelle ( 2017 ) use LSTMs to predict parameter updates while Temporal Convolution based Meta Learners Mishra et al . ( 2017 ) use deep recurrent networks based on dilated convolutions for few-shot learning tasks . The meta-learning framework is quite powerful as it can be used as a means of rapid adaptation of deep nets in a model-agnostic way . Attentive Recurrent ComparatorsShyam & Dukkipati ( 2017 ) frame the one-shot learning task using an LSTM that takes multiple “ glimpses ” of a pair of images alternatively and repeatedly updating the hidden state of an RNN controller . The final hidden state of the controller is used to determine the similarity between pairs of images . The one-shot learning task can be made even more challenging when the test image does not correspond to any of the reference classes . The test image in this case belong to the open-set and the task of identifying such images is the open-set recognition task . Methods to solve the general open-set recognition task ( in the non-one-shot learning setting ) involve two primary components - inducing a distance metric over objects such that objects belonging to the same class have low distances and then establishing a Compact Abated Probability ( CAP ) model Scheirer et al . ( 2014 ) over these distances such that when the probability falls below a threshold for all known classes , then the object is determined to be in the open set . Weibull-calibrated SVMs ( W-SVM ) Scheirer et al . ( 2014 ) use support vector machines to learn a kernel function with which distances are computed and extreme value theory to establish the CAP model based on the distribution of distances obtained from the known classes . Sparse representation-based open-set recognition ( SROSR ) Zhang & Patel ( 2017 ) extend the sparse representation-based classification algorithm Wright et al . ( 2009 ) learns a sparse representation based on reconstruction errors and build a CAP model on the reconstruction error distribution to establish the open-set threshold . Bendale & Boult ( 2016 ) bring CAP models to deep learning models using a generalization of the softmax operator , the open-max , to learn the threshold for the open set . 3 METHOD . We present Alignment Based Matching Networks ( ABM Nets ) - a two tier matching model to match images via alignment . The first tier parameterizes the probability that a pixel in the test image matches a pixel in a reference image by forcing an alignment between the pair of images and computing an alignment score . The second tier aggregates the pixel alignment likelihoods between the test image and a set of reference images and uses the aggregate score to find the likelihood that the test image matches each of the reference images . 3.1 POINT MATCHING VIA ALIGNMENT . Formally , classifying a target image It given a set of support images S = { ( Ik , yk ) : i ∈ { 1 . . . k } } , where yk is the label for image Ik , corresponds to predicting the label yt which maximizesP ( yt|It , S ) . To align images , we first look pairwise matchings P ( yt = yk|It , Ik ) . Traditionally , this function is approximated using a neural net with parameters θ as Pθ ( yt = yk|It , Ik ) ∝exp ( D ( fθ ( It ) , gθ ( Ik ) ) ) Where f = g are the top layers of a CNNs that embed the images into a common space where a distance function can be applied ( such as cosine similarity ) . While this provides us a straightforward mechanism to match images , it treats the CNN as a black box and does not yield any insight into correspondences between images . Two examples from the same class can not be aligned in a straightforward way as the finer resolution information is lost as the network architecture gets deeper . To overcome this problem , we propose classifying images through an alignment rather than a full embedding of the image . We assume that there is an alignment M ( It , S ) ∈ M which matches points from It to S , but is not directly observable to us , whereM is the set of all possible image alignments of the test image to the reference image set . We express the alignment M ( It , S ) as a binary tensor Mi , j , k ( It , S ) which is 1 when pixel i in It matches to pixel j in Ik ∈ S. For brevity , we write M ( It , S ) =M |It , S. We can now express the probability assigning the label yk associated with image Ik ∈ S to the rest image as P ( yt = yk|It , S ) = ∑ M∈M P ( yt = yk|M , It , S ) · P ( M |It , S ) This formulation captures how closely two images are related to each other through the individual point-wise mapping . However , marginalizing over all possible matchings is intractable . To make the objective tractable , we make two main approximations . First , we approximate each point mapping as an independent indicator of the alignment , we get P ( M |It , S ) = ∏ i Pθ ( Mi , · , · = 1|It , S ) . Second , we assume that the correct alignment M∗ is much more likely than incorrect ones giving P ( yt = yk|It , S ) ≈ ∏ i max j P ( Mi , j , k = 1|It , S ) ( 1 ) As the label ŷt = maxk P ( yt = yk|It , S ) , the ABM model becomes a two-tier matching scheme - first a matching to align pixels between the images and second to match the label to the best aligning image . Figure 1 visually illustrates the two tier matching . By finding the matching across all support images at once , the individual pixel mapping learns to better contrast the matchings across various reference images . Expressing the cost of setting Mi , j , k = 1 as Ci , j ( It , Ik ) , we express the probability of a matching M as an independent point matching across the pixels in the test image as P ( Mi , j , k|It , S ) ∝ exp ( −Ci , j ( It , Ik ) ) ( 2 ) We compute the alignment cost as distances in an embedded space Ci , j ( It , Is ) = D ( φ ( It , i ) , φ ( Is , j ) ) . As just the plain RGB values do not convey any semantic notions such as object type or shape , we use a hyper-column descriptor Hariharan et al . ( 2015 ) using a CNN with parameters θ to learn the pixel embedding φ = ψθ ( I· , i ) . The initial few dimensions of this feature space are designed to provide high resolution information of the pixels which is needed for fine-grained alignment while the remaining dimensions are designed provide increasingly more contextual information useful for distinguishing between pixels that would other wise be identical without context . In our experiments , we use cosine distances as the distance measure to be consistent with Vinyals et al . ( 2016 ) . Note that we can compare dissimilar test and reference images ( such as gray-scale reference images and RGB test images of different sizes ) by using separate encoders φ1 , φ2 . Also , note that the greedy matching can result in multiple points in the test image matching to the same point in the reference image . If the matching has to be unique , one can use the Hungarian algorithm Kuhn ( 1955 ) ( or its GPU capable counterpart , the Auction algorithm Vasconcelos & Rosenhahn ( 2009 ) ) . Finally , we note that in order to perform the alignments , we do require additional computation . The comparison phase of Matching Networks Vinyals et al . ( 2016 ) when encoding to a dL dimensional feature space takesO ( dLns ) time for comparing against ns reference images . On the other hand , a full pixel-to-pixel alignment on images of size m× n with ABM Networks takes time O ( mnns ( ∑ l dl ) ) . For both models , encoding the image using a CNN takes O ( mnnsd1 ) time for computing the first hidden layer . Hence , even with the pixel-wise matching , the asymptotic runtime complexity remains the same . In practice , the constant factor multiplier can make aligning all points relatively slow . In our experiments , we found that matching 10 % of the test image to 20 % of the reference image allows us to maintain high performance without impacting the total runtime significantly .

Summary: Authors argue that using average (independent) greedy matching of pixel embedding (based on 4-6 layer cnn hypercolumns) is a better metric for one-shot learning than just using final layer embedding of a 4-6 layer cnn for the whole image.  Their argument is backed by outperforming their baseline and getting competitive results on few shot learning tasks. Their method is much more computationally heavy than the baseline matching networks. In order to make training feasible, in practice they train with 90% dropout of test pixels embedding & 80% dropout of reference pixels embedding.