Alignment Based Mathching Networks for One-Shot Classification and Open-Set Recognition

1 INTRODUCTION . Convolutional neural networks ( CNNs ) in various realizations have come to dominate image , speech , and even text processing . Performance gains afforded by such architectures , with learned , multi-stage image features , emerge from large amounts of supervised training data . In addition to being data hungry , the networks are also challenging to interpret after the fact , potentially yielding confident yet unwarranted predictions for tailored examplesYosinski et al . ( 2015b ) ; Szegedy et al . ( 2013 ) . Our goal in contrast is to learn from very few examples , and specifically tailor the models to emphasize interpretability . Deep , complex architectures , despite large numbers of parameters involved , are not incompatible with one-shot or open-set learning . Instead of predicting categories directly , one can set up the learning problem to recognize sameness between pairs of images as in Siamese networks Koch et al . ( 2015 ) or between a set of support images as in Matching Networks Vinyals et al . ( 2016 ) . In this case , a relatively large dataset of pairs is required to help the network learn features that summarize images in a manner that is tailored for assessing category differences . Such a network can succeed in predicting sameness for entirely new categories or comparing new images to a set of given reference sets . We depart from this view by including a learned alignment step in the comparison . Each new image is aligned to a reference image and it is the resulting learned alignment score that guides the selection of which reference image it is most related to . Our approach of learning to align images to reference objects gives a degree of freedom to select reference objects that may differ substantially from the test images ( by shape , style or even type ) . By construction , the resulting alignment is also interpretable about the relation , and can be examined after the fact . The explicit alignment also allows a channel of feedback for learning beyond the label classification . We learn multi-scale image features , possibly different representations for reference and target images , and use them to learn point-to-point matches . These point matches which , are known for a self-alignment ( aligning an image to itself ) , can be used to learn with a much higher bandwidth signal than the traditional label-only signal . The method is specifically geared towards one-shot learning where a new set of reference objects are given , one example per category , and the new images are to be classified accordingly without additional labels . As our method computes an alignment between images , we call our model Alignment Based Matching Networks ( ABM Nets ) . We demonstrate that the matching images through ABM Nets can yield state-of-the-art accuracies for the one-shot learning task on Omniglot and MiniImageNet datasets . Further , we show that ABM networks outperforms these other state-of-the-art models on the open-set recognition task while in the one-shot setting by achieving a high accuracy of matching the non-open-set classes while maintaining a high F1 score for identifying samples in the open-set . Additionally , we show how ABM Nets provide for free , some understanding into why our network selects a label for a target image by querying individual point match likelihoods for pairs of points selected between the target and support images . Point matchings allow us to quickly determine the strength of the semantic information learned by the network . We contrast the strength of the semantic information learned for handwritten digit recognition to the more superficial information learned for real world images . Our main contributions can be summarized as follows : • We cast the one-shot learning task as the outcome of an alignment task and extend the Matching Network model for learning these explicit alignments • We introduce self-alignment based regularization as a mechanism of providing a high- bandwidth signal to an otherwise low-bandwidth task • We demonstrate state-of-the-art performance on the one-shot learning task for Omniglot and MiniImageNet tasks • We show how the same ABM network architecture can be used to perform open-set recogni- tion in the one-shot setting , outperforming existing alternatives The next section describes related work . We will then introduce the ABM model and then evaluate the model on the one-shot learning and one-shot open-set recognition tasks . 2 RELATED WORK . Matching points from one image to another has been traditionally performed with robust descriptors such as SIFT Lowe ( 1999 ) , HOG and their variants . A neural network can even be trained to learn the descriptor and orientation from dataYi et al . ( 2016 ) or to detect keypoints for patch-matching Altwaijry et al . ( 2016 ) . These methods enable us to identify distinctive objects under many different conditions such as affine transformations , orientation changes , different illuminations , etc . These descriptors can however , be sensitive to different realizations of an object type making them less suitable for object classification than CNNs . CNNs have dominated the visual classification tasks over the past few years and have grown increasingly complex and deep . Understanding why these networks produce specific results has not been an easy task . Most venues of explanations are provided either through visualizing the activation of various layers of a network or by generating images from the network Zeiler & Fergus ( 2014 ) ; Mahendran & Vedaldi ( 2015 ) ; Yosinski et al . ( 2015a ) . These give us insight into properties of the network such as the areas of visual attention of the network or types of properties captured by those nodes . However , the correspondence between parts of images and finer grained information is lost as we go to the higher layers of the network . Instead of finding correspondences between parts of images , image pairs can be directly embedded into a space where their similarity can be queried . Such models are used in one-shot learning settings to match pairs of images from previously unseen tasks . Networks like the Siamese neural networks Koch et al . ( 2015 ) encode images into a feature representation given by the top layer of a CNN and compute similarity in the CNN space . Matching networks Vinyals et al . ( 2016 ) take this a step further by comparing sets of labeled support images to unlabeled target images in the top level feature encoding space to find the closest match among the support images . They also design a training procedure to specifically train networks for one-shot learning . This has been extended by Prototypical Networks Snell et al . ( 2017 ) to use a prototype per class , corresponding to the mean embedded vector per class , rather than the mean distances to the embedded vectors to improve the performance of these networks in the few-shot learning setting . Finding the right training procedure specifically for one-shot learning has been abstracted out in meta-learning schemes which learn how to perform gradient descent to best optimize the models and how to best initialize them to good starting points . Meta-learning LSTMsRavi & Larochelle ( 2017 ) use LSTMs to predict parameter updates while Temporal Convolution based Meta Learners Mishra et al . ( 2017 ) use deep recurrent networks based on dilated convolutions for few-shot learning tasks . The meta-learning framework is quite powerful as it can be used as a means of rapid adaptation of deep nets in a model-agnostic way . Attentive Recurrent ComparatorsShyam & Dukkipati ( 2017 ) frame the one-shot learning task using an LSTM that takes multiple “ glimpses ” of a pair of images alternatively and repeatedly updating the hidden state of an RNN controller . The final hidden state of the controller is used to determine the similarity between pairs of images . The one-shot learning task can be made even more challenging when the test image does not correspond to any of the reference classes . The test image in this case belong to the open-set and the task of identifying such images is the open-set recognition task . Methods to solve the general open-set recognition task ( in the non-one-shot learning setting ) involve two primary components - inducing a distance metric over objects such that objects belonging to the same class have low distances and then establishing a Compact Abated Probability ( CAP ) model Scheirer et al . ( 2014 ) over these distances such that when the probability falls below a threshold for all known classes , then the object is determined to be in the open set . Weibull-calibrated SVMs ( W-SVM ) Scheirer et al . ( 2014 ) use support vector machines to learn a kernel function with which distances are computed and extreme value theory to establish the CAP model based on the distribution of distances obtained from the known classes . Sparse representation-based open-set recognition ( SROSR ) Zhang & Patel ( 2017 ) extend the sparse representation-based classification algorithm Wright et al . ( 2009 ) learns a sparse representation based on reconstruction errors and build a CAP model on the reconstruction error distribution to establish the open-set threshold . Bendale & Boult ( 2016 ) bring CAP models to deep learning models using a generalization of the softmax operator , the open-max , to learn the threshold for the open set . 3 METHOD . We present Alignment Based Matching Networks ( ABM Nets ) - a two tier matching model to match images via alignment . The first tier parameterizes the probability that a pixel in the test image matches a pixel in a reference image by forcing an alignment between the pair of images and computing an alignment score . The second tier aggregates the pixel alignment likelihoods between the test image and a set of reference images and uses the aggregate score to find the likelihood that the test image matches each of the reference images . 3.1 POINT MATCHING VIA ALIGNMENT . Formally , classifying a target image It given a set of support images S = { ( Ik , yk ) : i ∈ { 1 . . . k } } , where yk is the label for image Ik , corresponds to predicting the label yt which maximizesP ( yt|It , S ) . To align images , we first look pairwise matchings P ( yt = yk|It , Ik ) . Traditionally , this function is approximated using a neural net with parameters θ as Pθ ( yt = yk|It , Ik ) ∝exp ( D ( fθ ( It ) , gθ ( Ik ) ) ) Where f = g are the top layers of a CNNs that embed the images into a common space where a distance function can be applied ( such as cosine similarity ) . While this provides us a straightforward mechanism to match images , it treats the CNN as a black box and does not yield any insight into correspondences between images . Two examples from the same class can not be aligned in a straightforward way as the finer resolution information is lost as the network architecture gets deeper . To overcome this problem , we propose classifying images through an alignment rather than a full embedding of the image . We assume that there is an alignment M ( It , S ) ∈ M which matches points from It to S , but is not directly observable to us , whereM is the set of all possible image alignments of the test image to the reference image set . We express the alignment M ( It , S ) as a binary tensor Mi , j , k ( It , S ) which is 1 when pixel i in It matches to pixel j in Ik ∈ S. For brevity , we write M ( It , S ) =M |It , S. We can now express the probability assigning the label yk associated with image Ik ∈ S to the rest image as P ( yt = yk|It , S ) = ∑ M∈M P ( yt = yk|M , It , S ) · P ( M |It , S ) This formulation captures how closely two images are related to each other through the individual point-wise mapping . However , marginalizing over all possible matchings is intractable . To make the objective tractable , we make two main approximations . First , we approximate each point mapping as an independent indicator of the alignment , we get P ( M |It , S ) = ∏ i Pθ ( Mi , · , · = 1|It , S ) . Second , we assume that the correct alignment M∗ is much more likely than incorrect ones giving P ( yt = yk|It , S ) ≈ ∏ i max j P ( Mi , j , k = 1|It , S ) ( 1 ) As the label ŷt = maxk P ( yt = yk|It , S ) , the ABM model becomes a two-tier matching scheme - first a matching to align pixels between the images and second to match the label to the best aligning image . Figure 1 visually illustrates the two tier matching . By finding the matching across all support images at once , the individual pixel mapping learns to better contrast the matchings across various reference images . Expressing the cost of setting Mi , j , k = 1 as Ci , j ( It , Ik ) , we express the probability of a matching M as an independent point matching across the pixels in the test image as P ( Mi , j , k|It , S ) ∝ exp ( −Ci , j ( It , Ik ) ) ( 2 ) We compute the alignment cost as distances in an embedded space Ci , j ( It , Is ) = D ( φ ( It , i ) , φ ( Is , j ) ) . As just the plain RGB values do not convey any semantic notions such as object type or shape , we use a hyper-column descriptor Hariharan et al . ( 2015 ) using a CNN with parameters θ to learn the pixel embedding φ = ψθ ( I· , i ) . The initial few dimensions of this feature space are designed to provide high resolution information of the pixels which is needed for fine-grained alignment while the remaining dimensions are designed provide increasingly more contextual information useful for distinguishing between pixels that would other wise be identical without context . In our experiments , we use cosine distances as the distance measure to be consistent with Vinyals et al . ( 2016 ) . Note that we can compare dissimilar test and reference images ( such as gray-scale reference images and RGB test images of different sizes ) by using separate encoders φ1 , φ2 . Also , note that the greedy matching can result in multiple points in the test image matching to the same point in the reference image . If the matching has to be unique , one can use the Hungarian algorithm Kuhn ( 1955 ) ( or its GPU capable counterpart , the Auction algorithm Vasconcelos & Rosenhahn ( 2009 ) ) . Finally , we note that in order to perform the alignments , we do require additional computation . The comparison phase of Matching Networks Vinyals et al . ( 2016 ) when encoding to a dL dimensional feature space takesO ( dLns ) time for comparing against ns reference images . On the other hand , a full pixel-to-pixel alignment on images of size m× n with ABM Networks takes time O ( mnns ( ∑ l dl ) ) . For both models , encoding the image using a CNN takes O ( mnnsd1 ) time for computing the first hidden layer . Hence , even with the pixel-wise matching , the asymptotic runtime complexity remains the same . In practice , the constant factor multiplier can make aligning all points relatively slow . In our experiments , we found that matching 10 % of the test image to 20 % of the reference image allows us to maintain high performance without impacting the total runtime significantly .

Summary: This paper proposed a new way of learning point-wise alignment of two images, and based on this idea, one-shot classification and open-set recognition can be further improved. The idea is interesting. As a human, when we say two images are similar, we may compare them locally and globally in our mind. However, traditional CNN models do not make direct comparisons. And this work give a good direction to further improve this motivation.