Learning from One and Only One Shot

1 INTRODUCTION . Modern machine learning ( ML ) has made remarkable progress , but this is accompanied by increasing model complexity , with hundreds of neural layers ( e.g. , ResNet-152 ) and millions of parameters ( e.g. , AlexNet : 62.3M , VGG16 : 138M , BERT : 110M , GTP-3 : 175B ) . This results in a huge appetite for data and increasing difficulty in model interpretability—both for users to understand and for developers to tune ( e.g. , hyperparameters , architecture ) . As such , AI researchers have pushed for ML models that are prior- and data-efficient ( Chollet , 2019 ) , that are human-like ( Lake et al. , 2015 ) , and that exhibit human-interpretable behaviors ( Adadi & Berrada , 2018 ) . This poses the fundamental scientific question : How can humans learn so much from so little ( May , 2015 ) , whereas ML models , e.g. , those achieving near-perfection on MNIST using all 60k training images , deteriorate rapidly as ( pre ) training reduces ? Being data-hungry can pose a real challenge in data-scarce domains , e.g. , in a rapidly evolving pandemic or a low-resource environment . Focusing on its scientific contribution , this paper presents a theoretically sound , white-box model that learns like humans do , with initial success on a first set of benchmarks . This includes our state-of-the-art results of hitting 80 % / 90 % MNIST accuracy using the only first / first four training images per class and achieving 6.75 % error ( human performance is 4.5 % ) in the Omniglot one-shot learning challenge without pre-training—one and only one shot . We follow the nativist principle : given an example , humans make abstractions : we envision an equivalence class of unseen examples , equivalent to the given , in several abstract senses . Many abstraction abilities are inborn in humans and occur unconsciously . When a baby sees a mug , ( s ) he can immediately recognize it regardless of whether it is translated , rotated , scaled , or deformed . Such abstraction abilities are considered innate as Core Knowledge priors ( Spelke & Kinzler , 2007 ) , rather than acquired later in life such as learning that a mug is topologically a doughnut . We computationally realize the above intuition via our so-called distortable canvas—imagining every image smoothly painted on a rubber canvas that can be distorted in many ways , e.g. , bent , stretched , squeezed ( Figure 1A ) . Due to rubber ’ s viscosity , more distorted canvas transformations expend more energy . This induces a topological similarity , or distance , based on minimal energy : two images are similar if one can almost transform into the other with little distortion ( energy ) . Distance is computed by minimizing color and canvas distortions . In general , visual similarity involves not only a canvas but also a color transformation . Here , we focus on canvas transformations and grayscale images only . Besides a final desired transformation ( and distance ) , we apply the minimal-energy principle to the entire transformation process , i.e. , to keep canvas and color distortions small along the entire optimization process ( Mesa et al. , 2019 ) . When perceiving a translation , our mind does not process it as a sudden displacement from one location to another , but auto-completes a translation path— continuous and preferably short . Gradient descent naturally fits this goal by always choosing the steepest descent . Yet , it suffers from the curse of local minima . Our solution is to lift gradient descent to multiple levels of abstraction via multiscale canvas lattices and color blurring , mimicking human abstraction ability that is extremely flexible in multiscale optimization . This yields visualizable and interpretable transformation flows ( Figure 1B ) that either match human intuition ( e.g. , what humans would naturally do to transform a “ 7 ” to a “ 1 ” ) or provide human intuition to initially nonintuitive settings . The latter case suggests new , human-interpretable transformations : “ ah , I did not realize this other way of transforming ‘ 7 ’ into ‘ 1 ’ , but now I see it and it makes perfect sense ! ” We show initial success on abstract visual tasks such as character and doodle recognition ( future work generalizes to photorealistic images after a preprocessing technique to turn them into abstract doodles or “ emojis ” ) . Our model can be used with the simple nearest-neighbor method to classify images and also in a simple k-means style to cluster images . On benchmarks including MNIST ( LeCun et al. , 1998 ) , EMNIST-letters ( Cohen et al. , 2017 ) , and the more taxing Omniglot challenge ( Lake et al. , 2015 ) , running nearest-neighbor on our learned similarity space outperforms both neural-networkbased and classical ML in the “ tiny-data ” or single-datum regime . This includes beating few-shot models pre-trained on extra background data ( which our model does not require ) . On MNIST , we need one and only one training image per class to hit 80 % accuracy and four to hit 90 % . In an unsupervised setting , our model enables k-means-like clustering , but on our learned similarity space . We generate archetypes as cluster “ centroids ” , e.g. , different ways of writing “ 7 ” or doodling giraffes . Literature on data scarcity : few-shot learning ( FSL ) . One-/few-shot learning ( Lake et al. , 2011 ; Wang et al. , 2020 ) via transfer or meta learning ( Pan & Yang , 2009 ; Finn et al. , 2017 ; Hsu et al. , 2019 ) has achieved impressive success in data-scarce scenarios . FSL ’ s success relies on the assumption that source and target tasks are similar enough for pre-training to be relevant ( Storkey , 2009 ) . However , knowing a priori how relevant the tasks are and understanding what has been pre-learned are often considered “ black art ” . It remains challenging for FSL practitioners to pick proper source data/models for pre-training so as to transfer most effectively and avoid negative transfer ( Pan & Yang , 2009 ; Meiseles & Rokach , 2020 ) . This is especially the case in new domains ( e.g. , discovering rising trends ) over classical ones ( e.g. , vision , language ) . So , rather than “ big transfer ” ( Kolesnikov et al. , 2020 ) , the Omniglot challenge urges “ small transfer ” —advocating reduced pre-training ( Lake et al. , 2019 ) . This paper follows Omniglot ’ s pursuit and pushes such reduction to the limit : with absolutely zero pre-training , we still achieved near-human performance and human-interpretability . Literature on data scarcity : transformation and data augmentation . Our distortable canvas is conceptually akin to other transformation-based models , e.g. , optimal-transport maps ( Villani , 2009 ) , transformation-induced descriptors ( Lowe , 1999 ) and equivariances ( Bronstein et al. , 2017 ) . Like all these models , we build transformations into the model rather than into the data as through data augmentation ( Krizhevsky et al. , 2012 ) . This has the native advantage of harnessing transformational properties directly rather than learning them from data . Models with built-in transformations may be viewed as those perfectly learned from infinitely augmented data . Unlike transformations commonly considered in existing models and data augmentation techniques , we do not encode domain knowledge about preferred transformations such as translation , rotation , or scaling . Instead , our model considers all transformations while still maintaining efficiency via our abstracted gradient descent . 2 SMOOTH IMAGE ON DISTORTABLE CANVAS . We introduce a distortable canvas model : any image is thought of as smoothly painted on a rubber canvas that can be bent , stretched , etc . We further introduce canvas transformations that can flexibly “ distort ” an image as we naturally simulate in our mind . More specifically , we define a smooth image by a piecewise differentiableM : R2 → R+ , where R2 denotes an infinite canvas and R+ denotes color ( grayscale in this paper ) . We define a canvas transformation by α : R2 → R2 , which “ reshapes ” the underlying canvas of a smooth image . Examples include translation , rotation , scaling , and more . We also define a color transformation by χ : R+ → R+ , which “ repaints ” a color . In this paper , we simplify color transformation and only use it to adjust image contrast via affine χ ( c ) : = ac+ b . Nevertheless , we do not restrict canvas transformation , but consider all 2D transformations . Given M , α , χ , the composition χ ◦M ◦ α denotes the transformed image ofM by transformations α , χ . To mimic innate human intuition about topological similarity , we introduce canvas distortion DV ( α ) for any canvas transformation α and color distortion DC ( M , M′ ) between two smooth images M , M′ . Our idea is to search for a transformation that mimics what humans naturally do to transform one image into another . That is , a low-distorted α which makes little difference in color betweenM and the transformedM′ , or more precisely , an α that minimizes bothDV ( α ) andDC ( M , χ◦M′◦α ) . Representating digital and smoothed images . Anm×n digital image is a discrete M : [ m ] × [ n ] → [ 0 , 1 ] , where [ k ] : = { 0 , 1 , . . . , k − 1 } . We call [ m ] × [ n ] the canvas grid and any z ∈ [ m ] × [ n ] a grid point . For any m× n digital image M , we smooth it toM via a sum of kernels : M ( x ) : = ∑ z∈ [ m ] × [ n ] M ( z ) · κ ( ρ ( z , x ) ) for any x ∈ R2 , ( 1 ) where a kernel κ : R+ → R+ is a decaying function ( e.g. , linear , polynomial , Gaussian decay ) and ρ is a metric on R2 ( e.g. , ` 1 , ` 2 , ` ∞ ) . In this paper , we use linear decay and ` ∞ , i.e. , κ ( ρ ( z , x ) ) = 1− 1ρc ‖z − x‖∞ if ‖z − x‖∞ < ρc ( for some cutoff radius ρc > 0 ) and κ ( ρ ( z , x ) ) = 0 otherwise . Note : M is defined everywhere on R2 . This differs from Gaussian blurring as we do not discretize kernels . It is key to use the smoothed image as input , which allows computing gradients analytically . As such , we always smooth any digital image at first and then only manipulate the smoothed image . Representing arbitrary canvas transformations . We consider all 2D transformations ( including those without a formula ) , but how do we represent them in a computer ? With respect to the standard grid [ m ] × [ n ] , we use the transformed grid α ( [ m ] × [ n ] ) to represent α digitally . Thus , any canvas transformation α is digitally represented by ( d= ) a matrix α ∈ R ( mn ) ×2 whose ith row is the 2D coordinate of the transformed ith grid point . We use the lexicographical order of a 2D grid , e.g. , with respect to [ 2 ] × [ 3 ] , the identify transformation id d= id = [ [ 0 , 0 ] , [ 0 , 1 ] , [ 0 , 2 ] , [ 1 , 0 ] , [ 1 , 1 ] , [ 1 , 2 ] ] . Any transformed imageM◦α d=M ( α ) : = ( M ( α0 ) , . . . , M ( α ( mn−1 ) ) ) ∈ R ( mn ) , i.e. , a ( vectorized ) digital image sampled fromM at the transformed grid α . Representing color and canvas distortions . The color distortionDC measures the color discrepancy between M ( id ) and M′ ( α ) up to an affine color transformation χ . The canvas distortion DV measures the distortion between the original grid id and the transformed grid α . Formally , DC ( M , χ ◦M′ ◦ α ) d= DC ( M ( id ) , χ ( M′ ( α ) ) ) : = ‖aM′ ( α ) + b−M ( id ) ‖22 ( 2 ) DV ( α ) d= DV ( id , α ) : = max { { i , j } , { i′ , j′ } } ∈BE ∣∣∣∆α { i , j } −∆α { i′ , j′ } ∣∣∣ , ∆α { i , j } : = log ‖αi −αj‖2‖idi − idj‖2 ( 3 ) Here , BE comprises all pairs of neighboring edges in a canvas lattice ( introduced below ) . Eq . ( 3 ) is derived from the mathematical definition of distortion of a function by discretizing it across the canvas lattice . This formula measures how far an arbitrary transformation is from being conformal , which is flexible for local isometries and scaling . Given a canvas grid [ m ] × [ n ] , its corresponding canvas lattice is an undirected graphL = ( V , E ) , with the set of vertices V = [ m ] × [ n ] and the set of edges obtained by connecting neighboring vertices in the ` ∞ sense : E = { { i , j } | ‖vi − vj‖∞ = 1 for vi , vj ∈ V } . We say two edges are neighbors if they form a 45◦ angle ( Figure 2 ) . Computing topological distance by minimizing distortions . To minimize color and canvas distortions ( 2 ) and ( 3 ) , we consider two dual views : minimizingDC among low-distorted α ’ s or minimizing DV among best-matching α ’ s . We write the two views as the following two constrained optimization problems , together with their respective unconstrained equivalents : with → 0+ and µ→ 0+ , min . α , χ DC ( M , χ ◦M′ ◦ α ) s.t . DV ( α ) ≤ ⇐⇒ min . α , χ DV ( α ) + µDC ( M , χ ◦M′ ◦ α ) ( 4 ) min . α , χ DV ( α ) s.t . DC ( M , χ ◦M′ ◦ α ) ≤ ⇐⇒ min . α , χ DC ( M , χ ◦M′ ◦ α ) + µDV ( α ) ( 5 ) We let the optima D ? C for ( 4 ) and D ? V for ( 5 ) denote two versions of our desired topological distance that mimics innate human intuition . We call them DC-distance and DV -distance , respectively . Transformation flow . To obtain both transformations and transformation processes that are humanlike , we run ( projected ) gradient descent . The iterative gradient steps yield not only a transformation α ? in the end but also a transformation flow id = α ( 0 ) → α ( 1 ) → · · · → α ? . The resulting sequence of transformed imagesM′ = M′ ◦ α ( 0 ) → M′ ◦ α ( 1 ) → · · · → M′ ◦ α ? ≈ M ( we omit χ for simplicity ) makes up an animation ( Figure 1 ) , which helps with human intuition on transformingM′ toM . However , directly running ( projected ) gradient descent on ( 4 ) or ( 5 ) does not work , because it suffers from the curse of local minima , which we discuss and solve in the next section .

Summary: The paper on hand tackles the problem of learning from very few samples, which is of high relevance for many machine learning problems. To this end, an approach to model transformation-based topological similarity is introduced, allowing for covering many kinds of invariants in image data. The approach is demonstrated for well-known benchmark datasets for different classification models, demonstrating that in this way just using a small number of samples competitive classification results can be obtained.