Title: FEW-SHOT AUTOREGRESSIVE DENSITY ESTIMATION: TOWARDS LEARNING TO LEARN DISTRIBUTIONS

Abstract: Deep autoregressive models have shown state-of-the-art performance in density estimation for natural images on large-scale datasets such as ImageNet. However, such models require many thousands of gradient-based weight updates and unique image examples for training. Ideally, the models would rapidly learn visual concepts from only a handful of examples, similar to the manner in which humans learns across many vision tasks. In this paper, we show how 1) neural attention and 2) meta learning techniques can be used in combination with autoregressive models to enable effective few-shot density estimation. Our proposed modifications to PixelCNN result in state-of-the art few-shot density estimation on the Omniglot dataset. Furthermore, we visualize the learned attention policy and find that it learns intuitive algorithms for simple tasks such as image mirroring on ImageNet and handwriting on Omniglot without supervision. Finally, we extend the model to natural images and demonstrate few-shot image generation on the Stanford Online Products dataset.

1. INTRODUCTION:

Contemporary machine learning systems are still far behind humans in their ability to rapidly learn new visual concepts from only a few examples (Lake et al., 2013) . This setting, called few-shot learning, has been studied using deep neural networks and many other approaches in the context of discriminative models, for example Vinyals et al. (2016) ; Santoro et al. (2016) . However, comparatively little attention has been devoted to the task of few-shot image density estimation; that is, the problem of learning a model of a probability distribution from a small number of examples. Below we motivate our study of few-shot autoregressive models, their connection to meta-learning, and provide a comparison of multiple approaches to conditioning in neural density models.

WHY AUTOREGRESSIVE MODELS?:

Autoregressive neural networks are useful for studying few-shot density estimation for several reasons. They are fast and stable to train, easy to implement, and have tractable likelihoods, allowing us to quantitatively compare a large number of model variants in an objective manner. Therefore we can easily add complexity in orthogonal directions to the generative model itself.

Autoregressive image models factorize the joint distribution into per-pixel factors:

P (x|s; θ) = N t=1 P (x t |x <t , f (s); θ) (1)

where θ are the model parameters, x ∈ R N are the image pixels, s is a conditioning variable, and f is a function encoding this conditioning variable. For example in text-to-image synthesis, s would be an image caption and f could be a convolutional or recurrent encoder network, as in Reed et al. (2016) . In label-conditional image generation, s would be the discrete class label and f could simply convert s to a one-hot encoding possibly followed by an MLP.

A straightforward approach to few-shot density estimation would be to simply treat samples from the target distribution as conditioning variables for the model. That is, let s correspond to a few data examples illustrating a concept. For example, s may consist of four images depicting bears, and the task is then to generate an image x of a bear, or to compute its probability P (x|s; θ).

A learned conditional density model that conditions on samples from its target distribution is in fact learning a learning algorithm, embedded into the weights of the network. This learning algorithm is executed by a feed-forward pass through the network encoding the target distribution samples.

WHY LEARN TO LEARN DISTRIBUTIONS?:

If the number of training samples from a target distribution is tiny, then using standard gradient descent to train a deep network from scratch or even fine-tuning is likely to result in memorization of the samples; there is little reason to expect generalization. Therefore what is needed is a learning algorithm that can be expected to work on tiny training sets. Since designing such an algorithm has thus far proven to be challenging, one could try to learn the algorithm itself. In general this may be impossible, but if there is shared underlying structure among the set of target distributions, this learning algorithm can be learned from experience as we show in this paper.

For our purposes, it is instructive to think of learning to learn as two nested learning problems, where the inner learning problem is less constrained than the outer one. For example, the inner learning problem may be unsupervised while the outer one may be supervised. Similarly, the inner learning problem may involve only a few data points. In this latter case, the aim is to meta-learn a model that when deployed is able to infer, generate or learn rapidly using few data s.

A rough analogy can be made to evolution: a slow and expensive meta-learning process, which has resulted in life-forms that at birth already have priors that facilitate rapid learning and inductive leaps. Understanding the exact form of the priors is an active, very challenging, area of research (Spelke & Kinzler, 2007; Smith & Gasser, 2005) . From this research perspective, we can think of meta-learning as a potential data-driven alternative to hand engineering priors.

The meta-learning process can be undertaken using large amounts of computation and data. The output is however a model that can learn from few data. This facilitates the deployment of models in resource-constrained computing devices, e.g. mobile phones, to learn from few data. This may prove to be very important for protection of private data s and for personalisation.

FEW-SHOT LEARNING AS INFERENCE OR AS A WEIGHT UPDATE?:

A sample-conditional density model P θ (x|s) treats meta-learning as inference; the conditioning samples s vary but the model parameters θ are fixed. A standard MLP or convolutional network can parameterize the sample encoding (i.e. meta-learning) component, or an attention mechanism can be used, which we will refer to as PixelCNN and Attention PixelCNN, respectively.

A very different approach to meta-learning is taken by Ravi & Larochelle (2016) and Finn et al. (2017a) , who instead learn unconditional models that adapt their weights based on a gradient step computed on the few-shot samples. This same approach can also be taken with PixelCNN: train an unconditional network P θ (x) that is implicitly conditioned by a previous gradient ascent step on log P θ (s); that is, θ = θ -α∇ θ log P θ (s). We will refer to this as Meta PixelCNN.

In Section 2 we connect our work to previous attentive autoregressive models, as well as to work on gradient based meta-learning. In Section 3 we describe Attention PixelCNN and Meta PixelCNN in greater detail. We show how attention can improve performance in the the few-shot density estimation problem by enabling the model to easily transmit texture information from the support set onto the target image canvas. In Section 4 we compare several few-shot PixelCNN variants on simple image mirroring, Omniglot and Stanford Online Products. We show that both gradient-based and attention-based few-shot PixelCNN can learn to learn simple distributions, and both achieve state-of-the-art likelihoods on Omniglot.

2. RELATED WORK:

Learning to learn or meta-learning has been studied in cognitive science and machine learning for decades (Harlow, 1949; Thrun & Pratt, 1998; Hochreiter et al., 2001) . In the context of modern deep networks, Andrychowicz et al. (2016) learned a gradient descent optimizer by gradient descent, itself parameterized as a recurrent network. Chen et al. (2017) showed how to learn to learn by gradient descent in the black-box optimization setting.

Ravi & Larochelle (2017) showed the effectiveness of learning an optimizer in the few-shot learning setting. Finn et al. (2017a) advanced a simplified yet effective variation in which the optimizer is not learned but rather fixed as one or a few steps of gradient descent, and the meta-learning problem reduces to learning an initial set of base parameters θ that can be adapted to minimize any task loss L t by a single step of gradient descent, i.e. θ = θ -α∇L t (θ). This approach was further shown to be effective in imitation learning including on real robotic manipulation tasks (Finn et al., 2017b) . Shyam et al. (2017) train a neural attentive recurrent comparator function to perform oneshot classification on Omniglot.

Few-shot density estimation has been studied previously using matching networks (Bartunov & Vetrov, 2016) and variational autoencoders (VAEs). Bornschein et al. (2017) apply variational inference to memory addressing, treating the memory address as a latent variable. Rezende et al. (2016) develop a sequential generative model for few-shot learning, generalizing the Deep Recurrent Attention Writer (DRAW) model (Gregor et al., 2015) . In this work, our focus is on extending autoregressive models to the few-shot setting, in particular PixelCNN (van den Oord et al., 2016) .

Autoregressive (over time) models with attention are well-established in language tasks. Bahdanau et al. ( 2014) developed an attention-based network for machine translation. This work inspired a wave of recurrent attention models for other applications. Xu et al. (2015) used visual attention to produce higher-quality and more interpretable image captioning systems. This type of model has also been applied in motor control, for the purpose of imitation learning. Duan et al. (2017) learn a policy for robotic block stacking conditioned on a small number of demonstration trajectories.

Gehring et al. ( 2017) developed convolutional machine translation models augmented with attention over the input sentence. A nice property of this model is that all attention operations can be batched over time, because one does not need to unroll a recurrent net during training. Our attentive Pixel-CNN is similar in high-level design, but our data is pixels rather than words, and 2D instead of 1D, and we consider image generation rather than text generation as our task.

3. MODEL:



3.1. FEW-SHOT LEARNING WITH ATTENTION PIXELCNN:

In this section we describe the model, which we refer to as Attention PixelCNN. At a high level, it works as follows: at the point of generating every pixel, the network queries a memory. This memory can consist of anything, but in this work it will be a support set of images of a visual concept. In addition to global features derived from these support images, the network has access to textures via support image patches. Figure 2 illustrates the attention mechanism.

In previous conditional PixelCNN works, the encoding f (s) was shared across all pixels. However, this can be sub-optimal for several reasons. First, at different points of generating the target image x, different aspects of the support images may become relevant. Second, it can make learning difficult, because the network will need to encode the entire support set of images into a single global conditioning vector, fed to every output pixel. This single vector would need to transmit information across all pairs of salient regions in the supporting images and the target image. To overcome this difficulty, we propose to replace the simple encoder function f (s) with a contextsensitive attention mechanism f t (s, x <t ). It produces an encoding of the context that depends on the image generated up until the current step t. The weights are shared over t.

f t (s, x <t )

Target image, x Support image, s attn mul q t p value p key α reduce sum 1 x 1 x P K x K x P We will use the following notation. Let the target image be x ∈ R H×W ×3 . and the support set images be s ∈ R S×H×W ×3 , where S is the number of supports.

K x K x P W x H x P W x H x 3 W x H x 3 K x K x 1 K x K x P

To capture texture information, we encode all supporting images with a shallow convolutional network, typically only two layers. Each hidden unit of the resulting feature map will have a small receptive field, e.g. corresponding to a 10 × 10 patch in a support set image. We encode these support images into a set of spatiallyindexed key and value vectors.

After encoding the support images in parallel, we reshape the resulting S × K × K × 2P feature maps to squeeze out the spatial dimensions, resulting in a SK 2 × 2P matrix.

EQUATION

p key = p[:, 0 : P ], p value = p[:, P : 2P ]

where CNN is a shallow convolutional network. We take the first P channels as the patch key vectors p key ∈ R SK 2 ×P and the second P channels as the patch value vectors p value ∈ R SK 2 ×P . Together these form a queryable memory for image generation.

To query this memory, we need to encode both the global context from the support set s as well as the pixels x <t generated so far. We can obtain these features simply by taking any layer of a PixelCNN conditioned on the support set:

EQUATION

)

where L is the desired layer of hidden unit activations within the PixelCNN network. In practice we use the middle layer.

To incorporate the patch attention features into the pixel predictions, we build a scoring function using q and p key . Following the design proposed by Bahdanau et al. (2014) , we compute a normalized matching score α tj between query pixel q t and supporting patch p key j as follows:

e tj = v T tanh(q t + p key j )

(5)

EQUATION

The resulting attention-gated context function can be written as:

f t (s, x <t ) = SK 2 j=1 α tj p value j (7)

which can be substituted into the objective in equation 1. In practice we combine the attention context features f t (s, x <t ) with global context features f (s) by channel-wise concatenation.

This attention mechanism can also be straightforwardly applied to the multiscale PixelCNN architecture of Reed et al. (2017) . In that model, pixel factors P (x t |x <t , f t (s, x <t )) are simply replaced by pixel group factors P (x g |x <g , f g (s, x <g )), where g indexes a set of pixels and < g indicates all pixels in previous pixel groups, including previously-generated lower resolutions.

We find that a few simple modifications to the above design can significantly improve performance. First, we can augment the supporting images with a channel encoding relative position within the image, normalized to [-1, 1]. One channel is added for x-position, another for y-position. When patch features are extracted, position information is thus encoded, which may help the network assemble the output image. Second, we add a 1-of-K channel for the supporting image label, where K is the number of supporting images. This provides patch encodings information about which global context they are extracted from, which may be useful e.g. when assembling patches from multiple views of an object.

3.2. FEW-SHOT LEARNING WITH META PIXELCNN:

As an alternative to explicit conditioning with attention, in this section we propose an implicitlyconditioned version using gradient descent. This is an instance of what Finn et al. (2017a) called model-agnostic meta learning, because it works in the same way regardless of the network architecture. The conditioning pathway (i.e. flow of information from supports s to the next pixel x t ) introduces no additional parameters. The objective to maximize is as follows:

L(x, s; θ) = log P (x; θ ), where θ = θ -α∇ θ L inner (s; θ)

A natural choice for the inner objective would be L inner (s; θ) = log P (s; θ). However, as shown in Finn et al. (2017b) and similar to the setup in Neu & Szepesvári (2012) , we actually have considerable flexibility here to make the inner and outer objectives different.

Any learnable function of s and θ could potentially learn to produce gradients that increase log P (x; θ ). In particular, this function does not need to compute log likelihood, and does not even need to respect the causal ordering of pixels implied by the chain rule factorization in equation 1.

Effectively, the model can learn to learn by maximum likelihood without likelihoods.

As input features for computing L inner (s, θ), we use the L-th layer of spatial features q = PixelCNN L (s) ∈ R H×W ×Z , where Z is the number of feature channels used in the PixelCNN. Note that this is the same network used to model P (x; θ). The features q are fed through a convolutional network producing a scalar, which is the (learned) inner loss. In practice, we used α = 0.1, and the encoder had three layers of stride-2 convolutions with 3 × 3 kernels, followed by elementwise squaring and a sum over all dimensions. Since these convolutional weights are part of θ, they are learned jointly with the generative model weights by maximizing equation 8.

4. EXPERIMENTS:

In this section we describe experiments on image flipping, Omniglot, and Stanford Online Products.

In all experiments, the support set encoder f (s) has the following structure: in parallel over support images, a 5 × 5 conv layer, followed by a sequence of 3 × 3 convolutions and max-pooling until the spatial dimension is 1. Finally, the support image encodings are concatenated and fed through two fully-connected layers to get the support set embedding.

4.1. IMAGENET FLIPPING:

As a diagnostic task, we consider the problem of image flipping as few-shot learning. The "support set" contains only one image and is simply the horizontally-flipped target image. A trivial algorithm exists for this problem, which of course is to simply copy pixel values directly from the support to the corresponding target location. We find that the Attention PixelCNN did indeed learn to solve the task, however, interestingly, the baseline conditional PixelCNN and Meta PixelCNN did not.

We trained the model on ImageNet (Deng et al., 2009) images resized to 48×48 for 30K steps using RMSProp with learning rate 1e -4 . The network was a 16-layer PixelCNN with 128-dimensional feature maps at each layer, with skip connections to a 256-dimensional penultimate layer before pixel prediction. The baseline PixelCNN is conditioned on the 128-dimensional encoding of the flipped image at each layer; f (s) = f (x ), where x is the mirror image of x. The Attention PixelCNN network is exactly the same for the first 8 layers, and the latter 8 layers are conditioned also on attention features f t (s, x <t ) = f t (x , x <t ) as described in section 3.1.

Figure 3 shows the qualitative results for several validation set images. We observe that the baseline model without attention completely fails to flip the image or even produce a similar image. With attention, the model learns to consistently apply the horizontal flip operation. However, it is not Quantitatively, we also observe a clear difference between the baseline and the attention model. The baseline achieves 2.64 nats/dim on the training set and 2.65 on the validation set. The attention model achieves 0.89 and 0.90 nats/dim, respectively. During sampling, Attention PixelCNN learns a simple copy operation in which the attention head proceeds in right-to-left raster order over the input, while the output is written in left-to-right raster order.

4.2. OMNIGLOT:

In this section we benchmark our model on Omniglot (Lake et al., 2013) , and analyze the learned behavior of the attention module. We trained the model on 26 × 26 binarized images and a 45 -5 split into training and testing character alphabets as in Bornschein et al. (2017) .

To avoid over-fitting, we used a very small network architecture. It had a total of 12 layers with 24 planes each, with skip connections to a penultimate layer with 32 planes. As before, the baseline model conditioned each pixel prediction on a single global vector computed from the support set. The attention model is the same for the first half (6 layers), and for the second half it also conditions on attention features.

The task is set up as follows: the network sees several images of a character from the same alphabet, and then tries to induce a density model of that character. We evaluate the likelihood on a held-out example image of that same character from the same alphabet. Meta PixelCNN also achieves state-of-the-art likelihoods, only outperformed by Attention Pixel-CNN (see Table 2 ). Naively combining attention and meta learning does not seem to help. However, there are likely more effective ways to combine attention and meta learning, such as varying the inner loss function or using multiple meta-gradient steps, which could be future work. We observe that as expected, the network learns to attend to corresponding regions of the support set when drawing each portion of the output image. Figure 4 compares results with and without attention. Here, the difference in likelihood clearly correlates with improvement in sample quality.

Number of support set examples:



4.3. STANFORD ONLINE PRODUCTS:

In this section we demonstrate results on natural images from online product listings in the Stanford Online Products Dataset (Song et al., 2016) . The data consists of sets of images showing the same product gathered from eBay product listings. There are 12 broad product categories. The training set has 11, 318 distinct objects and the testing set has 11, 316 objects.

The task is, given a set of 3 images of a single object, induce a density model over images of that object. This is a very challenging problem because the target image camera is arbitrary and unknown, and the background may also change dramatically. Some products are shown cleanly with a white background, and others are shown in a usage context. Some views show the entire product, and others zoom in on a small region.

For this dataset, we found it important to use a multiscale architecture as in Reed et al. (2017) . 

:



Figures and Tables:

Figure 1: Sampling from Attention PixelCNN. Support images are overlaid in red to indicate the attention weights. The support sets can be viewed as small training sets, illustrating the connection between sample-conditional density estimation and learning to learn distributions.

Figure 2: The PixelCNN attention mechanism.

Figure 3: Horizontally flipping ImageNet images. The network using attention learns to mirror, while the network without attention does not.

Figure 4: Typical Omniglot samples from PixelCNN, Attention PixelCNN, and Meta PixelCNN.

Figure1shows several key frames of the attention model sampling Omniglot. Within each column, the left part shows the 4 support set images. The red overlay indicates the attention head read weights. The red attention pixel is shown over the center of the corresponding patch to which it attends. The right part shows the progress of sampling the image, which proceeds in raster order. We observe that as expected, the network learns to attend to corresponding regions of the support set when drawing each portion of the output image. Figure4compares results with and without attention. Here, the difference in likelihood clearly correlates with improvement in sample quality.

Figure 6: Flipping 24×24 images, comparing global-conditional, attention-conditional and gradientconditional (i.e. MAML) PixelCNN.

Figure 7: Comparison to ConvDRAW in 4-shot learning.

Omniglot test(train) few-shot density estimation NLL in nats/dim.Bornschein et al. (2017) refers to Variational Memory Addressing andGregor et al. (2016) to ConvDRAW.All PixelCNN variants achieve state-of-the-art likelihood results (see table1). Attention PixelCNN significantly outperforms the other methods, including PixelCNN without attention, across 1, 2, 4 and 8-shot learning. PixelCNN and Attention PixelCNN models are also fast to train: 10K iterations with batch size 32 took under an hour using NVidia Tesla K80 GPUs.We also report new results of training a ConvDRAWGregor et al. (2016) on this task. While the likelihoods are significantly worse than those of Attention PixelCNN, they are otherwise state-ofthe-art, and qualitatively the samples look as good. We include ConvDRAW samples on Omniglot for comparison in the appendix section 6.2.

Table content: <table><tr><td>Model</td><td>1</td><td>2</td><td>4</td><td>8</td></tr><tr><td>Bornschein et al. (2017)</td><td>0.128(--)</td><td>0.123(--)</td><td>0.117(--)</td><td>--(--)</td></tr><tr><td>Gregor et al. (2016)</td><td colspan="4">0.079(0.063) 0.076(0.060) 0.076(0.060) 0.076(0.057)</td></tr><tr><td>Conditional PixelCNN</td><td colspan="4">0.077(0.070) 0.077(0.068) 0.077(0.067) 0.076(0.065)</td></tr><tr><td>Attention PixelCNN</td><td colspan="4">0.071(0.066) 0.068(0.064) 0.066(0.062) 0.064(0.060)</td></tr></table>

Omniglot NLL in nats/pixel with four support examples. Attention Meta PixelCNN is a model combining attention with gradient-based weight updates for few-shot learning.

Table content: <table/>

Peer review (iclr2018):
 This paper focuses on the density estimation when the amount of data available for training is low. The main idea is that a meta-learning model must be learnt, which learns to generate novel density distributions by learn to adapt a basic model on few new samples. The paper presents two independent method. The first method is effectively a PixelCNN combined with an attention module. Specifically, the support set is convolved to generate two sets of feature maps, the so called "key" and the "value" feature maps. The key feature map is used from the model to compute the attention in particular regions in the support images to generate the pixels for the new "target" image. The value feature maps are used to copmpute the local encoding, which is used to generate the respective pixels for the new target image, taking into account also the attention values. The second method is simpler, and very similar to fine-tuning the basis network on the few new samples provided during training. Despite some interesting elements, the paper has problems. First, the novelty is rather limited. The first method seems to be slightly more novel, although it is unclear whether the contribution by combining different models is significant. The second method is too similar to fine-tuning: although the authors claim that .mathcal{L}_inner can be any function that minimizes the total loss .mathcal{L}, in the end it is clear that the log-likelihood is used. How is this approach (much) different from standard fine-tuning, since the quantity P(x; .theta*) is anyways unknown and cannot be "trained" to be maximized. Besides the limited novelty, the submission leaves several parts unclear. First, why are the convolutional features of the support set in the first methods divided into "key" and "value" feature maps as in p_key=p[:, 0:P], p_value=p[:, P:2*P]? Is this division arbitrary, or is there a more basic reason? Also, is there any different between key and value? Why not use the same feature map for computing the attention and computing eq (7)? Also, in the first model it is suggested that an additional feature can be having a 1-of-K channel for the supporting image label: the reason is that you might have multiple views of objects, and knowing which view contributes to the attention can help learning the density. However, this assumes that the views are ordered, namely that the recording stage has a very particular format. Isn*t this a bit unrealistic, given the proposed setup anyways? Regarding the second method, it is not clear why leaving this room for flexibility (by allowing L_inner to be any function) to the model is a good idea. Isn*t this effectively opening the doors to massive overfitting? Besides, isn*t the statement that the function .mathcal{L}_inner void? At the end of the day one can also claim the same for gradient descent: you don*t need to have the true gradients of the true loss, as long as the objective function obtains gradually lower and lower values? Last, it is unclear what is the connection between the first and the second model. Are these two independent models that solve the same problem? Or are they connected? Regarding the evaluation of the models, the nature of the task makes the evaluation hard: for real data like images one cannot know the true distribution of particular support examples. Surrogate tasks are explored, first image flipping, then likelihood estimation of Omniglot characters, then image generation. Image flipping does not sound a very relevant task to density estimation, given that the task is deterministic. Perhaps, what would make more sense would be to generate a new image given that the support set has images of a particular orientation, meaning that the model must learn how to learn densities from arbitrary rotations. Regarding Omniglot character generation, the surrogate task of computing likelihood of known samples gives a bit better, however, this is to be expected when combining a model without attention, with an attention module. All in all, the paper has some interesting ideas. I encourage the authors to work more on their submission and think of a better evaluation and resubmit.