Year: 2021

Abstract:
This work addresses meta-learning (ML) by considering deep networks with stochastic local winner-takes-all (LWTA) activations. This type of network units result in sparse representations from each model layer, as the units are organized into blocks where only one unit generates a non-zero output. The main operating principle of the introduced units lies on stochastic principles, as the network performs posterior sampling over competing units to select the winner. Therefore, the proposed networks are explicitly designed to extract input data representations of sparse stochastic nature, as opposed to the currently standard deterministic representation paradigm. We posit that these modeling principles, inspired from Bayesian statistics, yield representations of stronger generalization capacity; this is of immense importance in the case of ML, which is the focus of this work. As we experimentally show, our approach produces state-of-the-art predictive accuracy on standard few-shot image classification benchmarks. This improvement comes with an immensely reduced network sized required for achieving this accuracy; this amounts to a parameter reduction by one order of magnitude on average compared to the current state-of-the-art.

heading: 1 INTRODUCTION
text: When we train machine learning models on problems with limited amounts of training data, we cannot usually get good predictive performance (Lai, 2019; Sculley et al., 2015). This comes in contrast to the human ability to quickly derive information from a range of different tasks, and then adapt to a new task with limited available new examples (Kühl et al., 2020; Castro et al., 2008). In essence, this capability of the mind of learning how to learn (Black et al., 2006) has inspired researchers to investigate the concept of Meta-Learning (ML) (Lake et al., 2017; Vilalta & Drissi, 2002; Wang et al., 2016; Zoph & Le, 2017; Flennerhag et al., 2020; Lake et al., 2015; HutsebautBuysse et al., 2019).
There is a large variety of deep learning methods for ML (Finn et al., 2019; Baik et al., 2020; Andrychowicz et al., 2016). Specifically, Finn et al. (2017) presented the Model-Agnostic MetaLearning (MAML) algorithm that enables tuning the parameters of a trained network to quickly learn a new task with only a few gradient updates. To get away with the entailed second-order computations, which are expensive, several researchers proposed appropriate first-order approximations for MAML; such works are the First-Order MAML (FOMAML) of Finn et al. (2017) and the Reptile algorithm of Nichol et al. (2018). Recently, several researchers have also considered Bayesian inference-driven methods for deep learning ML, extending upon older works built for conventional machine learning approaches (Yoon et al., 2018; Finn et al., 2018; Ravi & Beatson, 2019; Patacchiola et al., 2020; Zou & Lu, 2020; Grant et al., 2018; Chen et al., 2020).
This work proposes a different regard toward improving generalization capacity for deep networks in the context of ML. Specifically, our proposed approach relies on the following main concepts:
• The concept of sparse learned representations. For the first time in the literature of deep network-driven ML, we employ a mechanism that inherently learns to extract sparse data representations. This consists in replacing standard unit nonlinearities (e.g., ReLU)
with a unit competition mechanism. Specifically, (linear) units are organized into blocks. Presented with some input, the units within a block engage in a competition process with only one winner. The outputs of all units except for the winner are zeroed out; the output of the winner retains its computed value (local winner takes-all, LWTA, architecture).
• The concept of stochastic representations. We establish a stochastic formulation for the previously described competition process. Specifically, we postulate that, within a block of competing units, winner is selected via sampling from an appropriate Categorical posterior. The corresponding winning probability of each unit is proportional to its linear computation (thus depending on the layer input). Via this competition process, we yield stochastic representations from network layers, that is representations that may change each time we present to the network layer exactly the same input.
Based on the results from existing approaches, we posit that the proposed treatment of the ML problem, which combines learned representation sparsity and stochasticity, will be extremely beneficial to the deep learning community. We dub our approach Stochastic LWTA for ML (StochLWTA-ML).
We perform a variational Bayes treatment of the proposed model. We opt for a full Bayesian treatment, by also handling network weights as latent variables. That is, we elect to impose an appropriate prior over the network weights and fit approximate (variational) posteriors. We evaluate our approach on a number of standard benchmarks in the field, namely Omniglot (Lake et al., 2017), Mini-Imagenet (Vinyals et al., 2016) and CIFAR-100 (Krizhevsky, 2009). We show that our approach offers a variety of advantages over the current state-of-the-art methods, namely: (i) incurring reduced predictive error rate compared to the currently state-of-the-art methods in the field; (ii) obtaining this performance with networks that comprise one order of magnitude less trainable parameters, and therefore give rise to better computational efficiency and imposed memory footprint.
The remainder of this paper is organized as follows: In Section 2, we briefly review related work. Section 3 introduces our approach and provides the related training and prediction algorithms. In Section 4, we perform a thorough experimental evaluation of StochLWTA-ML, and compare our findings to the current state-of-the-art. In the final Section 5, we end up with the conclusions of our work, and suggest lines of further research.

heading: 2 RELATED WORK

heading: 2.1 LWTA LAYERS IN DEEP LEARNING
text: LWTA layers are not new in the field of deep learning; see, e.g., Srivastava et al. (2013). Although not much work has been pursued along these lines, the recent works of Panousis et al. (2019; 2021) and Voskou et al. (2021) have spurred some fresh interest in the field. These works have presented alternative implementations of the basic concepts of LWTA units in the context of diverse deep network architectures. Specifically, Panousis et al. (2019) propose a stochastic LWTA formulation which is founded upon the Indian Buffet process (IBP) prior, borrowed from nonparametric statistics; they use this architecture to effect data-driven network compression. In their follow-up paper (Panousis et al., 2021), they exploit the same technique to train adversarially-robust deep networks. On the other hand, Voskou et al. (2021) consider a different incarnation of stochastic LWTA architectures, which relies on sampling the winner from a Categorical posterior, driven from the layer input. This layer architecture is used to replace dense ReLU layers in Transformer networks; it is then shown to yield important benefits in a Sign-Language Translation benchmark.
This paper is different from the previous works in various ways: (i) stochasticity does not stem from utilization of the IBP; we rather adopt an approach similar to Voskou et al. (2021); (ii) we do not use the proposed architecture as a replacement for a specific type of layer in a greater network architecture (Transformer) that remains largely unchanged; instead, we build completely new networks using these layers; (iii) we perform a full Bayesian treatment, by treating network weights as random variables; we do not employ this construction as a means of compressing the weights at prediction time, contrary to Panousis et al. (2019), Voskou et al. (2021); instead, we perform weight sampling at prediction time as a means of improving accuracy; and (iv) for the first time, we examine how these principles perform in the context of deep network-driven ML. Note that, apart from LWTA architectures, other data-driven sparsity models have also been proposed recently, e.g. Lee
et al. (2018), Kessler et al. (2021). However, none of these have been developed or evaluated in the context of an ML setting.

heading: 2.2 MODEL-AGNOSTIC META-LEARNING
text: ML (Schmidhuber, 1987), also referred to as learning to learn (Thrun & Pratt, 1998), has the goal of obtaining machine learning models that can learn new tasks with limited availability of data, and via only a few gradient steps, by capitalizing upon information stemming from previously learned tasks. In this context, Finn et al. (2017) suggested a model-agnostic algorithm for ML, that can be applied to any model trained via gradient descent. The introduced MAML algorithm initializes model parameters in a way that can be quickly adapted to several types of new tasks.
Let us consider a model with parameters θ and a parametric form fθ. When the model is adapting to an unseen task Ti, sampled from the distribution over tasks P (T ), MAML initially runs few steps of inner-loop gradient descent that yields the task-specific parameter set
θ′i = θ − α∇θLTi(fθ) (1)
where α is the step size hyperparameter and LTi denotes the loss on the task Ti.
Subsequently, training proceeds to optimize the function fθ′i with respect to the model parameters θ. Assuming that the batch of tasks has size M , we can define the targeted meta-objective as:
Lmeta(θ) = M∑ i=1 LTi(fθ′i) = M∑ i=1 LTi(fθ−α∇θLTi (fθ)). (2)
Optimization of this meta-objective over θ yields the outer-loop update:
θ ← θ − β∇θ M∑ i=1 LTi(fθ′i) (3)
where β stands for the outer-loop learning rate.
Finally, as MAML involves expensive computations stemming from the second-order updates of Eqs. (2) and (3), Finn et al. (2017) developed a first-order approximation that reduces the outer-loop update (3) to:
θ ← θ − β M∑ i=1 ∇θ′iLTi(fθ′i) (4)
In other words, FOMAML computes the gradients with respect to the updated parameter values θ′i, but omits the gradients of θ′i with respect to θ.
In a different fashion, the Reptile algorithm of Nichol et al. (2018) disposes the costly secondorder updates of MAML by: (i) applying some inner-loop gradient descent steps using (1); and (ii) suggesting a new outer-loop that simply subtracts the parameters, θ, from the updates θ′i (instead of computing derivatives):
θ ← θ + β(θ′i − θ). (5)

heading: 3 PROPOSED APPROACH

heading: 3.1 ARCHITECTURE
text: Let us denote as x ∈ RI an input vector presented to a dense ReLU layer of a conventional deep neural network, with corresponding weights matrix W ∈ RI×O. The output of the layer is the vector y ∈ RO and is fed to the subsequent layer. In our approach, a ReLU unit is replaced by J competing linear units, organized in one block; in the following, we denote with R the number of blocks in a layer. The input x is now presented to each block through weights that are organized into a three-dimensional matrix W ∈ RI×R×J . Then, the j-th competing unit within r-th block computes the sum ∑I i=1(wi,r,j) · xi. Competition means that, of the J units in the block, one unit (the ”winner”) will present its linear computation to the next layer; the rest will present zero
values. Traditionally in the literature, the winner unit is selected to be the unit with greatest linear computation. Recently, stochastic competition principles have been considered, e.g. Panousis et al. (2019; 2021), Voskou et al. (2021).
Let us denote as y ∈ RR·J the output of an LWTA layer; this is composed of R subvectors yr ∈ RJ and is sparse, since all units except for one, in each block, yield zero values. Let us introduce the discrete latent indicator vector ξ ∈ one hot(J)R to denote the winner units in the R blocks that constitute a considered stochastic LWTA layer. This vector comprises R component subvectors, where each component entails one non-zero value at the index position that corresponds to the winner unit of the respective LWTA block. On this basis, the output y of the stochastic LWTA layer’s (r · j)-th component yr,j is defined as:
yr,j = ξr,j I∑ i=1 (wi,r,j) · xi ∈ R (6)
where we denote as ξr,j the j-th component of ξr, and ξr ∈ one hot(J) holds the r-th subvector of ξ.
In Eq. (6), we postulate that the latent winner indicator variables are drawn from a Categorical distribution which is proportional to the intermediate linear computation that each unit performs. Therefore, the stronger the linearity the higher the chance of the unit winning the stochastic competition within its block. In detail, we postulate that, a posteriori, the winner distributions yield:
q(ξr) = Categorical ( ξr|softmax(
I∑ i=1 [wi,r,j ] J j=1 · xi)
) (7)
where [wi,r,j ]Jj=1 denotes the vector concatenation of the set {wi,r,j}Jj=1. A graphical illustration of the proposed stochastic architecture is provided in Fig. 1.
As a network composed of such (StochLWTA) layers entails latent variables ξ, we need to perform a Bayesian network treatment to perform effective parameter training. We opt for a (approximate) stochastic gradient variational Bayes treatment (Kingma & Welling, 2014), for scalability purposes. This means that the used objective function takes the form of an evidence lower-bound (ELBO) objective, as we describe next. In our work, we take one step further: we also elect to infer a posterior density over the network weightsW , as opposed to obtaining point-estimates. This results in a second source of stochasticity for our approach, which may further increase its generalization capacity under uncertain conditions arising from limited training data availability. Note that the use of these posteriors is totally different from Panousis et al. (2019) and Voskou et al. (2021): therein, posterior variance is used for compressing posterior mean bit-precision; then, predictions are performed using only the compressed posterior mean. Instead, in our work we sample multiple times from the trained weight posterior and perform model averaging (in a Bayesian sense), as a means of increasing generalization capacity.
We postulate: q(vec(W )) = N(vec(W )|µ,diag(σ2)) (8)
where ψ ≜ {µ,σ2} are the means and variances of the Gaussian weight posteriors, respectively.

heading: 3.2 A MODEL-AGNOSTIC ML ALGORITHM
text: Let us define an ML problem where we are given a training dataset D of tasks, T , that are governed by a distribution P (T ). We consider an N -way, K-shot learning setting, where each task contains K labelled examples for each of N available classes.
Our approach entails parameter sets θ which coincide with the hyperparameter sets ψ = {µ,σ2} of the weights W ∈ RI×R×J ; these sets ψ = {µ,σ2} are the target of our MAML-type training algorithm.
Therefore, to perform model training, we have to first initialize the trainable parameters µ and σ2 across layers. To this end, one can appropriately exploit popular initialization schemes, such as
Glorot Uniform (Glorot & Bengio, 2010). Then, our approach entails an inner-outer loop scheme, in a vein similar to existing approaches, but with some crucial differences:
1. The stochastic nature of the postulated weights,W , results in the updates taking place over the posterior means, µ and variances, σ2.
2. The stochastic nature of both the inferred representations and the network weights themselves implies that proper training must rely on optimization of the ELBO function of the network. Let us consider an inner-loop dealing with task Ti ∼ P (T ), with data Di = (Xi, Yi) ⊂ D. Let CE(Yi, fψ(Xi; ξ̂, Ŵ )) be the categorical cross-entropy between the data labels Yi and the class probabilities fψ(Xi; ξ̂, Ŵ ) generated by the penultimate Softmax layer. Then, we yield:
LTi(ψ) = −CE(Yi, fψ(Xi; ξ̂, Ŵ ))−KL[ q(ξ) || p(ξ) ]−KL[ q(W ) || p(W ) ] (9)
for task Ti with dataset Di which is dealt with on the i-th training iteration.
Here, for simplicity and without harming generality, we consider that the weights prior p(vec(W )) is a Gaussian distribution N(0, I) and the latent variables prior p(ξ) is a symmetric Categorical distribution Categorical(1/J). In addition, in our notation we stress that the output fψ(Xi; ξ̂, Ŵ ) depends on the winner selection process, which is stochastic, and the outcomes of sampling the network weights. Specifically, in our work we perform Monte-Carlo (MC) sampling using one reparameterized sample of the corresponding latent variables.
Let ξ̄ be the unnormalized probabilities of the Categorical distribution q(ξ) (Eq. (7)). The sampled instances of ξ, ξ̂r,j , are expressed as (Maddison et al., 2017):
ξ̂r,j = Softmax((log ξ̄r,j + gr,j)/τ), ∀r = 1, . . . , R, j = 1, . . . , J (10) where gr,j = − log(− logUr,j), Ur,j ∼ Uniform(0, 1), and τ ∈ (0,∞) is a temperature factor that controls how closely the Categorical distribution is approximated by this continuous relaxation.
Similarly, the Gaussian weights yield: ŵt,r,j = µt,r,j + σt,r,j ϵ̂, and ϵ̂ ∼ N(0, 1). On this basis, the KL divergences in Eq. (9) become:
KL[ q(ξr,j || p(ξr,j) ] = Eq(ξr,j)[log q(ξr,j)− log p(ξr,j)]
≈ log q(ξ̂r,j)− log p(ξ̂r,j), ∀r, j (11)
and KL[ q(wt,r,j) || p(wt,r,j) ] = Eq(wt,r,j)[log q(wt,r,j)− log p(wt,r,j)] ≈ log q(ŵt,r,j)− log p(ŵt,r,j), ∀t = 1, . . . , I, r, j
(12)
Hence, the ELBO becomes: LTi(ψ) = −CE(Yi, fψ(Xi; ξ̂, Ŵ ))− ∑ r,j ( log q(ξ̂r,j)− log p(ξ̂r,j) ) − ∑ t,r,j (log q(ŵt,r,j)− log p(ŵt,r,j))
(13)
Therefore, we establish a MAML-type algorithm, where: (i) the used networks comprise blocks of stochastic LWTA units visually depicted in Fig. 1; (ii) the trainable parameters are the means µ and variances σ2 of the synaptic weights; and (iii) the objective function of the inner-loop process is given in Eq. (13).
We summarize our training algorithm in Alg. 1.
Algorithm 1: Model training with StochLWTA-ML Require: P (T ): distribution over tasks Initialize ψ := {µ,σ2} Define outer-step size β and inner learning rate α for i = 1,2, . . . do
Inner training loop: Sample task Ti ∼ P (T ), where Ti contains data Di = (Xi, Yi) ⊂ D Compute LTi(ψ) using Eq. (13) Compute adapted parameters with SGD: ψ′i = ψ − α∇ψLTi(fψ) Outer training loop: Derive ψ ← ψ + β(ψ′i −ψ)
end

heading: 3.3 PREDICTION ALGORITHM
text: At prediction time, we draw a set of B samples of the Gaussian connection weights from the trained posteriors N (µ,σ2). Then, we select the winning units in each block of the network by similarly sampling from the posteriors q(ξ). This results in a set of B output logits of the network, which we average to obtain the final predictive outcome:
fψ(Xi; ξ̃, W̃ ) ≈ 1
B B∑ s=1 fψ(Xi; ξ̃s, W̃s) (14)
where ξ̃s and W̃s are sampled directly from the posteriors q(ξ) and q(W ), respectively.
This concludes the formulation of the proposed model-agnostic ML approach.
Algorithm 2: Prediction with StochLWTA-ML Require: Learned parameters ψ = {µ,σ2}, input data X ′
SampleW ∼ q(vec(W )) = N(vec(W )|µ, diag(σ2)) Sample ξ ∼ q(ξ) defined in Eq. (7), forw = ψ and (xi = x′i) ∈ X ′
Compute output logits, given the sampled values ξ and ψ Repeat B times Use Eq. (14) to average over the resulting set of B logits and derive the final prediction.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUP
text: We evaluate StochLWTA-ML on Omniglot, Mini-Imagenet and CIFAR-100 which are popular few-shot image classification datasets, and compare its performance to state-of-the-art prior results. After thorough exploration on the number of LWTA layers as well as the number of blocks for each layer and the competing units per block, we end up with using networks comprising 2 layers with 16 blocks and 2 competing units per block on the former layer, and 8 blocks with 2 units per block on the latter. The penultimate network layer is a Softmax. Weight mean initialization, as well as point-estimate initialization for our competitors, is performed via Glorot Uniform. Weight log-variance initialization is performed via Glorot Normal, by sampling from N(0.0005, 0.01). The Gumbel-Softmax relaxation temperature is set to τ = 0.67.
In the inner-loop updates, we use the Stochastic Gradient Descent (SGD) (Robbins, 2007) optimizer with a learning rate of 0.003. For the outer-loop, we use SGD with a linear annealed outer step size to 0, and an initial value of 0.25. Additionally, all the experiments were ran with task batch size of 50 for both training and testing mode. Prediction is carried out averaging over B = 4 output logits.
The results presented in Sections 4.2 and 4.3 stand for the average performance over three runs with different random seeds. Note that each experiment consists of different number of iterations for training, depending on its convergence speed. The code was implemented in Tensorflow (Abadi et al., 2016).

heading: 4.2 RESULTS
text: In Table 1, we show how StochLWTA-ML performs on Omniglot 20-way, Mini-Imagenet 5-way and CIFAR100 5-way few-shot settings. We compare our findings to state-of-the-art ML algorithms such as LLAMA (Grant et al., 2018) and PLATIPUS (Finn et al., 2018) as reported in Gordon et al. (2018), Amortized Bayesian Meta-Learning (ABML) (Ravi & Beatson, 2019), MAML, FOMAML (Finn et al., 2017), Reptile (Nichol et al., 2018) and others. Using the original architectures with the same hyperparameters and data preprocessing as in Finn et al. (2017), we have also locally reproduced ABML, BMAML (with 5 particles), PLATIPUS, MAML, FOMAML and Reptile (dubbed ”local” in Table 1). For completeness sake, we also compare our findings to other state-of-the-art ML models as reported in Finn et al. (2017), including Matching Nets (Santoro et al., 2016) and LSTM Meta-Learner (Ravi & Larochelle, 2017). As we observe, our method outperforms the existing state-of-the-art in both the 1-shot and 5-shot settings.

heading: 4.3 ABLATION STUDY

heading: 4.3.1 DOES STOCHASTIC COMPETITION CONTRIBUTE TO CLASSIFICATION ACCURACY?
text: To check whether the accuracy improvements stem from the LWTA-induced sparsity or the proposed stochastic competition concept, we evaluate both our approach as well as MAML, FOMAML, ABML, BMAML and PLATIPUS, considering both ”deterministic LWTA” and ”stochastic LWTA” setups; deterministic LWTA networks have been adopted from Srivastava et al. (2013). As we see in Table 2, replacing ReLU with deterministic LWTA yields negligible differences. On the other hand, stochastic LWTA units yield a clear improvement in all cases. This improvement becomes even more important in the case of our approach, where we sample from stochastic weights.

heading: 4.3.2 EFFECT OF BLOCK SIZE J
text: As it is presented in Table 3, increasing the number of competing units per block to J = 4 or J = 8 does not notably improve the results of our approach. On the contrary, it increases the number of trained parameters, thus leading to higher network computational complexity. This corroborates our initial choice of using J = 2 competing units per block in our approach.

heading: 4.3.3 HOW DOES THE NUMBER OF SAMPLES AT PREDICTION TIME AFFECT ACCURACY?
text: We scrutinize the effect of the number of drawn samples, B, on StochLWTA-ML’s predictive accuracy. To this end, we repeat our experiments using B = 10 logits sets. In Table 4, we provide the comparative outcomes concerning the two sample size configurations of B = 4 and B = 10. As we observe, an increase in prediction sample size, B, yields a slight accuracy increase. However, the aforementioned increase in sample size imposes some computational overhead, which we elaborate upon in the following Section. We argue that this overhead might not be worth it for the slight performance increase reported in Table 4. More information on the effect of sample size in our approach’s predictive performance are provided in the Supplementary.

heading: 4.3.4 IS THERE A COMPUTATIONAL TIME TRADE-OFF FOR THE INCREASED ACCURACY?
text: It is also important to investigate whether our approach represents a trade-off between accuracy and computational time compared to our competitors. To facilitate this investigation, in Table 5 we provide training iteration wall-clock times for our approach and the existing locally reproduced state-of-the-art, as well as the total number of iterations each model needs to achieve the reported performance of Table 1. It appears that our methodology takes 77% less training time than the less efficient algorithms ABML, BMAML, PLATIPUS, and is comparable to other approaches. This happens because our approach yields the reported state-of-the-art performance by employing a network architecture (that is, number of LWTA layers, as well as number of blocks and block size on each layer) that result in a total number of trainable parameters that is one order of magnitude less on average than the best performing baseline methods. This can be seen in the last three columns of Table
5 (dubbed DA, DB and DC for Omniglot, Mini-Imagenet and CIFAR-100 respectively). In addition, training for our approach converges fast.
The situation changes when it comes to prediction: our approach imposes a slight computational time overhead compared to MAML, FOMAML and Reptile, but still much less than the time-consuming PLATIPUS. BMAML and ABML. This is a rather negligible increase when we are dealing with a low number of drawn samples, B, but increases as we increase B. The provided results suggest that a selection of B = 4 represents a favorable accuracy/prediction wall-clock time for our approach.
Finally, we provide an example of how training for our approach converges, and how this compares to the alternatives. We illustrate our outcomes on the Omniglot 20-way 1-shot benchmark; similar outcomes have been observed in the rest of the considered datasets. Fig. 2(a) compares StochLWTA-ML with prior traditional ML methods: MAML, FOMAML and Reptile. It becomes apparent that our approach converges equally fast to these competitors. Further, Fig. 2(b) compares StochLWTA-ML with the probabilistic ML models ABML, BMAML, PLATIPUS. Since, as we see in the ablation study of Section 4.3.4, these methods are quite timeconsuming and less efficient regarding to memory consumption, StochLWTA-ML gives rise to an easier time training MAML based probabilistic model.

heading: 5 CONCLUSION
text: In this paper, we proposed a sparse and stochastic network paradigm for ML, with novel network design principles compared to currently used model-agnostic ML models. We introduced stochastic LWTA activations in the context of a variational Bayesian treatment that gave rise to a doubly-stochastic ML framework, bearing the promise of stronger generalization capacity. We evaluated our approach using standard benchmarks in the field, and showed that it outperformed the state-of-the-art in terms of both predictive accuracy and computational costs. The results have provided strong empirical evidence supporting our claims. In the future, we plan to study the effect of StochLWTA-ML in areas related to ML, such as Continual Learning (Javed & White, 2019) and Reinforcement Learning (Zhu et al., 2020).

heading: A FURTHER DETAILS ON DATASETS EXPERIMENTAL FORMATION
text: Omniglot is a dataset of 1623 characters from different alphabets, containing 20 examples per character scaled down to 28x28 grayscale pixels. The ratio between training and testings sets is 3:2, so after shuffling the character classes we randomly choose the first 974 classes for training and the remaining are left for testing. As for the Mini-Imagenet dataset, it has color images of size 84x84 and contains 100 classes with 600 examples from the ImageNet dataset. We randomly choose 45000 examples for the training phase and the rest constitute the testing population. The CIFAR-100 dataset consists of color images of size 32x32 and contains 100 classes with 600 images per class. We randomly choose 500 images per class for training and the rest 100 images per class constitute the testing population.

heading: B ADDITIONAL EXPERIMENTS
text: In Table B1 we provide the results of the ablation study of Section 4.3.1 on the Omniglot 20-way dataset. The full-fledged StochLWTA-ML approach yields again better predictive performance compared to the alternative variants of ML algorithms.
Table B1: Omniglot 20-way Few-Shot ablation study (% accuracy)
Algorithm Network type 1-shot 5-shot MAML (local) deterministic LWTAstochastic LWTA 95.52 95.91 98.15 98.78 FOMAML (local) deterministic LWTAstochastic LWTA 95.01 95.80 98.18 98.41 ABML (local) deterministic LWTAstochastic LWTA 90.30 91.21 93.64 93.91 BMAML (local) deterministic LWTAstochastic LWTA 96.96 97.11 98.21 98.30 PLATIPUS (local) deterministic LWTAstochastic LWTA 94.48 95.13 98.31 98.56 StochLWTA-ML deterministic LWTAstochastic LWTA 96.95 97.79 98.63 98.97

heading: C FEW-SHOT CLASSIFICATION NETWORK ARCHITECTURES
text: For the local replicates of prior ML algorithms in the experiments of our work, we follow the same architecture for the deep neural network as the one used by Vinyals et al. (2016). For Omniglot, the network is composed of 4 convolutional layers with 64 filters, 3 x 3 convolutions and 2 x 2 strides, followed by a Batch Normalization layer (Ioffe & Szegedy, 2015) and the final values of each layer are processed by an activation function. For both Mini-Imagenet and CIFAR-100, we use 4 convolutional layers with 32 filters to reduce overfitting like Ravi & Larochelle (2017), 3 x 3 convolutions followed by Batch Normalization layer and 2 × 2 max-pooling layer with the values of each layer finally passed again through an activation block. The activation functions used for experiments of main paper’s Tables 1 and 5 is ReLU, and LWTA for experiments of main paper’s Table 2 and Supplementary’s Table B1.

heading: D HOW DOES THE TASK BATCH SIZE AFFECT STOCHLWTA-ML’S PERFORMANCE?
text: In Fig. 3 and 4, we illustrate the performance of our model on the Mini-Imagenet 5-way 1-shot setting with different task batch sizes. As we see, our model performs optimally with task batch size of 50 for both training and testing phase. Using this value for batch size, we noticed that the classification accuracy as well as training time per iteration were optimal. This outcome was also observed in the rest of the considered datasets’ settings.

heading: E HOW DOES THE SAMPLE SIZE B AT PREDICTION TIME AFFECT STOCHLWTA-ML’S ACCURACY?
text: As we observe in Fig. 5, an increase in sample size, B, does not always yield an accuracy increase. We finally choose B = 4, since at that point our model achieves its best predictive performance at most of the experiments. In this figure, we illustrate our outcomes on Omniglot 20-way 1-shot, Mini-Imagenet 5-way 1-shot and CIFAR100 5-way 5-shot settings. Our approach has shown similar behaviour to the rest of experimental settings, leading us to the same choice for the number of sample size B.

heading: F WHAT PARAMETERS DO WE COUNT FOR THE OUTCOMES OF MAIN PAPER’S TABLE 5?
text: The included parameters of each baseline for the outcomes of Table 5 are:
• PLATIPUS: Θ = {µθ,σ2θ ,vθ,γp,γq} • BMAML: Θ = {θm}5m=1, for using 5 particles • ABML: θ = {µθ,σ2θ} • MAML: θ = {µθ} • FOMAML: θ = {µθ} • Reptile: θ = {µθ} • StochLWTA-ML: θ = {µθ,σ2θ}

Review summaries:
This paper proposes a stochastic local winner-takes-all (SLWTA) approach to learn data-driven (stochastic) sparsity. This is motivated by the limited availability of training data, especiially in few-shot meta learning scenarios. The authors propose a meta-learning algorithm for their SLWTA approach. As this is a probabilistic approach, one can draw from the approximate posterior at inference to obtained the posterior predictive distribution. The authors compare their method against MAML, FOMAML and Reptile on the Omniglot and Mini-Imagenet displaying better performance than the origiinal MAML algorithm.

The paper proposes to use neural networks with so-called "stochastic local winner-takes-all (LWTA)" activations in the place of standard ReLU activations. These stochastic LWTA activations result in a model with sparse representations. The sparse gating is treated as a random variable. The authors also treat the weights of the neural network as random variables. Both sets of random variables are learnt using variational inference.

In this paper, the authors proposed to replace the classical activation units in MAML with stochastic local winner-takes-all (LWTA) units. The authors argued the biologically motivated LWTA units would lead to better performance in the few-shot setting where the support set in the target task is small, in which case the embedding representation of the target task would be noisy. The authors framed their proposal in the Bayesian setting, and proposed algorithms to estimate the distribution parameters.

This paper proposed a novel meta-learning method by replacing standard nonlinearity functions in MAML with stochastic local winner-takes-all (LWTA) activations. The authors claim that such design results in sparse representations and benefit meta-learning. This method demonstrated superior performance over MAML and FOMAML over standard benchmarks such as Omniglot and Mini-ImageNet.