Stochastic Deep Networks with Linear Competing Units for Model-Agnostic Meta-Learning
Authors: Konstantinos Ι. Kalais
Sotirios Chatzis
Venue: ICLR 2022 Submitted (2022)
Keywords: Stochastic Deep Networks
LWTA
Meta-Learning

Abstract: This work addresses meta-learning (ML) by considering deep networks with stochastic local winner-takes-all (LWTA) activations. This type of network units result in sparse representations from each model layer, as the units are organized into blocks where only one unit generates a non-zero output. The main operating principle of the introduced units lies on stochastic arguments, as the network performs posterior sampling over competing units to select the winner. Therefore, the proposed networks are explicitly designed to extract input data representations of sparse stochastic nature, as opposed to the currently standard deterministic representation paradigm. We posit that these modeling arguments, inspired from Bayesian statistics, allow for more robust modeling when uncertainty is high due to the limited availability of task-related training data; this is exactly the case with ML, which is the focus of this work. At training time, we rely on the reparameterization trick for Discrete distributions to perform reliable training via Monte-Carlo sampling. At inference time, we rely on Bayesian Model Averaging, which effectively averages over a number of sampled representations. As we experimentally show, our approach produces state-of-the-art predictive accuracy on standard few-shot image classification benchmarks; this is achieved without compromising computational efficiency.

Under review as a conference paper at ICLR 2022
STOCHASTIC DEEP NETWORKS WITH LINEAR COM- PETING UNITS FOR MODEL-AGNOSTIC META- LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
This work addresses meta-learning (ML) by considering deep networks with stochastic local winner-takes-all (LWTA) activations. This type of network units result in sparse representations from each model layer, as the units are organized into blocks where only unit generates non-zero output. The main operat- ing principle of the introduced units lies on stochastic principles, as the network performs posterior sampling over competing units to select the winner. Therefore, the proposed networks are explicitly designed to extract input data representa- tions of sparse stochastic nature, as opposed the currently standard determin- istic representation paradigm. We posit that these modeling principles, inspired from Bayesian statistics, yield representations of stronger generalization capacity: this is of immense importance in the case of ML, which is the focus of this work. As we experimentally show, our approach produces state-of-the-ard predictive ac- curacy on standard few-shot image classification benchmarks. This improvement comes with an immensely reduced network sized required for achieving this accu- racy; this amounts parameter reduction by one order of magnitude average compared to the current state-of-the-art.
INTRODUCTION
When we train machine learning models on problems with limited amounts of training data, cannot usually get good predictive performance (Lai, 2019; Sculley et al., 2015). This comes in contrast to the human ability quickly derive information from range of different tasks, and then adapt to a new task with limited available new examples (Kühl et al., 2020; Castro et al., 2008). In essence, this capability of the mind of learning how learn (Black al., 2006) has inspired researchers to investigate the concept of Meta-Learning (ML) (Lake et al., 2017; Vilalta & Drissi, 2002; Wang al., 2016; Zoph & Le, 2017; Flennerhag et al., 2020; Lake et al., 2015; Hutsebaut- Buysse et al., 2019).
There is large variety of deep learning methods for ML (Finn et al., 2019; Baik al.. 2020; Andrychowicz et al., 2016). Specifically, Finn et al. (2017) presented the Model-Agnostic Meta- Learning (MAML) algorithm that enables tuning the parameters of trained network to quickly learn new task with only few gradient updates. To get away with the entailed second-order com- putations, which are expensive, several researchers proposed appropriate first-order approximations for MAML; such works are the First-Order MAML (FOMAML) of Finn et al. (2017) and the Rep- tile algorithm of Nichol et al. (2018). Recently, several researchers have also considered Bayesian inference-driee methods for deep learning ML. extending upon older works built for conventional machine learning approaches (Yoon et al., 2018; Finn et al., 2018; Ravi & Beatson, 2019; Patacchi- ola al., 2020; Zou & Lu, 2020; Grant et al., 2018; Chen al., 2020).
This work proposes a different regard toward improving generalization capacity for deep networks in the context of ML. Specifically, proposed approach relies on the following main concepts:
The concept of sparse learned representations. For the first time in the literature of deep network-driven ML, we employ a mechanism that inherently learns to extract sparse data representations. This consists in replacing standard unit nonlinearities (e.g., ReLUU

---

Under review as a conference paper at ICLR 2022
with unit competition mechanism. Specifically, (linear) units are organized into blocks. Presented with some input, the units within a block engage in a competition process with only one winner. The outputs of all units except for the winner are zeroed out; the output of the winner retains its computed value (local winner takes-all, LWTA, architecture).
The concept of stochastic representations. We establish stochastic formulation for the previously described competition process. Specifically, we postulate that, within a block of competing units, winner is selected via sampling from an appropriate Categorical posterion. The corresponding winning probability of each unit is proportional to its linear computation (thus depending on the layer input). Via this competition process, we yield stochastic representations from network layers, that is representations that may change each time we present the network layer exactly the same input.
Based on the results from existing approaches, we posit that the proposed treatment of the ML prob- lem, which combines learned representation sparsity and stochasticity, will be extremely beneficial to the deep learning community. We dub our approach Stochastic LWTA for ML (StochLWTA-ML))
We perform variational Bayes treatment of the proposed model. We opt for a full Bayesian treat- ment, by also handling network weights as latent variables. That is, we elect to impose an appro- priate prior over the network weights and fit approximate (variational) posteriors. We evaluate approach on a number of standard benchmarks in the field, namely Omniglot (Lake et al., 2017), Mini-Imagenet (Vinyals et al., 2016) and C1FAR-100 (Krizhevsky, 2009). We show that our ap- proach offers a variety of advantages over the current state-of-the-arr methods, namely: (i) incurring reduced predictive error rate compared obtaining this performance with networks that comprise one order of magnitude less trainable pa- rameters, and therefore give rise to better computational efficiency and imposed memory footprint.
the currently state-of-the-ar methods in the field; (ii)
The remainder of this paper is organized as follows: In Section 2, we briefly review related work. Section 3 introduces our approach and provides the related training and prediction algorithms. In and compare our
Section 4, we perform a thorough experimental evaluation of StochLWTA-Ne findings to the current ofate-of-the-arr. In the final Section 5, we end up with the conclusions of our work, and suggest lines of further research.
2 RELATED WORK
2.1 LWTA LAYERS IN DEEP LEARNING
LWTA layers are not new in the field of deep learning; see, e.g., Srivastava et al. (2013). Although not much work has been pursued along these lines, the recent works of Panousis et al. (2019; 2021) and Voskou et al. (2021) have spurred some fresh interest in the field. These works have presented alternative implementations of the basic concepts LWTA units in the context of diverse deep network architectures. Specifically, Panousis et al. (2019) propose stochastic LWTA formulation which is founded upon the Indian Buffet process (IBP) prior, borrowed from nonparametric statis- ties; they this architecture effect data-driven network compression. In their follow-up paper (Panousis et al., 2021), they exploit the same technique to train adversarially-rouus deep networks. On the other hand, Voskou et al. (2021) consider different incarnation of stochastic LWTA archi- tectures, which relies on sampling the winner from Categorical posterior, driven from the layer input. This layer architecture is used to replace dense ReLU layers in Transformer networks; it is then shown to yield important benefits in a Sign-Language Translation benchmark.
This paper is different from the previous works in various ways: (i) stochasticity does not stem from utilization of the IBP: we rather adopt an approach similar to Voskou et al. (2021): (ii) we do not use the proposed architecture as a replacement for specific type of layer architecture (Transformer) that remains largely unchanged; instead, we build completely new net- works using these layers; (iii) we perform a full Bayesian treatment, by treating network weights as random variables; we do not employ this construction as a means of compressing the weights at prediction time, contrary to Panousis et al. (2019), Voskou et al. (2021); instead, we perform weight sampling at prediction time as a means of improving accuracy: and (iv) for the first time, we exam- ine how these principles perform in the context of deep network- ML. Note that, apart from I LWTA architectures, other data-driven sparsity models have also been proposed recently, e.g. Lee
greater network
2

---

Under review as a conference paper at ICLR 2022
al. (2018), Kessler al. (2021). However, none of these have been developed or evaluated in the context of an ML setting.
2.2 MODEL-AGNOSTIC META-LEARNING
ML (Schmidhuber, 1987), also referred to as learning learn (Thrun Pratt, 1998), has the goal of obtaining machine learning models that can learn new tasks with limited availability of data, and only a few gradient steps, by capitalizing upon information stemming from previously learned tasks. In this context, Finn et al. (2017) suggested model-agnostic algorithm for ML, that can be applied to any model trained via gradient descent. The introduced MAML algorithm initializes model parameters in way that can be quickly adapted to several types of new tasks.
Let us consider a model with parameters 0 and a parametric form fo- When the model is adapting to unseen task T, sampled from the distribution over tasks P(T), MAML initially runs few steps of inner-loop gradient descent that yields the task-specific parameter set
0 = a VOLT,
where a is the step size hyperparameter and LT, denotes the loss on the task T.
Subsequently, training proceeds to optimize the function for with respect to the model parameters 0. Assuming that the batch of tasks has size M, we can define the targeted meta-objective as:
M
M
Limeta (0) = Σ LL. (fo:
(fo
(2)
Optimization of this meta-objective over yields the outer-lopp update:
M
0 レ 0
Vo
Lr. (fo:)
(3)
where I stands for the outer-loop learning rate.
Finally, as MAML involves expensive computations stemming from the second-order updates of Eqs. (2) and (3), Finn et al. (2017) developed first-order approximation that reduces the outer-loop update (3) to:
-
Σ
V LT, (Jo:))
(4)
In other words, FOMAML computes the gradients with respect to the updated parameter values 0% but omits the gradients of 0' with respect O.
In a different fashion, the Reptile algorithm of Nichol et al. (2018) disposes the costly second- order updates of MAML by: (i) applying some inner-loop gradient descent steps using (1); and (ii) suggesting a new outer-loop that simply subtracts the parameters, 0, from the updates 0 (instead of computing derivatives):
0 0+ 0+0+3000
(5)
PROPOSED APPROACH
3.1 ARCHITECTURE
Let us denote as æ E I an input vector presented to dense ReLU layer of a conventional deep neural network, with corresponding weights matrix W e / The output of the layer is the vector E Eº and is fed to the subsequent layer. In our approach, ReLU unit is replaced by J competing linear units, organized in one block; in the following, we denote with R the number of blocks in a layer. The input x: is now presented to each block through weights that are organized into a three-dimensional matrix W E Ix Then, the j-th competing unit within r-th block computes the sum [ Z;. Competition means that, of the J units in the block, one unit (the "winner") will present its linear computation the next layer; the rest will present

---

Under review as a conference paper at ICLR 2022
values. Traditionally in the literature, the winner unit is selected to be the unit with greatest linear computation. Recently, stochastic competition principles have been considered, e.g. Panousis et al. (2019; 2021), Voskou et al. (2021).
Let us denote as y E RRJ the output of an LWTA layer; this is composed of R subvectors y, E RR' and is sparse, since all units except for one, in each block, yield zero values. Let us introduce the discrete latent indicator vector & E one (J) to denote the winner units in the R blocks that constitute considered stochastic LWTA layer. This vector comprises R component subvectors, where each component entails one non-zero value at the index position that corresponds to the winner unit of the respective LWTA block. On this basis, the output y of the stochastic LWTA layer's (r (r-j)= -))t component Yr.j is defined as:
Yr.j Eri Σ (Wars) x4
where we denote as Ery the j-th component of e., and & E one hot()) holds the r-th subvector of
In Eq. (6), we postulate that the latent winner indicator variables are drawn from Categorical distribution which is proportional to the intermediate linear computation that each unit performs. Therefore, the stronger the linearity the higher the chance of the unit winning the stochastic compe- tition within its block. In detail, we postulate that, posteriori, the winner distributions yield:
((E)) Categorical
Er
softmax
Σ Jurnalye xi)
where (0 denotes the vector concatenation of the set (wirs 1 A graphical illustration of the proposed stochastic architecture is provided in Fig. 1.
As a network composed of such (StochLWTA) layers entails latent variables &. we need to perform Bayesian network treatment perform effective parameter training. We opt for (approximate) stochastic gradient variational Bayes treatment (Kingma & Welling, 2014). for scalability purposes. This means that the used objective function takes the form of an evidence lower-bound (ELBO) objective, as we describe next. In our work, we take one step further: we also elect to infer posterior density over the network weights W, as opposed to obtaining point-estimates. This results in second source of stochasticity for our approach, which may further increase its generalization capacity under uncertain conditions arising from limited training data availability. Note that the use of these posteriors is totally different from Panousis et al. (2019) and Voskou et al. (2021): therein, posterior variance is used for compressing posterior mean bit-precision; then, predictions performed using only the compressed posterior mean. Instead, in work we sample multiple times from the trained weight posterior and perform model averaging (in a Bayesian sense), as a means of increasing generalization capacity.
We postulate:
q( @(vec(W) (W)) N diago())
where vp A (1s, 22) are the means and variances of the Gaussian weight posteriors, respectively.
3.2 A MODEL-AGNOSTIE ML ALGORITHM
Let us define an ML problem where we are given a training dataset of tasks, T, that are governed by distribution P(T). We consider N-way, K-shot learning setting, where each task contains K labelled examples for each of N available classes.
Our approach entails parameter sets 0 which coincide with the hyperparameter sets (H, a²) of the weights W E R Rx these sets ab = {μ, )²} are the target of our MAML-type training algorithm.
Therefore, to perform model training, we have to first initialize the trainable parameters μ and o² across layers. To this end, appropriately exploit popular initialization schemes, such as
4

---

Under review as a conference paper at ICLR 2022
LWTA layer
X1 "esnmMinroofy
(edithua))
Via
X2 Wary Waxy Nipres Raes)
fr
x
Di
wen -0 for
Wir' Nipiza mtrs)
X1
Figure 1: A zoomed-in graphical illustration of the r-th block of stochastic LWTA layer. Input æ [ri, D2, , | is presented to each unit in the block. Assume that the index of the winner unit Jo. Then, the output of the block is a vector with single non-zero value index ju-
..,
Glorot Uniform (Glorot Bengio, 2010). Then, our approach entails an inner-outer loop scheme, in vein similar existing approaches, but with some crucial differences:
1. The stochastic nature of the postulated weights, W. results in the updates taking place over the posterior means, μ and variances, o².
2. The stochastic nature of both the inferred representations and the network weights them- selves implies that proper training must rely on optimization of the ELBO function of the network. Let us consider an inner-loop dealing with task T; P(T), with data
D. = (X,, Yi) C D. Let CE(Y,, fo (X,; E. W)) be the categorical cross-entropy between the data labels Y; and the class probabilities for (X; W) generated by the penultimate Softmax layer. Then, we yield:
LT. (v) -CE(Y, for (X W)) KL( (I(()) (L(((W(WW))))))) (I))))
(9)
for task T. with dataset D; which is dealt with on the i-th training iteration.
Here, for simplicity and without harming generality, consider that the weights prior p(vec( W)) Gaussian distribution NN0, I) and the latent variables prior p(E) is a symmetric Categorical distribution Categorical(1//1) In addition, in our notation we stress that the output (N; & W) depends on the winner selection process, which is stochastic, and the outcomes of sampling the network weights. Specifically, in our work we perform Monte-Carlo (MC) sampling using one reparameterized sample of the corresponding latent variables.
Let be the unnormalized probabilities of the Categorical distribution g(E) (Eq. (7)). The sampled instances of &. E..j expressed as (Maddison et al., 2017):
E.. Softmax (log Er.j 9rr)/T), Vr 1,..., R, 1,..,, where 9r. = log( log Ur Unj ~ Uniform(() (0, 1), and r E (0, oo) a temperature factor that controls how closely the Categorical distribution is approximated by this continuous relaxation.
(10)
Similarly, the Gaussian weights yield: wr,r.j = Mt.r + Jenge, and NN0, 1).
On this basis, the KI. divergences in Eq. (9) become:
KT. KTTggEns [Ip(Er)) By(e.,))log ((rg) log (Er.(()))))
(11)
2 log g(Erg) - log I(Er))) Vr, j
and
K KLLg(wrrr))ll (I =B: Sy(weny)llop g(w),))) log pl p(wer)))
(12)
so log g(We.r)) - log (wrn.)) Vt = I,..., r,
5

---

Under review as a conference paper at ICLR 2022
Hence, the ELBO becomes:
Lr, (1b) CE(Y,,
Σ nd
(logo(Ens)) log logg(En)))
Σ (log (1b),,)))) log pl (v....)))
(13)
Therefore, we establish a MAML-type algorithm, where: (i) the used networks comprise blocks of stochastic LWTA units visually depicted in Fig. I; (ii) the trainable parameters are the means μ and variances o² of the synaptic weights; and (iii) the objective function of the inner-loop process is given in Eq. (13).
We summarize our training algorithm in Alg. 1.
Algorithm 1: Model training with StochLWTA-ML
Require: P(T): distribution over tasks
Initialize := {²),a²)) a²)
Define outer-step size 3 and inner learning rate
for = ..
Inner training loop:
Sample task T; ~ P(T), where T, contains data D, = (X,, c
Compute Lr, (+b) using Eq. (13)
Compute adapted parameters with SGD: upy p (fv)
Outer training loop:
Derive &o + + + 8(1) v)
end
3.3 PREDICTION ALGORITHM
At prediction time, we draw set of B samples of the Gaussian connection weights from the trained posteriors o²). Then, select the winning units in each block of the network by similarly sampling from the network, which we average obtain final
posteriors (E). This results in set B output logits predictive outcome:
fo (x W) R E (x. W.)
(14)
where &. and W. are sampled directly from the posteriors q(E) and g( (W), respectively. This concludes the formulation of the proposed modellagnostie ML approach.
Algorithm 2: Prediction with StochLWTA-MLL
Require:
Learned parameters sb = {14, " input data
Sample W ((vee(())) N NVvevvVVV [u,, diag(o ))
Sample v(E) defined in Eq. (7), for vb and (rr: x!) X'
Compute output logits, given the sampled values & and
Repeat B times
Use Eq. (14) to average over the resulting set of B logits and derive the final prediction.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We evaluate StochLWTA-ML on Omniglot, Mini-Imagenet and CIFAR-100 which are popular few-shot image classification datasets, and compare its performance to state-of-the-art prior results. After thorough exploration the number of LWTA layers as well the number of blocks for each layer and the competing units per block, we end up with using networks comprising layers with 16 blocks and competing units per block on the former layer, and blocks with 2 units per block on the latter. The penultimate network layer is a Softmax. Weight mean initialization, as well as roint-estimate initialization for our competitors. is performed via Glorot Uniform. Weight log-variance initialization is performed via Glorot Normal, by sampling from (0.0005, 0.01). The Eumbel-Softma relaxation temperature is
0.67.

---

Under review as a conference paper at ICLR 2022
In the inner-loop updates, learning rate of 0.003, For the outer-loop, we use SGD with a lincar annealed outer step size and initial value of 0.25. Additionally, all the experiments were ran with task batch size of 50 for both training and testing mode. Prediction is carried out averaging over B = 4 output logits.
the Stochastic Gradient Descent (SGD) (Robbins, 2007) optimizer with
The results presented in Sections 4.2 and 4.3 stand for the average performance over three runs with different random seeds. Note that each experiment consists of different number of iterations for training, depending on its convergence speed. The code was implemented in Tensorflow (Abadi et al., 2016).
4.2 RESULTS
Table I, we show how StochLWTA-ML performs on Omniglot 20-way, Mini-Imagenet 5-way and CIFAR- 100 5-way few-shot settings. We compare our findings to state-of-the-an art ML algorithms such as LLAMA (Grant et al., 2018) and PLATIPUS (Finn et al., 2018) reported in Gordon al. (2018), Amortized Bayesian Meta-Leamingg (ABML) (Ravi & Beatson, 2019), MAML, FOMAML (Finn al., 2017). Reptile (Nichol al., 2018) and others. Using the original architectures with the same hyperparameters. and data preprocessing as in Finn et al. (2017), we have also locally reproduced ABML, BMAML (with 5 particles), PLATIPUS, MAML, FOMAML and Reptile (dubbed "local" in Table 1). For completeness sake, we also compare our findings other oftt--o-thee-al ML models as reported in Finn et al. (2017), including Matching Nets (Santoro al., 2016) and LSTM Meta-Learner (Ravi Larochelle, 2017). As we observe, our method outperforms the existing state-of-the-art both the 1-shot and 5-shot settings.
Table I: N-way K-shot (%) classification accuracies on Omniglot, Mini-Imagenet and CIFAR-100
Omniglot
Mini-Imagen Hinneetm
CIFAR-
20-way
5-way
5-way
Algorithm
1-shot
5-shot
I-shot
5-shot
shot
5-shot
Matching Nets
93.80
98.50
43.56
55.31
LSTM Meta-Leamer
43.44
60.60
MAML
95.80
98.90
48.70
63.11
FOMAML
48.07
63.15
Reptile
88.14
96.65
47.07
62.74
PredCP (Nalisnick et al., 2021)
49.30
61.90
Neural Statistician (Edwards & Storkey, 2016)
93.20
98.10
MAP-SSVM (Triantafillou et al., 2017)
95.20
98.60
50.32
63.94
LLAMA
49.40
PLATIPUS
50.13
GEM-BML+ (Zou & Lu, 2020)
96.24
98.94
50.03
DKT (Patacchiola et al., 2020)
49.73
64.00
ABML
45.00
49.50
BMAML (with 5 particles) (Yoon al., 2018)
53.80
ABML (local)
90.21
93.39
44.23
52.12
49.23
53.60
BMAML (local)
96.92
98.11
53.10
64,80
52.60
65.80
PLATIPUS (local)
94.35
98.30
49.97
63.13
51.14
63.61
MAML (local)
95.48
98.61
48.60
63.01
50.67
62.89
FOMAMI (local)
94.92
98.12
47.93
63.10
49.13
63.80
Reptile (local)
87.98
96.36
46.97
62.53
48.19
63.45
StochI.WA-MIL.
97.79
98.97
54.11
66.70
54.60
66.73
4.3
ABLATION STUDY
4.3.1 DOES STOCHASTIC COMPETITION CONTRIBUTE TO CLASSIFICATION ACCURACY?
To check whether the accuracy improvements stem from the LWTA-induced sparsity or the proposed stochas- tie competition concept, we evaluate both our approach as well MAML, FOMAMI, ABML, BMAML and PLATIPUS, considering both "deterministic LWTA" and "stochastic LWTA" setups; deterministic LWTA net- works have been adopted from Srivastava et al. (2013). As we see in Table 2, replacing ReLU with deterministic LWTA yields negligible differences. On the other hand, stochastic LWTA units yield a clear improvement in all cases. This improvement becomes even more important in the case of our approach, where sample from stochastic weights.
7

---

Under review as a conference paper at ICLR 2022
Table 2: Mini-Imagenn 5-way Few-Shot ablation study (% accuracy)
Algorithm
Network type
I-shot
5-shot
deterministic LWTA
48.88
63.15
MAML (local)
stochastic LWTA
49.61
64.03
deterministic LW IA
48.11
63.54
FOMAMI. (local)
stochastic LWTA
49.24
64.54
deterministic LWTA
44.31
52.27
ABML (local)
stochastic LWTA
45.11
53.31
deterministic LWTA
53.12
64.84
BMAML (local)
stochastic LWTA
53.50
65.31
deterministic LWTA
49.99
63.21
PLATIPUS (local)
stochastic LWTA
51.06
64.18
deterministic LWTTA
53.12
64.93
WTOMLLWTA-ML
stochastic LWTA
54.11
66.70
4.3.2 EFFECT OF BLOCK SIZE J
As presented Table increasing the number competing units per block 4 or 8 does not notably improve the results of our approach. On the contrary, it increases the number of trained parameters, thus leading to higher network computational complexity. This corroborates our initial choice of using J = competing units per block in our approach.
Table 3: Effect of block size J in StochLWTA-ML' classification (%) accuracy
Omniglot 20-way
Mini-Imagenet 5-way
CIFAR-100 5-way
Number of units
I-shot
5-shot
1-shot
5-shot
I-shot
5-shot
97.79
98.97
54.11
66.70
54.60
66.73
96.33
98.55
53.99
66.65
54.51
66.13
=
95.38
98.83
53.70
67.08
54.45
66.18
4.3.3 How DOES THE NUMBER OF SAMPLES AT PREDICTION TIME AFFECT ACCURACY?
We scrutinize the effect of the number of drawn samples, B, StochLWTA ML's predictive accuracy. To this end, we repeat our experiments using B = 10 logits sets. In Table 4, we provide the comparative outcomes concerning the two sample size configurations of B = 4 and B = 10. As we observe, an increase in prediction sample size, B. yields a slight accuracy increase. However, the aforementioned increase in sample size imposes computational overhead, which elaborate upon in the following Section. We argue that this overhead might not worth it for slight performance increase reported Table More information on the effect of sample size in our approach's 's predictive performance are provided in the Supplementary.
Table 4: Effect of sample size in StochL WTA-ML3 classification (%) accuracy
Omniglot 20-way
Mini-Imagenet 5-way
CIFAR-
100
5-way
Number of samples
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
B=4 4
97.79
98.97
54.11 54.89
66.70
54.60
66.73
B= 10
96.91
99.23
67.22
55.18
66.15 15
4.3.4 IS THERE A COMPUTATIONAL TIME TRADE-OFF FOR THE INCREASED ACCURACY?
is also important to investigate whether our approach represents a trade-off between accuracy and com- putational time compared to our competitors. To facilitate this investigation, in Table we provide training iteration wall-clock times for our approach and the existing locally reproduced state-of-the-art, well as the total number of iterations each model needs to achieve the reported performance of Table It appears that methodology takes 77% less training time than the efficient algorithms ABML. BMAML, PLATIPUS, and comparable to other approaches. This happens because our approach yields the reported state-of the art per- formance by employing network architecture (that is. number of LWTA layers, as well as number of blocks and block size on each layer) that result in a total number of trainable parameters that is order magnitude less on average than the best performing haseline methods. This be in the last three columns of Table
8

---

Under review as a conference paper at ICLR 2022
S (dubbed DA, Da and De for Omniglot, Mini-Imagenet and CIFAR-100 respectively). In addition, training for our approach converges fast.
The situation changes when it comes to prediction: our approach imposes slight computational time overhead compared to MAML, FOMAML and Reptile, but still much less than the time-consuming PLATIPUS. BMAML and ABML. This is rather negligible increase when are dealing with low number of drawn samples, B, but increases increase B. The provided results suggest that selection of B 4 represents favorable accuracylprediction wall-clock time our approach.
Table 5: Performance comparison: average wall-clock time (in msecs), training iterations for each locally reproduced method and number of baselines' trainable parameters over the considered datasets of Table
Number training
Algorithm
Training
Prediction
Da parameters
Dg parameters
De parameters
iterations
PLATIPUS (local)
1603.39
602.77
333600
560025
613395
580440
BMAML (local)
514.43
560025
615395
580440
ABML (local)
678.48
265.78
138000
224010
246158
232176
MAML (local)
288.2 25
103.28
60000
112005
123079
116088
FOMAML (local)
284.49
102.34
60000
112005
123079
116088
Reptile (local)
284.30
102.27
60000
113221
124613
117463
(13.44(2)) =
StochL -M-
282.90
60000
54549
60112
56745
121.87
Finally, provide example of how training for approach converges, and how this compares the alternatives. We illustrate our outcomes on Omniglot 20-way 1-shot benchmark: similar outcomes have been observed in rest of considered datasets. Fig. 2(a) compares Stachl WTA-MI. with prior traditional MI. methods: MAMI., FOMAMI. and Reptile. It becomes apparent that our approach converges equally fast these competitors. Further, Fig. 2(b) compares MocchLWTA-ME with the probabilistic MIL models ABMI,, BMAML, PLATIPUS. Since,
in the ablation study of Section 4.3.4, these methods quite time- consuming and less efficient regarding memory consumption, StochLWTA-ML gives rise to an easier time training MAML based probabilistic model.
ML aigorithms convergence
ML aigorthmms convergence
100-
30-
SEOCHEWIIK.N
50-
Jope
NAME FORAS
Jul
40022
(a)
(b)
Figure 2: MI. algorithms' training convergence comparison
CONCLUSION
In this paper, we proposed sparse and stochastic network paradigm for MI, with novel network design prin- ciples compared to currently used model-agnostic ML models. We introduced stochastic LWTA activations in the context variational Bayesion treatment that gave rise doubly-stochastic ML framework, bearing the promise of stronger generalization capacity. We evaluated our approach using standard benchmarks in the field, and showed that it outperformed the state-of the-art in terms of both predictive accuracy and computa- tional costs. The results have provided strong empirical evidence supporting our claims. In the future, we plan study the effect of StnchLWTTA-M in areas related to MI,, such as Continual I Learning (Javed & White, 2019) and Reinforcement Learning (7hu al., 2020).

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, and Matthicu Devin. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. 265-20 263. In Proceedings of the 12th USENIX conference on Operating Systems Design and Implementation, 2016.
Marcin Andrychowicz. Misha Denil, Sergio Gomez, Matthew W. Hoffiman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning learn by gradient descent by gradient descent. In Neural Information Processing Systems, 2016.
Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, and Kyoung Mu Lee. Meta-learning with adaptive hyperparameters. Neural Information Processing Systems, 2020.
Paul Black, Robert McCormick, Mary James, and David Pedderd. Learning how learn and assessment for learning: theoretical inquiry. Research Papers in Education, 21:119-132, 2006.
Rui Castro, Charles Kalish, Robert Nowak, Ruichen Qian, Tim Rogers, and Jerry Zhu, Human active learning. In Neural Information Processing Systems, 2008
Yutian Chen, Abram L. Friesen, Feryal Behbahani, Arnaud Doucet, David Budden, Matthew Hoffman, and Nando de Freitas. Modular meta-learning with shrinkage. In Neural Information Processing Systems, 2020.
Harrison Edwards and Amos Storkey. Towards a neural statistician. 2016. URL arXLvpraprintarxXw 1606. 02185.
Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. In International Conference Machine Learning, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model- aadel-agnostic meta- meta-learning, In Neural Informa- tion Processing Systems, 2018
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning, 2019.
Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell, Meta- learning with warped gradient descent. In International Conference on Learning Representations, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Journal of Machine Learning Research 9, pp. 249-256, 2010.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E. Turner. Meta-learning probabilistic inference for prediction. In International Conference on Learning Representations, 2018.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- rradient-based meta-learning as hierarchical bayes. In International Conference on Learning Representations, 2018.
Matthias Hutsebaut- Buysss, Kevin Mets, and Steven Latré. Fast task-adaptation for tasks labeled using natural
language in reinforcement learning. 2019. URL https:/ /tapss:///aiiiiii/ieeeeeeee/rmas org/ abs /9100. 04040.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
K. Javed and M White. Meta-learning representations for continual learning, pp. 1818-1828. In Neural Infor-
mation Processing Syxtems, 2019.
Samuel Kessler, Vu Nguyen, Stefan Zohren, and Stephen Roberts. Hierarchical indian buffet neural networks for hayesian continual learning. In UAI, 2021.
D. P. Kingma and M Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Alex Krizhevsky, Leaminng multiple layers of features from tiny images. Technical report, 2009.
Niklas Kühl, Mare Goutier, Lucas Baier, Clemens Wolff, and Dominik Martin. Human vs. supervised machine learning: Who learns patterns faster? 2020. URL https: /ataaaiivvvvroo//eaa wwt.sbs.///eeo.t..cccc.........t hbtp://sss///a/./ee//eea 20t...2....0002 03661.
Yunfei Lai. A comparison of traditional machine learning and deep learning in image recognition. Journal of Physics: Conference Series, 1314, 2019.
10

---

Under review as a conference paper at ICLR 2022
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266;:3332 1338, 2015.
Brenden M Lake, Tomer Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40, 2017.
Juho Lee, Sachoon Kim, Jachong Yoon, Hac Beom Lee, Eunho Yang, and Sung Hwang. Adaptive network sparsification with dependent variational beta-bernoulli dropout. 2018. URL axxivpreerintarX....
1805. 10896.
Chris Maddison, Andriy Mnih, and Yee whye Tch. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations, 2017.
Eric Nalisnick, Jonathan Gordon, and José Miguel Hernández-Lobato Predictive complexity priors. In Neural Information Processing Systems, 2021.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. 2018. URL
https: /ttp/aaxivvvv////////////aaaavva org/ abs / 11003/8000000 02999.
Konstantinos Panousis, Sotirios Chatzis, Antonios Alexos, and Sergios Theodoridis. Local competition and stochasticity for adversarial robustness in deep learning. In International Conference on Artificial Intelli- gence and Statistics, 2021.
Konstantinos P. Panousis, Sotirios Chatzis, and Sergios Theodoridis. Nonparametric bayesian deep networks with local competition. International Conference on Machine Learning, 2019.
Massimiliano Patacchiola, Jack Turner, Elliot J. Crowley, Michael O'Boyle, and Amos Storkey. Bayesian meta-learning for the few-shot setting via deep kernels. In Neural Information Processing Systems, 2020.
Sachin Ravi and Alex Beatson. Amortized bayesian meta-leaminn. In International Conference Learning Representations, 2019.
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In International Conference on Learning Representations, 2017.
H. Robbins. stochastic approximation method. Annals of Mathematical Statisties, 2007.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-Icarning with megore-d neural networks. In International Conference on Machine Learning. 2016.
Jurgen Schmidhuber. learning how to leam: The meta-meta-... hook., evolutionary principles in self- referential learning. Master's 's thesis, Institut f. Informatik, Tech. Univ. Munich, 1987.
D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips. Hidden technical debt machine learning systems. In Neural Information Processing Systems, 2015.
Rupesh Kumar Srivastava, Jonathan Masci, Sohroh Kazerounian, Faustino Gomez, and Jürgen Schmidhube. Compete to compute. In Neural Information Processing Systems, 2013.
Thrun and L. (cds.). Pratt, Learning to Learn. Kluwer Academic Publishers, USA, 1998. ISBN 0792380479.
Triantafillou. R. Zemel, and R. Urtasun. Few-shot learning through an information retrieval lens. Neural Information Processing Systems, 2017.
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial Intelligence Review, 18(2):77-95,,) 2002.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching net- works for one shot learning. In Neural Information Processing Systems, 2016.
Andreas Voskou, Konstantinos Panousis, Dimitrios Kosmopoulos, Dimitris Metaxas, and Sotirios Chatzis. Stochastic transformer networks with linear competing units: Application to end-to-end sl translation. In International Conference on Computer Vision, 2021.
J. X Wang, Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. Z Leibo, R. Munos. C. Blundell, D. Kumaran, and M. Botvinick. Learning reinforcement learn. In International Conference on Machine Learning, 2016.
J. Yoon, T. Kim, O. Dia, O. Kim, Y. Bengio, and S. Ahn. Bayesian model-agnostic meta-lcamingg In Neural Information Processing Systems, 2018.
11

---

Under review as a conference paper at ICLR 2022
Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The ingredients of real-world robotic reinforcement learning. In International Conference on Learning Representations, 2020.
Barret Zoph and Quoc V Le. Neural architecture search with reinforeement learning. International Confer- ence on Machine Learning, 2017.
Yayi Zou and Xiaoqi Lu. Gradient-em bayesian meta-leamning In Neural Information Processing Systems, 2020.
A FURTHER DETAILS ON DATASETS EXPERIMENTAL FORMATION
Omniglot is dataset of 1623 characters from different alphabets, containing 20 examples per character scaled down to 28x28 grayscale pixels. The ratio between training and testings sets is 3:2, after shuftling the character classes we randomly choose the first 974 classes for training and the remaining are left for testing. As for Mini-Image Imagenet dataset, it has color images of size 84x84 and contains from the ImageNet dataset. We randomly choose 45000 examples for the training phase and the rest constitute the testing population. The CIFAR -000 dataset consists of color images of size 32x32 and contains 100 classes with 600 images per class. We randomly choose 500 images per class for training and the rest 100 images per class constitute the testing population.
classes with 600 examples
B ADDITIONAL EXPERIMENTS
In Table B1 we provide the results of the ablation study of Section 4.3.1 on the Omniglot 20- dataset. The full fledge StochLWTA-ML approach yields again better predictive performance compared to the alternative variants ML algorithms.
Table B1: Omniglot 20-way Few-Shot ablation study (% accuracy)
Algorithm
Network type
1-shot
5-shot
deterministic LWTA
95.52
98.15
MAML (local)
stochastic LWTA
95.91
98.78
deterministic LW
95.01
98.18
FOMAML (local)
stochastic LWTA
95.80
98.41
deterministic LWTA
90.30
93.64
ABML (local)
stochastic LWTA
91.21
93.91
deterministic LWTA
96.96
98.21
BMAML (local)
stochastic LWTA
97.11
98.30
deterministic LWTA
94.48
98.31
PLATIPUS (local)
stochastic LWTA
95.13
98.56
deterministic LWTA
96.95
98.63
StochL.WTA-MI.
stochastic LWTA
97.79
98.97
C FEW-SHOT CLASSIFICATION NETWORK ARCHITECTURES
For the local replicates of prior ML algorithms in the experiments of our work, we follow the same architecture for the deep neural network as the one used by Vinyals et al. (2016). For Omniglot, the network is composed of 4 convolutional layers with 64 filters. 3 convolutions and 2 strides, followed by Batch Normalization layer (loffe & Szegedy, 2015) and the final values of each layer are processed by activation function. For both Mini-Imagenet and CIFAR-
we use convolutional layers with 32 filters to reduce overfitting like Ravi & Larochelle (2017), 3 x convolutions followed by Batch Normalization layer and 2 x 2 max- mnax-pooling layer with the values of each layer finally passed again through an activation block. The activation functions used for experiments of main paper's Tables and 5 is ReL.U, and LWTA for experiments of main paper's 's Table 2 and Supplementary's Table B1.
12

---

Under review as a conference paper at ICLR 2022
D HOW DOES THE TASK BATCH SIZE AFFECT STOCHLWTA-ML'S PERFORMANCE?
In Fig. and 4, we illustrate the performance of our model on the Mini-Imagenet 5-way 1-shot setting with different task batch sizes. As we see, model performs optimally with task batch size of 50 for both training and lesting phase. Using this value for batch size, noticed that the classification accuracy well training time per iteration were optimal. This outcome was also observed rest of considered datasets' settings.
Mini-lmagenet 5-way 1-shot ablation study
56
55
Classifiiation accuracy (%)
Task
Figure 3: The effect of task batch size in StochLWTA's predictive accuracy
Mini-Imagenet 5-way 1-shot ablation study
300
Tranning time per iteration (in msecs)
295
290
285
280
275
35
65
100
Task batch size
Figure 4: The effect of task batch size in StochLWTA's training time per iteration (in msecs)
13

---

Under review as a conference paper at ICLR 2022
E HOW DOES THE SAMPLE SIZE B AT PREDICTION TIME AFFECT STOCHLWTA-ML' S ACCURACY?
As we observe in Fig. 5, an increase in sample size, B, does not always yield an accuracy increase. We finally choose B = 4, since at that point our model achieves its best predictive performance most of the experiments. In this figure, 100 5-way 5-shot settings. Our approach has shown similar behaviour the rest experimental settings, leading us to the same choice for the number sample size B.
illustrate
outcomes Omniglot 20-way 1-shot, Mini-Imagenet 5-way 1-shot and CIFAR-
100
Classification accuracy (%)
Omniglot way 1 shot
Mini-Imagenet 5-way 1-shot
CIFAR-100 5-way 5-shot
60
50
Sample
Figure 5: The effect sample size B in StochLWTA-MIL classification (%) accuracy
F WHAT PARAMETERS DO WE COUNT FOR THE OUTCOMES OF MAIN PAPER'S S TABLE 5?
The included parameters of each baseline for the outcomes of Table are:
PLATIPUS: θ {no, oZ vo, 7p: 7
BMAMI:: θ { 3 for using 5 particles
ABML: {po, o
MAML: 0 {mO}
: 0 = {on}
Reptile: 0 {oa}
StochLWTA ML. 0
14


Reviews:
[{"review_id": "Y6NBFDfOjqe", "reviewer": "Reviewer_Nj3T", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The novelty of the paper is limited. It applies stochastic LWTA to the standard MAML framework. Using Bayesian Model Averaging for its inferencing is also not new. Also, I do not find any advantage of this stochastic design which in fact induces large computation overhead.\n\n2. This paper lacks insight into how the proposed design is specifically related to meta-learning. Why does the \"sparse representation\" induced by the LWTA function help? \n \n3. The experiment results are strong. The authors provided a detailed experimental setting for reproducing the result. The improvement of the proposed model over MAML and FOMAML is significant. However, I would still suggest the authors compare their method to other recent state-of-the-art meta-learning methods.\n\n4. The authors provided an ablation study to empirically prove the stochastic LWTA is the underlying force for improving the model. However, I think the ablation study needs to be broken down into a deterministic LWTA function and a stochastic nature. It's not clear actually the model benefits from which part. The authors also discussed the computation overhead. These experiments are limited to small model settings without discussion of the role of batch size in the play.\n", "summary_of_the_paper": "This paper proposed a novel meta-learning method by replacing standard nonlinearity functions in MAML with stochastic local winner-takes-all (LWTA) activations. The authors claim that such design results in sparse representations and benefit meta-learning. This method demonstrated superior performance over MAML and FOMAML over standard benchmarks such as Omniglot and Mini-ImageNet.", "summary_of_the_review": "This paper improves standard MAML by replacing the nonlinearity with stochastic LWTA. The idea has limited novelty and limited insight.\nThe pros are the experiments are well performed, and the results are significant. I appreciate the ablation study and the overhead complexity discussion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RcuOLck-V8O", "reviewer": "Reviewer_rRh2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is interesting, but is not very clear. Here are a few examples that I was confused with when I first read the paper. I suggest the authors do a thorough revision to make the paper more approachable.\n\n1. Intuitively, why does LWTA units improve the performance with fewer data?\n2. What are the definitions of [\\xi]_{r, j} and [\\xi]_r? Because of this confusion, I couldn't really understand (9) or (10).\n3. What is a \"Discrete\" distribution? is it a \"multinoulli distribution\"?\n4. Why should the density of [\\xi]_r (again, I don't quite understand this notation) be proportional to the size of the activation?\n5. It is unclear to me why we need to sample W from a distribution, rather than obtaining its point estimate using ERM?\n6. The description of the training procedure on page 6 (especially the paragraph above (13) is very dense. I think I got a big picture of the proposal, but I don't think I can reproduce their methodology based on this description. ", "summary_of_the_paper": "In this paper, the authors proposed to replace the classical activation units in MAML with stochastic local winner-takes-all (LWTA) units. The authors argued the biologically motivated LWTA units would lead to better performance in the few-shot setting where the support set in the target task is small, in which case the embedding representation of the target task would be noisy. The authors framed their proposal in the Bayesian setting, and proposed algorithms to estimate the distribution parameters.", "summary_of_the_review": "I think the paper is interesting, but the presentation needs to be greatly improved. Many notations are used without introduction, and the description of the methodology is very dense that I am afraid that unless the reader is very familiar with the field, they could not reproduce the methods based on the description", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AhB8PgRSyc1", "reviewer": "Reviewer_czsy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n\n* The idea to embed data-driven/learned sparsity in a meta-learning framework is really interesting. These methods have been shown to improve robustness, are open to continual learning and have been shown to enable learning independent mechanisms. It is a nice idea and this is backed up by competitive performance of the algorithm compared to original instantiations of MAML.\n\n* The paper is well written and clear. As the reviewer of this paper, I feel that I have sufficient information and understanding to implement it myself\n\n[Weaknesses]\n\n* I find the methodological advances proposed in the manuscript too incremental for publication at ICLR. The SLWTA method proposed within appears to be the method of Panousis et al. (2021) (cited in text) but adapted to the MAML paradigm.\n\n* The probabilistic model introduced by the authors bares many simiilarities with other data-driven sparsity models that are inline with Panousis et al. (2021). There is:\n1. https://arxiv.org/abs/1805.10896 - Adaptive Network Sparsification with Dependent Variational Beta-Bernoulli Dropout\n2. https://arxiv.org/abs/1912.02290 - Hierarchical Indian Buffet Neural Networks for Bayesian Continual Learning\nBoth who learn modularity with a similar ELBO as proposed by the authors. \n\n* I think there are too few competiting baselines. Why were more recent gradient-based ML algorithms not considered such as Bayesian MAML (https://arxiv.org/abs/1806.03836) which first framed MAML in a Bayesian perspective?\n\n\n", "summary_of_the_paper": "This paper proposes a stochastic local winner-takes-all (SLWTA) approach to learn data-driven (stochastic) sparsity. This is motivated by the limited availability of training data, especiially in few-shot meta learning scenarios. The authors propose a meta-learning algorithm for their SLWTA approach. As this is a probabilistic approach, one can draw from the approximate posterior at inference to obtained the posterior predictive distribution. The authors compare their method against MAML, FOMAML and Reptile on the Omniglot and Mini-Imagenet displaying better performance than the origiinal MAML algorithm.", "summary_of_the_review": "I like the idea to embed an stochastic LWTA approach into meta-learning. It makes sense and the motivations are well presented. However, the technical novelty of the paper is lacking and does not offer enough methodological advancement to be a strong candidate for acceptance. \n\n**** Post-rebuttal ****\nI have decided to increase my score from 3 to 5. My reasoning is in my response to the author rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ARBpPTI_aJL", "reviewer": "Reviewer_3Mcm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n* Strong empiracle results \n\n* Great set of ablation studies\n\n## Weaknesses\n\n### Lack of novelty\n\nThere already exist many probabilistic approaches to meta-learning which take uncertainty into account, which the authors have failed to acknowledge. Thus their claims of novelty in this regard are incorrect. For example, the statement \"these innovations constitute a radical departure from the currently prominent design paradigm in ML\" is far too strong a statement given the work of Finn et al. (2018), Nalisnick et al. (2021), Chen et al. (2020), and Gordon et al. (2019).\n\n### Somewhat narrow set of main experiments\n\nThe main set of results in the paper are based only on the Omniglot 20-way and Mini-imagenet 5-way tasks. While the results on these tasks are good, more tasks are required to paint an accurate picture of the method. Cifar100 few shot would be a good start, but even better would be a non-image classification task. I might suggest the ShapeNet View Reconstruction task as used in Gordon et al. (2019). \n\nFurthermore, at least some of the methods from Finn et al. (2018), Nalisnick et al. (2021), Chen et al. (2020), and Gordon et al. (2019) should be compared against. \n\nAn additional ablation study comparing the performance of Stochastic LWTA and standard LWTA would be informative. \n\nFor the effect of sample size ablation, plots with performance on the y-axis and B on the x-axis for B in {1,...,10} would be very interesting as they would hopefully show a \"knee-point\" at which it stops being worthwhile to increase B.\n\n### Lack of clarity\n\nOverall I found the presentation to be confusing in many places and occasionally incorrect. I will list the issues I spotted below. \n\n1. The authors frequently refer to \"stochastic arguments\", \"modelling arguments\", \"deep network arguments\", etc. However, the usage of the word \"argument\" in these contexts does not make sense to me and I am unsure exactly what is meant.\n\n2. The authors frequently use the word \"inference\" to refer to test time prediction. However, this word also refers to the probabilistic inference for random variables. Given that the authors' proposed method makes use of such inference this becomes somewhat confusing. I would recommend only using inference in the probabilistic sense. \n\n3. \\vec{y}_r is introduced in sec 2.2 but it is not mentioned until much later in sec 3.1 that \\vec{y} is the concatenation of \\vec{y}_r. \n\n4. It is not clear why it is a good idea to use a fixed competition function that always chooses the maximum activation. It would be great if the authors could provide some motivation/intuition for this choice. (As I mention below, I think that the Sparse-MoE approach with an input-dependent gating network makes more sense). \n\n5. It is not clear what the authors mean by \"We posit that this novel stochastic paradigm will give rise to more **potent** learned representations ...\" (emphasis my own). (This is another example of a claim that overstates the novelty of this work.)\n\n6. (Minor) I assume that the authors mean \"model-agnostic ML\" rather than \"meta-agnostic ML\" at the start of sec 3.1. \n\n7. (Minor) The authors introduce the symbols A_1 and A_2 to indicate the input and output dimensions of their linear units. Would I and O not be more clear? \n\n8. Paragraphs 2 and 3 are largely redundant as they mainly contain repeated information from sec 2.2.\n\n9. (Minor) In eqn 9, the notation [\\vec{y}]_{r, j} is used, but just above the notation y = [\\vec{y}_{r, j}]. I found this somewhat confusing. \n\n10. In the paragraph below eon 9, the authors first say that they \"postulate\" that the indicator variable is drawn from a Discrete distribution. Later in the paragraph they say that they \"stipulate\" this. Which one is it?\n\n11. Speaking of the \"Discrete\" probability distribution, would it not be more clear to refer to a \"Categorical\" probability distribution?  \n\n12. In the paragraph after eqn 10, the authors state that they impose a posterior density over the weights. However, it is more accurate to say that the prior density is impose and the posterior is inferred. \n\n13. The authors frequently refer to \"stochasticity\" as increasing the generalisation capacity. However, it should be noted that simply adding stochasticity does not help. It must be added in a principled and theoretically justified manner. \n\n14. \"This concludes the formulation of the proposed model-agnostic ML approach.\" – I suspect this line should be at the end of sec 3.3, rather than sec 3.1.\n\n15. The authors refer to independence among layers in the context of the mean-field approximation. I assume that this was meant to be independence among weights? \n\n16. Eqn 13, for the Gumbel-softmax does not show the purpose of the temperature \\tau.\n\n17. In algorithm 2, line 2, the authors suggest that the variational parameters {\\mu, \\sigma} should be sampled. I suspect that the authors mean \\theta or \\vec{w}, should be sampled? \n\n17. In algorithm 2, the authors should show the equation for the BMA. \n\n18. At the top of page 7, the authors refer to \"2 competing units per block\" and then \"2 neurons per block\". Are neurons the same as competing units? \n\n19. \"For the outer-loop, we use SGD with a linear annealed outer step size equal to 0.\" Should this actually be 0? \n\n20. What is the architecture that was used for the baseline methods? \n\n21. The authors claim a 10% improvement over the \"best alternatives\" however, I think it is a 10% improvement over the *worst* alternatives. \n\n22. In table 4, it surprises me that Reptile runs slower than MAML. I don't think this makes sense. Also in table 4, the best value in the \"inference\" column should be bolded. \n\n23. In table 5, does the parameter count for LWTA include both variational parameters \\mu and \\sigma^2? OR is it just \\theta? \n\n24. More motivation is needed for the statement \"It appears that convergence of our approach is even smoother; ...\". \n\n25. Figure 2b, should be replaced with Figure 2a but for Mini-imagenet. \n\n## Other comments\n\n1. There is a broad range of work involving so-called Sparse Mixture of Experts layers which are conceptually very similar to the stochastic LWTA activated layers. Sparse MoEs have the advantage that they are more computationally efficient. They also have, in my opinion, a more sensible input-dependent gating mechanism compared to simply choosing the most positive activation. These have not been mentioned at all in this work. See (Riquelme et al., 2021) and (Shazeer et al., 2017).\n\n2. I think that the statement \"there is an increasing body of evidence from Neuroscience that neurone with similar functions ...\" requires a citation. Perhaps (Lasner, 2009) is the source for this statement, but that is not clear.\n\n3. \"To ensure statistical significance ...\" – providing multiple runs (while greatly appreciated) does not make the results statistically significant. If the authors wish to claim statistical significance, they should perform the appropriate statistical tests.\n\n4. It is not clear to me that the authors' proposed method actually provides better results for the harder Mini-imagenet task. It is true that the raw gain of 10% Mini-imagenet is larger than the 2% for Omniglot. However, because Omniglot accuracy is nearly saturated by the baselines already, this 2% improvement may actually be even more impressive. \n\n## References\n\nCarlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, Neil Houlsby:\nScaling Vision with Sparse Mixture of Experts. CoRR abs/2106.05974 (2021)\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, Jeff Dean:\nOutrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR (Poster) 2017\n\nChelsea Finn, Kelvin Xu, Sergey Levine:\nProbabilistic Model-Agnostic Meta-Learning. NeurIPS 2018: 9537-9548\n\nEric T. Nalisnick, Jonathan Gordon, José Miguel Hernández-Lobato:\nPredictive Complexity Priors. AISTATS 2021: 694-702\n\nYutian Chen, Abram L. Friesen, Feryal Behbahani, Arnaud Doucet, David Budden, Matthew Hoffman, Nando de Freitas:\nModular Meta-Learning with Shrinkage. NeurIPS 2020\n\nJonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, Richard E. Turner:\nMeta-Learning Probabilistic Inference for Prediction. ICLR (Poster) 2019", "summary_of_the_paper": "The paper proposes to use neural networks with so-called \"stochastic local winner-takes-all (LWTA)\" activations in the place of standard ReLU activations. These stochastic LWTA activations result in a model with sparse representations. The sparse gating is treated as a random variable. The authors also treat the weights of the neural network as random variables. Both sets of random variables are learnt using variational inference.", "summary_of_the_review": "While this paper does have some strengths (strong empirical results, great set of ablation studies), it is let down by many issues of clarity, a lack of novelty, and a somewhat narrow set of main experiments. \n\n++++ Post-revision update ++++\n\nThe authors have addressed many of my concerns, including correcting some of my misunderstandings, improving the clarity of the text in many ways, and adding additional experimental results in the form of ablations, baselines, and a new task. With this in mind, I have increased my score from 3 to 5.\n\nI have not increased my score further as I still find some parts of the presentation slightly confusing (e.g. I don't think the word \"principals\" is much better than the word \"argument\") and more importantly, I still believe that having experiments only with image classification tasks is slightly too narrow.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a meta-learning method where the standard ReLU activation units are replaced by the stochastic LWTA units to learn data-driven sparse representation.  Most of reviewers like the idea of embedding the stochastic LWTA into a MAML framework. Initial assessment pointed out the lack of clarity in various places in the paper. Authors did a good job in the author’s rebuttal period, to clarify the paper. Experiments demonstrated the competitive performance over existing meta-learning methods. Two of reviewers raised their overall score to 5 (from 3). However, all reviewers have concerns in the incremental technical novelty and feel that presentation should be improved to make the paper more clear and friendly to readers. While the idea is interesting, which is backed up by experiments, the paper is not ready for the publication at the current stage. I encourage to resubmit the paper after addressing these concerns.
Decision: Reject