Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a stochastic local winner-takes-all (SLWTA) approach to learn data-driven (stochastic) sparsity. This is motivated by the limited availability of training data, especiially in few-shot meta learning scenarios. The authors propose a meta-learning algorithm for their SLWTA approach. As this is a probabilistic approach, one can draw from the approximate posterior at inference to obtained the posterior predictive distribution. The authors compare their method against MAML, FOMAML and Reptile on the Omniglot and Mini-Imagenet displaying better performance than the origiinal MAML algorithm.
main_review: [Strengths]

* The idea to embed data-driven/learned sparsity in a meta-learning framework is really interesting. These methods have been shown to improve robustness, are open to continual learning and have been shown to enable learning independent mechanisms. It is a nice idea and this is backed up by competitive performance of the algorithm compared to original instantiations of MAML.

* The paper is well written and clear. As the reviewer of this paper, I feel that I have sufficient information and understanding to implement it myself

[Weaknesses]

* I find the methodological advances proposed in the manuscript too incremental for publication at ICLR. The SLWTA method proposed within appears to be the method of Panousis et al. (2021) (cited in text) but adapted to the MAML paradigm.

* The probabilistic model introduced by the authors bares many simiilarities with other data-driven sparsity models that are inline with Panousis et al. (2021). There is:
1. https://arxiv.org/abs/1805.10896 - Adaptive Network Sparsification with Dependent Variational Beta-Bernoulli Dropout
2. https://arxiv.org/abs/1912.02290 - Hierarchical Indian Buffet Neural Networks for Bayesian Continual Learning
Both who learn modularity with a similar ELBO as proposed by the authors. 

* I think there are too few competiting baselines. Why were more recent gradient-based ML algorithms not considered such as Bayesian MAML (https://arxiv.org/abs/1806.03836) which first framed MAML in a Bayesian perspective?



summary_of_the_review: I like the idea to embed an stochastic LWTA approach into meta-learning. It makes sense and the motivations are well presented. However, the technical novelty of the paper is lacking and does not offer enough methodological advancement to be a strong candidate for acceptance. 

**** Post-rebuttal ****
I have decided to increase my score from 3 to 5. My reasoning is in my response to the author rebuttal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use neural networks with so-called "stochastic local winner-takes-all (LWTA)" activations in the place of standard ReLU activations. These stochastic LWTA activations result in a model with sparse representations. The sparse gating is treated as a random variable. The authors also treat the weights of the neural network as random variables. Both sets of random variables are learnt using variational inference.
main_review: ## Strengths

* Strong empiracle results 

* Great set of ablation studies

## Weaknesses

### Lack of novelty

There already exist many probabilistic approaches to meta-learning which take uncertainty into account, which the authors have failed to acknowledge. Thus their claims of novelty in this regard are incorrect. For example, the statement "these innovations constitute a radical departure from the currently prominent design paradigm in ML" is far too strong a statement given the work of Finn et al. (2018), Nalisnick et al. (2021), Chen et al. (2020), and Gordon et al. (2019).

### Somewhat narrow set of main experiments

The main set of results in the paper are based only on the Omniglot 20-way and Mini-imagenet 5-way tasks. While the results on these tasks are good, more tasks are required to paint an accurate picture of the method. Cifar100 few shot would be a good start, but even better would be a non-image classification task. I might suggest the ShapeNet View Reconstruction task as used in Gordon et al. (2019). 

Furthermore, at least some of the methods from Finn et al. (2018), Nalisnick et al. (2021), Chen et al. (2020), and Gordon et al. (2019) should be compared against. 

An additional ablation study comparing the performance of Stochastic LWTA and standard LWTA would be informative. 

For the effect of sample size ablation, plots with performance on the y-axis and B on the x-axis for B in {1,...,10} would be very interesting as they would hopefully show a "knee-point" at which it stops being worthwhile to increase B.

### Lack of clarity

Overall I found the presentation to be confusing in many places and occasionally incorrect. I will list the issues I spotted below. 

1. The authors frequently refer to "stochastic arguments", "modelling arguments", "deep network arguments", etc. However, the usage of the word "argument" in these contexts does not make sense to me and I am unsure exactly what is meant.

2. The authors frequently use the word "inference" to refer to test time prediction. However, this word also refers to the probabilistic inference for random variables. Given that the authors' proposed method makes use of such inference this becomes somewhat confusing. I would recommend only using inference in the probabilistic sense. 

3. \vec{y}_r is introduced in sec 2.2 but it is not mentioned until much later in sec 3.1 that \vec{y} is the concatenation of \vec{y}_r. 

4. It is not clear why it is a good idea to use a fixed competition function that always chooses the maximum activation. It would be great if the authors could provide some motivation/intuition for this choice. (As I mention below, I think that the Sparse-MoE approach with an input-dependent gating network makes more sense). 

5. It is not clear what the authors mean by "We posit that this novel stochastic paradigm will give rise to more **potent** learned representations ..." (emphasis my own). (This is another example of a claim that overstates the novelty of this work.)

6. (Minor) I assume that the authors mean "model-agnostic ML" rather than "meta-agnostic ML" at the start of sec 3.1. 

7. (Minor) The authors introduce the symbols A_1 and A_2 to indicate the input and output dimensions of their linear units. Would I and O not be more clear? 

8. Paragraphs 2 and 3 are largely redundant as they mainly contain repeated information from sec 2.2.

9. (Minor) In eqn 9, the notation [\vec{y}]_{r, j} is used, but just above the notation y = [\vec{y}_{r, j}]. I found this somewhat confusing. 

10. In the paragraph below eon 9, the authors first say that they "postulate" that the indicator variable is drawn from a Discrete distribution. Later in the paragraph they say that they "stipulate" this. Which one is it?

11. Speaking of the "Discrete" probability distribution, would it not be more clear to refer to a "Categorical" probability distribution?  

12. In the paragraph after eqn 10, the authors state that they impose a posterior density over the weights. However, it is more accurate to say that the prior density is impose and the posterior is inferred. 

13. The authors frequently refer to "stochasticity" as increasing the generalisation capacity. However, it should be noted that simply adding stochasticity does not help. It must be added in a principled and theoretically justified manner. 

14. "This concludes the formulation of the proposed model-agnostic ML approach." – I suspect this line should be at the end of sec 3.3, rather than sec 3.1.

15. The authors refer to independence among layers in the context of the mean-field approximation. I assume that this was meant to be independence among weights? 

16. Eqn 13, for the Gumbel-softmax does not show the purpose of the temperature \tau.

17. In algorithm 2, line 2, the authors suggest that the variational parameters {\mu, \sigma} should be sampled. I suspect that the authors mean \theta or \vec{w}, should be sampled? 

17. In algorithm 2, the authors should show the equation for the BMA. 

18. At the top of page 7, the authors refer to "2 competing units per block" and then "2 neurons per block". Are neurons the same as competing units? 

19. "For the outer-loop, we use SGD with a linear annealed outer step size equal to 0." Should this actually be 0? 

20. What is the architecture that was used for the baseline methods? 

21. The authors claim a 10% improvement over the "best alternatives" however, I think it is a 10% improvement over the *worst* alternatives. 

22. In table 4, it surprises me that Reptile runs slower than MAML. I don't think this makes sense. Also in table 4, the best value in the "inference" column should be bolded. 

23. In table 5, does the parameter count for LWTA include both variational parameters \mu and \sigma^2? OR is it just \theta? 

24. More motivation is needed for the statement "It appears that convergence of our approach is even smoother; ...". 

25. Figure 2b, should be replaced with Figure 2a but for Mini-imagenet. 

## Other comments

1. There is a broad range of work involving so-called Sparse Mixture of Experts layers which are conceptually very similar to the stochastic LWTA activated layers. Sparse MoEs have the advantage that they are more computationally efficient. They also have, in my opinion, a more sensible input-dependent gating mechanism compared to simply choosing the most positive activation. These have not been mentioned at all in this work. See (Riquelme et al., 2021) and (Shazeer et al., 2017).

2. I think that the statement "there is an increasing body of evidence from Neuroscience that neurone with similar functions ..." requires a citation. Perhaps (Lasner, 2009) is the source for this statement, but that is not clear.

3. "To ensure statistical significance ..." – providing multiple runs (while greatly appreciated) does not make the results statistically significant. If the authors wish to claim statistical significance, they should perform the appropriate statistical tests.

4. It is not clear to me that the authors' proposed method actually provides better results for the harder Mini-imagenet task. It is true that the raw gain of 10% Mini-imagenet is larger than the 2% for Omniglot. However, because Omniglot accuracy is nearly saturated by the baselines already, this 2% improvement may actually be even more impressive. 

## References

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, Neil Houlsby:
Scaling Vision with Sparse Mixture of Experts. CoRR abs/2106.05974 (2021)

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, Jeff Dean:
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR (Poster) 2017

Chelsea Finn, Kelvin Xu, Sergey Levine:
Probabilistic Model-Agnostic Meta-Learning. NeurIPS 2018: 9537-9548

Eric T. Nalisnick, Jonathan Gordon, José Miguel Hernández-Lobato:
Predictive Complexity Priors. AISTATS 2021: 694-702

Yutian Chen, Abram L. Friesen, Feryal Behbahani, Arnaud Doucet, David Budden, Matthew Hoffman, Nando de Freitas:
Modular Meta-Learning with Shrinkage. NeurIPS 2020

Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, Richard E. Turner:
Meta-Learning Probabilistic Inference for Prediction. ICLR (Poster) 2019
summary_of_the_review: While this paper does have some strengths (strong empirical results, great set of ablation studies), it is let down by many issues of clarity, a lack of novelty, and a somewhat narrow set of main experiments. 

++++ Post-revision update ++++

The authors have addressed many of my concerns, including correcting some of my misunderstandings, improving the clarity of the text in many ways, and adding additional experimental results in the form of ablations, baselines, and a new task. With this in mind, I have increased my score from 3 to 5.

I have not increased my score further as I still find some parts of the presentation slightly confusing (e.g. I don't think the word "principals" is much better than the word "argument") and more importantly, I still believe that having experiments only with image classification tasks is slightly too narrow.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors proposed to replace the classical activation units in MAML with stochastic local winner-takes-all (LWTA) units. The authors argued the biologically motivated LWTA units would lead to better performance in the few-shot setting where the support set in the target task is small, in which case the embedding representation of the target task would be noisy. The authors framed their proposal in the Bayesian setting, and proposed algorithms to estimate the distribution parameters.
main_review: The paper is interesting, but is not very clear. Here are a few examples that I was confused with when I first read the paper. I suggest the authors do a thorough revision to make the paper more approachable.

1. Intuitively, why does LWTA units improve the performance with fewer data?
2. What are the definitions of [\xi]_{r, j} and [\xi]_r? Because of this confusion, I couldn't really understand (9) or (10).
3. What is a "Discrete" distribution? is it a "multinoulli distribution"?
4. Why should the density of [\xi]_r (again, I don't quite understand this notation) be proportional to the size of the activation?
5. It is unclear to me why we need to sample W from a distribution, rather than obtaining its point estimate using ERM?
6. The description of the training procedure on page 6 (especially the paragraph above (13) is very dense. I think I got a big picture of the proposal, but I don't think I can reproduce their methodology based on this description. 
summary_of_the_review: I think the paper is interesting, but the presentation needs to be greatly improved. Many notations are used without introduction, and the description of the methodology is very dense that I am afraid that unless the reader is very familiar with the field, they could not reproduce the methods based on the description
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a novel meta-learning method by replacing standard nonlinearity functions in MAML with stochastic local winner-takes-all (LWTA) activations. The authors claim that such design results in sparse representations and benefit meta-learning. This method demonstrated superior performance over MAML and FOMAML over standard benchmarks such as Omniglot and Mini-ImageNet.
main_review: 1. The novelty of the paper is limited. It applies stochastic LWTA to the standard MAML framework. Using Bayesian Model Averaging for its inferencing is also not new. Also, I do not find any advantage of this stochastic design which in fact induces large computation overhead.

2. This paper lacks insight into how the proposed design is specifically related to meta-learning. Why does the "sparse representation" induced by the LWTA function help? 
 
3. The experiment results are strong. The authors provided a detailed experimental setting for reproducing the result. The improvement of the proposed model over MAML and FOMAML is significant. However, I would still suggest the authors compare their method to other recent state-of-the-art meta-learning methods.

4. The authors provided an ablation study to empirically prove the stochastic LWTA is the underlying force for improving the model. However, I think the ablation study needs to be broken down into a deterministic LWTA function and a stochastic nature. It's not clear actually the model benefits from which part. The authors also discussed the computation overhead. These experiments are limited to small model settings without discussion of the role of batch size in the play.

summary_of_the_review: This paper improves standard MAML by replacing the nonlinearity with stochastic LWTA. The idea has limited novelty and limited insight.
The pros are the experiments are well performed, and the results are significant. I appreciate the ablation study and the overhead complexity discussion.
review_title: null

Meta-review: This paper presents a meta-learning method where the standard ReLU activation units are replaced by the stochastic LWTA units to learn data-driven sparse representation.  Most of reviewers like the idea of embedding the stochastic LWTA into a MAML framework. Initial assessment pointed out the lack of clarity in various places in the paper. Authors did a good job in the author’s rebuttal period, to clarify the paper. Experiments demonstrated the competitive performance over existing meta-learning methods. Two of reviewers raised their overall score to 5 (from 3). However, all reviewers have concerns in the incremental technical novelty and feel that presentation should be improved to make the paper more clear and friendly to readers. While the idea is interesting, which is backed up by experiments, the paper is not ready for the publication at the current stage. I encourage to resubmit the paper after addressing these concerns.
Decision: Reject