Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a meta-training framework underlain by a novel Hierarchical Gaussian Mixture based Task Generative Model (HTGM). The basic assumption is that task embedding fits mixture distribution of Gaussian. The model parameters are learned end-to-end by maximum likelihood estimation via an Expectation-Maximization algorithm. Extensive experiments on benchmark datasets indicated the effectiveness of the method for both sample classification and novel task detection.
strength_and_weaknesses: Strength

S1. I like the idea. Based on the observation that embedding is the key of few-shot learning, modeling with mixture of Gaussian makes perfect sense and is mathmatically solvable.

S2. The paper is well written and easy to follow.

Weaknesses

W1. Task grouping was extensively studied in the context of multi-task learning, e.g., [R1]. As both such mult-task learning algorithms (with metric-based few-shot algorithm pluggined) and the proposed method address a similar problem, agnostic to feature extractors, I think it is worthy discussing them, especially those with similar mathematical tools, in Related works.

[R1] Flexible Modeling of Latent Task Structures in Multitask Learning. ICML 2012.

W2. Experimental comparisons could be more solid. E.g.,  to compare with baselines also designed for dealing with mixture distributions of tasks, i.e., HSML and ARML, both the backbone and base algorithm are different, making it less convincing that the proposed method is better.
clarity,_quality,_novelty_and_reproducibility: Could you please discuss the limitation or in which case the algorithm will fail?
summary_of_the_review: This paper proposed a novel algorithm for meta learning. It is based on embedding-based few shot learning and mixture distribution of Gaussian for task generation. The experiments also demonstrated the effectiveness of few-shot learning and novel task detection. Overall the paper is well written.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes an approach for few-shot learning, that provides a way to model not only class distributions, but the distribution of tasks.  This provides two benefits: (i) improved performance on few-shot tasks, and more interestingly, (ii) the ability to determine whether a task (support set of (images, class) pairs) is "novel" and too far outside the domains the few-shot learner was trained on.  While other works always evaluate on novel classes, sometimes chosen to come from semantically different subtrees than those in meta-training, this work explicitly models the distribution of tasks themselves using Gibbs distribution with basis {W}, in order to identify out-of-distribution tasks.  This forms a hierarchy of mixture models, describing the tasks, classes and embeddings.  The system is optimized using EM, with SGD in the M step.  It is evaluated on Plain-Multi and Art-Multi datasets, finding improved performance over current few-shot-learning techniques augmented with GMM in-distribution model.

strength_and_weaknesses: Modeling task distribution in this way is an interesting idea, and the results of the system are promising, improving upon appropriate baselines.  The formulation is rather complex, leading to a somewhat complicated system, and whether the improvements are worth the complexity is debatable -- but it's certainly a reasonable trade-off.

I think some of the notation and descriptions could be made simpler (see some suggestions below), which might help (see below comments), but the system is pretty described as it is.  I had some trouble understanding the inference procedure, though:  In sec 3, I didn't understand the process for inferring the task embedding $v_{\tau'}$.  This section says a single $v$ is sampled and then means are adapted to drawn towards the Wv means; but should this use either the most likely $v$ given class prototypes $\mu^s$, or a larger sample of $v$ from the distribution?

Also, another approach might have been to use backpropagation, rather than EM, perhaps combined with Gumbel-softmax on the categorical from a direct weight parameterization.  Have you explored other potentially simpler approaches (I'm not sure if this one would work) that still capture the task distribution model?

Overall this is an interesting idea and model, with reasonable effectiveness.  It is somewhat complex, though, and I wonder if it (or some of its descriptions) could be simplified.




Further questions and comments:

The notation is a little involved and can be hard to follow sometimes, e.g. differences between $\mu$, $\hat\mu$, $\bar\mu$.  Since all the distributions are MOG or MOG-like, except the top level categorical, maybe it could work to index each level instead, e.g. calling the means $\mu^l$ where $l$ is the hierarchy level, and similarly for other variables.

Is there some intuition for this form of W parameterization, for how it models a task distribution?  it looks like a dictionary W * coeffs v, is one interpretation?  In the limiting case it might have a blockwise W_i = [0 | w_i1, ..., w_ik | 0], with w_ik being a basis for task i, and coeffs in the corresponding block of v.  The actual system of course is not constrained to this --- but is there a similar way to look at this with subspaces, and are the subspaces "tied together" between tasks, so that instead of forming a basis of each class mean they are forming one of task space?

eq 2 Gibbs distribution is basically another mixture of gaussians, with means {W_j v}.  Why not use MOG here exactly?

challenges 1, 2:  To me, the l_neg sampling in challenge 1 looks like the contrastive term that would be supplied by partition function mentioned in challenge 2 --- certainly it's of the same form and is subtracted.  I'm not entirely sure of its exact placement, whether it's a substitute for this or if they are in separate places.  Do they correspond in this way, and if so, would the trivial solution in challenge 1 come up if the partition function weren't replaced with a constant bound?

Text in 3.3 uses $\psi$ referring to Fig 1b, but Fig 1b doesn't mention $\psi$ anywhere in it.  In general there is a bit of back-and-forth between use of $\psi$ and $\theta$, since it's mentioned $\psi = \theta$.  Could this be made more consistent?

Sec 4.1:  The ProtoNet + GMM baseline is good, simple and very appropriate.  How many GMM components were used, and how large was the in-distribution training sample?  Were these tuned in any way to best performance for this baseline approach?

clarity,_quality,_novelty_and_reproducibility: This approach to task-modeling in few-shot learning is an interesting idea.  The descriptions are clear with a reasonable amount of attention and study, but I think they could be simplified or made more consistent (see comments above).
summary_of_the_review: Overall this is an interesting idea and model, with reasonable effectiveness.  It is somewhat complex, though, and I wonder if it (or some of its descriptions) could be simplified.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a Hierarchical Gaussian Mixture method to parametrize the generation process of tasks in meta-learning.  The proposed model can be used for fitting a mixture of task distributions and evaluating the scoring of testing tasks for evaluating the novelty of the testing task.  Experiments on several datasets demonstrate the effectiveness of the proposed methods. 
strength_and_weaknesses: **Strengths**:

* The proposed method can handle both a mixture of task distributions and detecting novelty in testing tasks is interesting.


**Weakness**:

* The proposed method is an application of Hierarchical Gaussian Mixture to the meta learning setting. The method itself is not novel and is a well-studied model in the machine learning community. 


* For the novel task detection experiments, it would be better to include more experiments that use Mini-ImageNet as source, and  CIFAR, aircraft, etc as novel tasks datasets to see whether it works well in practice. 


* The compared methods of optimization-based methods appear difficult to scale to a large backbone, but it is not always the case. For example, you can pre-train the backbone and fix most of layers, and only train the last or last few layers, similar to ANIL [1]

References;

[1] Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML. ICLR 2020

clarity,_quality,_novelty_and_reproducibility: The paper is overall clearly written

The proposed method is not new itself.

The results seem to be reproducible.
summary_of_the_review: This paper proposes a Hierarchical Gaussian Mixture method to parametrize the generation process of tasks in meta-learning. The method itself is not new and experiments need to be further improved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposed a new meta learning framework based on a Hierarchical Gaussian Mixture based Task Generative Model (HTGM) to address the issues in meta-learning: (1) the various sources of tasks may compose a multi-component mixture distribution, and (2) novel tasks may come from a distribution that is unseen during meta-training. The paper is well written, the proposed idea is well motivated, and the work is overall well executed. However, as pointed by the reviewers, the novelty of the proposed technique (most technical contributions are there in literature) is limited, and the empirical studies are not convincing enough (results on the selected dataset do seem strong but there is no evaluation on larger-scale general-domain image datasets such as Mini-ImageNet, etc. Overall, this is a well-written paper but the technical merits and empirical results are slightly below the acceptance bar.
Decision: Reject